JP2005123907A - Data reconstruction apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a data reconstruction apparatus capable of acquiring display time for each access unit in encoded data of an MPEG-4 AVC multiplexed by a PES packet. <P>SOLUTION: In step 205, by analyzing access unit data or information on a descriptor, a method for acquiring the display time of the access unit is determined, while in step 206, the display time is acquired on the basis of the method determined in step 205. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a data reconstruction device for reconstructing and outputting data obtained by packet-multiplexing encoded media data such as moving images and voices.

  In recent years, devices and services that handle encoded multimedia data such as moving images and voices have become widespread as storage media and communication networks have increased in capacity and transmission technology has advanced. As storage media, the rapid spread of DVD (Digital Versatile Disk) -related equipment is a notable example, and in the broadcasting field, broadcasting of digitally encoded media data has started instead of conventional analog broadcasting. . When storing and distributing media data, the header information and media data necessary for media data playback are multiplexed, but in digital broadcasting and DVD, ISO / IEC JTC1 / SC29 is used as a multiplexing standard. The MPEG (Moving Picture Experts Group) -2 system standard standardized in / WG 11 (International Standardization Organization / International Engineering Consortium) is used. As a moving picture coding system, MPEG-4 AVC (Advanced Video Coding) is being standardized as a successor to the currently popular MPEG-2 Visual and MPEG-4 Visual. It is expected that multiplexed video data will be multiplexed and stored or distributed according to the MPEG-2 system standard.

Hereinafter, a conventional example of separating data in which MPEG-2 Visual video encoded data is multiplexed by the MPEG-2 system will be described based on an example of use in the broadcast field. In broadcasting, encoded data is first multiplexed into a PES (Packetized Elementary Stream) packet, and further the PES packet is multiplexed into a TS (Transport Stream) packet. FIGS. 15A and 15B show the structures of the PES packet and the TS packet, respectively. Access unit (AU) data is stored in the payload of the PES packet. One access unit always includes encoded data for one picture, and may additionally store a sequence header and a GOP (Group of Pictures) header. Hereinafter, in the present specification, AU is used as a reference unit when handling encoded data. (1) to (3) in FIG. 15 (a) show an example of storing AU data in the payload of the PES packet. As shown in (1) and (2), one or more AUs are grouped together. The AU data may be divided and stored as shown in (3). The header of the PES packet can store the decoding time or display time of the first AU starting in the payload. Here, the decoding time and the display time are stored in accordance with the provisions of the MPEG-2 system standard. However, the time information is not necessarily stored in all PES packets, and there are PES packets in which time information is not stored. Exists. When the time information of the AU whose decoding time or display time is not indicated by the header of the PES packet is required, the AU data is analyzed to obtain the difference value of the decoding time or display time from the immediately preceding AU. The time information is acquired by adding the difference values based on the display time of the AU whose time information is indicated by the header of the PES packet or the decoding time. On the other hand, as shown in FIG. 15B, the data of the PES packet is divided and stored in the payload of the TS packet. A TS packet is a fixed-length packet composed of a 4-byte header and a 184-byte payload. When the TS packet is received and reproduced, the data reconstructing unit separates the PES packet from the received data, obtains the AU data and the AU display time from the separated PES packet, and then decrypts the data. To decrypt the AU data. The processing of the conventional data reconstruction device will be described with reference to the flowchart of FIG. Here, the processing after separating the PES packet data from the payload data of the TS packet will be described. First, in step 101, the PES packet header and payload are separated. In step 102, an AU boundary is detected from the payload of the separated PES packet. Steps 101 and 102 are repeated until an AU boundary is detected in step 102, it is determined that 1 AU of data has been prepared when the AU boundary is detected, and AU data is separated. In step 103, the AU display time is acquired based on the display time stored in the header of the PES packet and the display time information acquired from the AU data by a predetermined method. Here, the predetermined method is to obtain a temporal reference value for each AU and compare it with the temporal reference value of the immediately preceding AU, thereby calculating a difference value of the display time with the immediately preceding AU.
Japanese Patent Laid-Open No. 9-139937 (Section 47-55, FIG. 6)

  In MPEG-4 AVC, the display time information can be indicated by two types of parameters in the AU, but it is not guaranteed that these parameters always exist, and even if they exist, the display time information is not displayed. May not show. However, since the conventional data reconstruction device acquires display time information from AU data by a predetermined method, only one parameter can be referred to, and there is no parameter to be referred to, or display time information is not displayed. When not shown, there was a first problem that the AU display time could not be acquired.

  In MPEG-4 AVC, there are two types of picture data storage formats in AU. In the conventional data reconstruction device, the input AU data is output as it is without being changed, so there are two types of data storage formats in the output AU. For this reason, when a decoding device that decodes output AU data supports only one of the data storage methods, it cannot be decoded if an AU with a data storage method that is not supported is output. There was a second problem.

  Further, since the conventional data reconstruction device outputs AU data and AU display time separately, a third problem is that a new means for managing AU data and display time in association with each other is required. there were.

The present invention has been made to solve the above problems.
A data reconstruction device according to claim 1 of the present invention is a data reconstruction device that separates and outputs each media data from data obtained by packet-multiplexing encoded media data such as moving images and voices. Separating means for separating the header and payload of the packet; detection separating means for detecting and separating a frame boundary of the media data from the payload data; and a decoding time of the frame of the separated media data; Alternatively, it includes a time information acquisition unit that acquires a display time, and an output unit that outputs the separated frame data and the acquired time information. The time information acquisition unit includes identification information included in the media data. Or based on parameter information related to the media data acquired separately from the media data. There are, is characterized in that for switching the method of obtaining the decoding time, or display time.

A data reconstruction device according to a second aspect of the present invention is the data reconstruction device according to the first aspect, wherein the time information acquisition unit is configured to generate a time for each frame encoded in a specific encoding mode. It is characterized by determining whether or not the information acquisition method is switched.
A data reconstruction device according to claim 3 of the present invention is the data reconstruction device according to claim 1, wherein the frame data is composed of one or more subframe units, and the top of the subframe unit includes the frame data. When subframe size information is stored, storage format conversion means for converting the size information into an identification code indicating the start of the subframe is further provided.

A data reconstruction device according to a fourth aspect of the present invention is the data reconstruction device according to the first aspect, wherein when the frame data is composed of one or more subframe units, the output means includes: The time information acquired by the time information acquisition means is stored in the storage unit of the subframe, added to the frame data, and output.
A data reconstruction device according to claim 5 of the present invention is a data reconstruction method for separating and outputting each media data from data obtained by packet-multiplexing encoded media data such as moving images and voices. A separation step of separating the header and payload of the packet; a detection separation step of detecting and separating a frame boundary of the media data from the payload data; and a decoding time of the frame of the separated media data; Alternatively, it includes a time information acquisition step for acquiring a display time, and an output step for outputting the separated frame data and the acquired time information. In the time information acquisition step, the identification information included in the media data Or the media data acquired separately from the media data Based on the associated parameter information, it is characterized in that for switching the method of obtaining the decoding time, or display time.

  A program according to a sixth aspect of the present invention is a program for a data reconstruction device that separates and outputs each media data from data obtained by packet-multiplexing encoded media data such as moving images and sounds. A separation step of separating the header and payload of the packet, a detection separation step of detecting and separating a frame boundary of the media data from the payload data, and a decoding time of the frame of the separated media data Or a time information obtaining step for obtaining a display time, and an output step for outputting the separated frame data and the obtained time information. In the time information obtaining step, the identification included in the media data Information or the media data acquired separately from the media data Iadeta based on the associated parameter information, it is characterized in that to execute the steps, characterized in that for switching the method of obtaining the decoding time, or the display time on the computer.

According to the data reconstruction device of the first aspect of the present invention, since the display time acquisition method is switched based on the identification information included in the AU data or the parameter information related to the encoded data, the header of the PES packet As a result, it is possible to obtain the display time even for the AU whose display time is not indicated.
According to the data reconstruction device of the second aspect of the present invention, in order to determine whether to switch the display time acquisition method for each AU encoded in the specific encoding mode, the AU in the middle of the encoded data is determined. Even when the setting method of the display time information or the parameter information necessary for obtaining the display time information changes, an effect that the correct display time can be obtained can be obtained.

  According to the data reconstruction device of the third aspect of the present invention, by converting the data storage format of the AU data, the data storage format of the output AU data is changed to one regardless of the data storage format of the input AU. Since the data can be unified, an output data can always be decoded even in a decoding apparatus that only supports decoding of AU data that is a specific data storage format.

According to the data reconstruction device of the present invention, the means for associating and managing the AU data and the AU display time by adding the AU display time to the AU data and outputting it. The effect that it becomes unnecessary is acquired.
According to the data reconstruction method of the fifth aspect of the present invention, since the display time acquisition method is switched based on the identification information included in the specific AU data or the parameter information related to the encoded data, the PES packet Even for an AU whose display time is not indicated by the header, the effect that the display time can be obtained without fail is obtained.

  According to the program described in claim 6 of the present invention, the computer is caused to perform a process of switching the display time acquisition method based on the identification information included in the specific AU data or the parameter information related to the encoded data. Therefore, an effect that a data reconstruction device that can always acquire the display time can be realized even for an AU whose display time is not indicated by the header of the PES packet is obtained.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(Embodiment 1)
First, the data reconstruction device according to the first embodiment of the present invention will be described. Here, it is assumed that a PES packet multiplexed with MPEG-4 AVC encoded video data is input to perform reconstruction processing. As a multiplexing method, RTP (Real Time Transport Protocol) packets mainly used for streaming on the Internet, or a file format such as MP4 defined by ISO / IEC 14496-1 may be used. Good. Also, the encoded data to be packet-multiplexed is not limited to MPEG-4 AVC, but is video encoded data such as MPEG-4 Visual or audio encoded data such as MPEG-4 AAC or AMR. May be.

  FIG. 1 is a diagram showing the overall configuration of the data reconstruction device. This data reconstruction device inputs PES packet data, stores it in the memory 102, and outputs data AU data after the reconstruction processing is completed, and data I / O means 101 for outputting the related data, and the PES packet from the memory 102. Reconfiguration processing input data necessary for reconfiguration is acquired, reconfiguration is performed in units of AUs, and reconfiguration means 103 is configured to output the result of the reconfiguration processing to the memory 102. The reconfiguration processing input data refers to PES packet data input from the data I / O unit 101 and held in the memory 102, or the reconfiguration processing result of the reconfiguration unit 103 already held in the memory 102. Here, the memory 102 has a storage area for input / output data and a work area for temporarily storing the reconstruction result and AU data, and the size of each area can be made variable. In addition, the reconfiguration unit 103 is a CPU here, but may be dedicated hardware.

  First, the structure of AU data in MPEG-4 AVC will be described. In MPEG-4 AVC, in addition to data essential for picture decoding, auxiliary information called SEI (Supplemental Enhancement Information) that is not essential for decoding and boundary information in the encoded data are included in the encoded data. These data are all stored in a NAL (Network Adaptation Layer) unit. As shown in FIG. 2A, the NAL unit is composed of a header and a payload, and the header includes a field indicating the type of data stored in the payload (hereinafter referred to as a NAL unit type). The NAL unit type has a value defined for each type of data such as slice and SEI, and refers to the NAL unit type when acquiring the type of data stored in the NAL unit. As shown in FIGS. 2B and 2C, in addition to slice data for one picture, the AU stores NAL units such as header information and SEI. The NAL unit stores NAL unit data. Since there is no information for identifying the boundary, the boundary information can be added to the head of each NAL unit when storing the AU. As boundary information, a method of adding a start code indicated by 3 bytes of 0x000001 as shown in FIG. 2B (hereinafter referred to as a byte stream format) and the size of the NAL unit as shown in FIG. There are two types of methods (hereinafter referred to as NAL size format). In the byte stream format, one or more zero_bytes (1 byte whose value is 0x00) is added before the start code for the first NAL unit of the AU and the NAL unit having a specific NAL unit type value. It is prescribed. When multiplexing MPEG-4 AVC AU data in a PES packet, a byte stream format is used. Note that the NAL size format is used in MP4, and the boundary information of the NAL unit is not added when AU data is stored in the RTP packet. Next, the slice and header information will be described in detail. The slices are divided into two types: IDR (Instantaneous Decoder Refresh) slices and other slices. The IDR slice is slice data encoded in the screen, and header information such as SPS (Sequence Parameter Set) described later can be switched only in the IDR. When an IDR slice is included in a picture, all other slices in the same picture are also IDR slices. Here, a sequence from a picture including an IDR slice to a picture immediately before a picture including the next IDR slice is referred to as a sequence. Next, there are two types of header information, SPS and PPS (Picture Parameter Set), SPS is fixed header information in sequence units, and PPS is header information that can be switched in picture units. There can be a plurality of SPS and PPS, and each SPS or PPS is distinguished by an index number. Further, one SPS or PPS is stored in one NAL unit. The index numbers of the SPS and the PPS referred to by each picture are obtained as follows. First, the PPS index number referenced by the picture is indicated in the header portion of the slice data. Next, since the PPS indicates the index number of the SPS referred to by the PPS, the index number of the SPS referenced by the picture is obtained by analyzing the PPS referenced by the picture. FIG. 3 shows the NAL unit in the AU data stored in the PES packet. An access unit delimiter (AUD) NAL unit is always placed at the head of the AU. Subsequent NAL units are arranged in the order of SPS, PPS, SEI, and slice, but all AUs always include only the NAL unit of the slice and the NAL unit of the access unit delimiter. FIG. 3A illustrates an example of storage of NAL units in an AU (IDR AU) including an IDR slice, and FIG. 3B illustrates storage of NAL units in an AU including a non-IDR slice (non-IDR AU). An example is shown. In (a), NAL units of Buffering Period SEI and Picture Timing SEI indicating decoding time or display time information are included. The Picture Timing SEI stores a difference value between the decoding time and the display time difference between the nearest AU including the Buffering Period SEI and the AU including the Picture Timing SEI, and the AU including the Picture Timing SEI. Is done. When the AU decoding time and display time are indicated by these two SEIs, the Buffering Period SEI is always stored in the IDR-AU, and the Picture Timing SEI is always stored in all AUs. Furthermore, although not shown here, a NAL unit called an end-of-stream indicating the end of the encoded data can be added as the last NAL unit of the final AU in the encoded data.

The operation of the reconstruction unit 103 will be described in detail below.
FIG. 4 is a block diagram showing the configuration of the reconfiguration unit 103. The reconstruction unit 103 includes a PES separation unit 201, an AU separation unit 202, and a PTS acquisition unit 203. The PES separation unit 201 separates the header and payload of the PES packet from the input PES packet data, and inputs them to the AU separation unit 202 and the PTS acquisition unit 203. The AU separation unit 202 separates the NAL unit data from the payload data of the PES packet input by the PES separation unit, and analyzes the header information of the PES packet, the NAL unit type, or the payload data of the NAL unit, thereby defining the AU boundary. After detection, the AU data is separated and output. Further, the PTS acquisition unit 203 displays the display time in the PES packet header input by the PES separation unit, the display time information obtained by analyzing the AU data separated by the AU separation unit 202, and the display held in the memory 102. AU display time is acquired from the time information and output. A processing flow of the reconfiguration unit 103 will be described with reference to FIG. First, in step 201, the header and payload of the PES packet are separated. In step 202, each NAL unit constituting the AU is separated from the payload of the separated PES packet. In step 203, the NAL unit type of the NAL unit separated in step 202 or the payload data of the NAL unit is analyzed. Detect AU boundaries. Next, in step 204, it is determined whether the AU is an IDR AU. If it is IDR AU, the process proceeds to step 205. If it is not IDR AU, the process proceeds to step 206. Whether or not AU is IDR AU is acquired at the time of detecting the AU boundary in step 203. In step 205, an AU display time (PTS, Presentation Time Stamp) acquisition method is determined based on parameter values in the SPS. There are two methods for indicating display time information in AU data: a method using Buffering Period SEI and Picture Timing SEI, and a method using a parameter called POC (Picture Order Count) set for each AU. Here, the POC value is calculated from the field value in the SPS and the field value in the header of the slice data. Normally, POC represents the display order of AUs. However, under a predetermined condition, the POC difference value between AUs matches the difference value of the display time between AUs. Can be used for acquisition. Subsequently, in step 206, the AU display time is determined by the acquisition method determined in step 205. The reconstruction unit 103 determines that the AU data input has been completed by detecting the end-of-stream NAL unit, and terminates the AU data reconstruction process. The termination instruction from the user is determined in advance. The reconstruction process may be terminated based on conditions or the like. Note that when separating AU data from RTP packets in which MPEG-4 AVC AU data is multiplexed, the AU boundary can be identified only from the header information of the RTP packet, so the NAL unit type and NAL of the NAL units that constitute the AU. The unit payload data may not be analyzed.

  Hereinafter, the processing of step 205 and step 206 will be described in detail. FIG. 6 is a flowchart showing the processing of step 205. Here, poc_flag is a flag indicating whether display time information can be acquired from POC, and sei_flag is a flag indicating whether display time information can be acquired from SEI. Both flags have an initial value of 0. First, in step 301, it is determined whether or not the POC can be used to acquire display time information based on information other than AU data. In the MPEG-2 system standard, information about encoded data and the like can be stored separately from AU data by data called a descriptor. A descriptor is also prepared for MPEG-4 AVC, and flag information called temporal_poc_flag in AVC-timing and HRD Descriptor, which is one of them, indicates whether or not display time information can be acquired from the POC. Here, when temporal_poc_flag is set to 1, poc_flag is set to 1. When a PES packet in which encoded data is multiplexed is transmitted as a TS packet, the descriptor is stored in a data storage unit called a Program Map Table (PMT), and is multiplexed into the TS packet separately from the PES packet. For this reason, when receiving a PES packet multiplexed with a TS packet, the PMT is analyzed prior to the start of analysis of the PES packet, and when the AVC-timing and HRD Descriptor is included in the PMT, temporal_poc_flag Get the value of. On the other hand, when a TS packet is not used as in a storage format called a program stream in the MPEG-2 system, the descriptor is stored in the header of the PES packet. Therefore, when analyzing the PES packet data, the AVC-timing and HRD is performed prior to the analysis of the AU data in the PES packet by analyzing the AU data stored in the payload of the PES packet after analyzing the header of the PES packet. The Descriptor is analyzed to obtain a temporal_poc_flag value. Here, when the temporal_poc_flag value in the AVC-timing and HRD Descriptor indicated by the PMT or PES packet is updated, the updated value is acquired. When there is no AVC-timing and HRD Descriptor in the PMT or PES packet, it cannot be determined whether the display time information can be acquired by the POC, and the poc_flag value is 0. Note that the value of poc_flag may be set to indicate whether display time information can be acquired from the POC according to a descriptor other than AVC-timing and HRD Descriptor, or an operation rule. When the poc_flag value is set to 1 based on information other than AU data by the above method, the process proceeds to step 303, and when the poc_flag value is 0, the process proceeds to step 302. In step 302, information in the AU data is analyzed to determine a display time acquisition method, and a poc_flag value and a sei_flag value are determined based on the analysis result. Subsequently, in step 303, the poc_flag value acquired in step 301 or 302 is checked. If the poc_flag value is 1, the process proceeds to step 305. If it is 0, the process proceeds to step 304. In step 304, the sei_flag value acquired in step 302 is checked. If the sei_flag value is 1, the process proceeds to step 306. If it is 0, the process proceeds to step 307. In step 305, it is determined that the AU display time is acquired from the POC, and in step 306, it is determined that the AU display time is acquired from the SEI. When the values of poc_flag and sei_flag are both 0, the AU display time cannot be acquired from the POC or SEI. Therefore, in step 307, it is determined that the AU display time is not acquired from the AU data, and the display time is displayed. Are all obtained from the header information of the PES packet. With the above processing, the acquisition method of the AU display time is determined and the processing is terminated.

  Next, a method of determining the display time acquisition method by analyzing information in the AU data in step 302 will be described with reference to the flowchart of FIG. In step 302, the processing from step 401 to step 416 is repeated in order from the first NAL unit in the AU until the method determination flag is set to 1 in step 416. Here, the method determination flag is a flag indicating whether or not the display time information acquisition method has been determined, and the initial value at the start of the processing in step 302 is 0. First, in step 401, it is determined whether SEI is stored in the NAL unit. Since the NAL unit type value of the NAL unit storing the SEI is 6, it is determined whether or not the NAL unit type of the input NAL unit is 6, and when it is 6, the NAL unit type is stored in the payload of the NAL unit in step 402 Get the type of SEI. In step 403, it is determined whether or not the Buffering Period SEI is included in the NAL unit analyzed in step 402. If included, the process proceeds to step 404. If not included, the process proceeds to step 406. In step 404, bp_flag, which is a flag indicating that Buffering Period SEI exists in the AU, is set to 1. Note that the initial value of bp_flag at the start of the processing in step 302 is 0. The Buffering Period SEI indicates the SPS index number referenced by the picture in the AU in which the SEI is stored, so that the index number is acquired in step 405 and set in target_sps_id. It is assumed that target_sps_id is the index number of the SPS referenced by the picture data in the AU. In step 406, it is determined whether the AU is an IDR AU. Since the NAL unit type of the NAL unit that stores the IDR slice is 5, it is determined whether the NAL unit type of the input NAL unit is 5, and if it is 5, the process proceeds to step 407. The analysis of the current NAL unit is terminated, and the next NAL unit is analyzed. In step 407, the IDR slice detected in step 406 is analyzed, the index number of the SPS referenced by the picture in the AU is acquired, and set to target_sps_id. Next, in Step 408, SPS Video Usability Information (VUI) having the same index number as target_sps_id is acquired. Since the VUI includes fixed_frame_rate_flag that is a flag indicating whether or not the display of the AU is a fixed frame rate, this flag value is acquired, and it is determined whether or not the fixed_frame_rate_flag value is 1 in step 409. When the fixed_frame_rate_flag value is 1, the process proceeds to step 410, and when it is 0, the process proceeds to step 412. In step 410, poc_flag is set to 1. In step 411, delta_dpb which is a value indicating the relationship between the increment of the POC value between the AUs and the difference value of the display time is acquired. delta_dpb indicates the increment of the display time when POC increases by 1, and the time scale uses the value indicated by the field in the VUI. For example, if the POC values of two consecutive AUs au1 and au2 are poc1 and poc2, the difference value between the display times of au1 and au2 is (poc2−poc1) * delta_dpb. When the difference value is positive, au2 is displayed after au1, and when it is negative, au2 is displayed before au1. Here, when the picture in the AU has a frame structure, two types of values for the top field and for the bottom field are defined as POC values. When calculating the AU display time, the POC values of the top field and the bottom field are compared, and the POC value of the field with the earlier display time is used. Next, in step 412, poc_flag is set to 0, and the process proceeds to step 413. In step 413, it is determined whether the display time of AU can be acquired from SEI. In the VUI, there are two flags, vcl_hrd_parameters_present_flag and nal_hrd_parameters_present_flag, and if at least one of both flags is set to 1, it indicates that the display time of AU can be acquired from SEI. Accordingly, in step 413, it is determined whether one of vcl_hrd_parameters_present_flag and nal_hrd_parameters_present_flag is 1. When either one is 1, sei_flag1 is set to 1 at step 414, and when both are 0, sei_flag1 is set to 0 at step 415. Here, when the display time is calculated by SEI, the time scale uses a field in the VUI. Finally, after the processing of step 411, step 414, or step 415 is completed, the method determination flag is set to 1 in step 416. In the above description, it is assumed that the VUI always exists in the SPS and the fixed_frame_rate_flag field always exists in the VUI. However, the use of the VUI or the fixed_frame_rate_flag field in the VUI is not essential. For this reason, the processing when the VUI does not exist or the fixed_frame_rate_flag field does not exist in the VUI even if the VUI exists will be described below. First, in MPEG-4 AVC, information in the VUI such as a fixed_frame_rate_flag value and a time scale value can also be indicated by the descriptor. Therefore, when the fixed_frame_rate_flag value is indicated by the descriptor, the process of step 409 is performed according to the value. I will do it. Otherwise, both poc_flag and sei_flag are set to 0 and the process of step 416 is performed. In addition, when the operation rules indicate that the AU display time can be acquired by POC or SEI, the poc_flag value and the sei_flag value may be determined according to the operation rules.

Here, the SPS referred to by the AU can be updated only in the IDR AU. Since the acquisition method of the AU display time is updated only when the VPS information of the SPS is updated, the determination process of the display time acquisition method shown in step 205 of FIG. 5 is performed only in the IDR AU.
When it is indicated in advance that the acquisition method of the AU display time is fixed, the display time acquisition method determination process may be performed only in the first IDR AU. Further, the process of step 205 may be performed in the non-IDR AU.

  Note that the processing of step 302 may be included in the loop A0 in the processing flow of the reconfiguration unit 103 shown in FIG. In this case, when the NAL unit is separated in step 202, the NAL unit data is analyzed to determine the acquisition method of the AU display time.

  Next, a method of determining the AU display time in step 206 of FIG. 5 will be described with reference to the flowchart of FIG. Here, pts_flag is a flag indicating whether or not the AU display time is indicated by the header of the PES packet, and pes_pts indicates the AU display time indicated by the header of the PES packet. Further, curr_pts and prev_pts respectively indicate the current AU display time and the display time of the immediately preceding AU in decoding order, and delta_pts indicates the current AU display time calculated from POC or SEI, and decoding. The difference value with the display time of the immediately preceding AU is shown in order. delta_pts takes a positive or negative value and uses the same time scale as pes_pts. Also, max_PTS indicates the display time of the AU with the latest display time among the AUs whose decoding order is earlier than the current AU. Here, it is assumed that the value of pts_flag is acquired prior to step 501. In step 501, the value of pts_flag is checked. When the flag value is 1, in step 502, pes_pts is substituted for curr_pts, and the display time acquisition process ends. When the display time of all AUs can be acquired from the header of the PES packet, the display time is always determined by step 502. Next, when the pes_flag value is 0, the process proceeds to step 503. In step 503, the poc_flag value for AU is checked. When the flag value is 1, the process proceeds to step 504, and when it is 0, the process proceeds to step 506. In step 504, it is determined whether the current AU is an IDR AU. If the current AU is an IDR AU, the process proceeds to step 507. If not, the process proceeds to step 505. In step 505, the sei_flag value for AU is checked. If the flag value is 1, the process proceeds to step 506, and if it is 0, the process proceeds to step 508. In step 506, the display time of the current AU is obtained by adding the difference value between the current AU and the previous AU to the display time of the previous AU. On the other hand, in step 507, the AU display time is determined by adding delta_pts to max_PTS. Thus, when calculating the AU display time from the POC, the calculation method is switched depending on whether or not the AU is an IDR AU because the POC value is reset to 0 for each IDR AU. When the POC value is reset in the IDR AU, since the difference value of the POC from the immediately previous AU cannot be acquired, the display time is calculated by the method of step 507 using the fact that the frame rate is fixed. In step 508, since the display time information cannot be acquired from the POC and SEI, the display time of the AU whose pts_flag is 0 is calculated on the assumption that the frame rate is fixed. Specifically, when two AUs consecutive in decoding order with pts_flag being 1 are au1 and au2, and pes_pts of au1 and au2 are pts1 and pts2, respectively, values obtained by subtracting pts1 from pts2 are au1 and au2 The frame rate is calculated by dividing by the number of AUs included in between. Note that a predetermined value may be used as the frame rate, or the frame rate may be calculated only in a specific AU such as an IDR AU. In this method, the correct display time cannot be acquired in the AU using bi-directional prediction. Therefore, when the prediction structure of each AU is known in advance, the display time of the AU may be acquired in consideration of the prediction structure. Good.

In the display time in the header of the PES packet, in POC, Buffering Period SEI, and Picture Timing SEI, when the field value reaches the maximum value defined by the standard and circulates, the display time in consideration of the circulation, or The display time difference value is calculated.
In the above description, the reconstruction unit 103 outputs the display time, but the decoding time may also be output. The decoding time can also be calculated by the same procedure as the calculation of the display time, except for the method of using POC. POC can only display display time information, but po_flag is 1 only at a fixed frame rate, so the AU decoding time monotonously increases at a fixed frame rate. Therefore, the decoding time when poc_flag is 1 can be obtained by adding a constant value to the decoding time of the immediately preceding AU in decoding order.

  Note that the above method can be applied to other than MPEG-4 AVC as long as the encoding method needs to switch the display time acquisition method. In MP4, encoded data is managed in units called samples, which are composed of one or more AUs. When a sample includes a plurality of AUs, the AUs in the samples are displayed by the above method. You can get time.

(Embodiment 2)
Next, a data reconstruction device according to Embodiment 2 of the present invention will be described. The data reconstruction device according to the second embodiment is common in the main components to the data reconstruction device according to the first embodiment, but the reconstruction unit 103 has a characteristic configuration. This is different from the data reconstruction apparatus according to the first embodiment. Hereinafter, this difference will be mainly described. In addition, about the same component as the said Embodiment 1, the same code | symbol shall be used and description is abbreviate | omitted.

  FIG. 9 is a block diagram showing a configuration of the reconstruction unit 103 according to the second embodiment. The reconfiguration unit 103 has a function of converting and outputting the storage format of the NAL unit in the input AU. The AU separation unit 301 separates the NAL unit from the payload of the PES packet input by the PES separation unit, and inputs the separated NAL unit data and the size of the NAL unit to the storage format conversion unit 302. Here, in the NAL unit data input to the storage format conversion unit 302, the zero_byte added to the NAL unit in the byte stream format and the start code are removed. The storage format conversion unit 302 performs conversion on the input NAL unit data, and outputs the converted data to the AU separation unit 301. The AU separation unit 301 outputs AU data in which the NAL unit is stored in the storage format after being converted by the storage format conversion unit 302. FIG. 10 is a flowchart showing the processing in the reconfiguration unit 103, and the operation flow is processing for converting the storage format of the NAL unit into the processing of the loop A0 in the operation flow of the first embodiment shown in FIG. Step 301 is added. In step 301, the storage format of the NAL unit is converted from the byte stream format to the NAL size format. As a result, the storage format of the NAL unit in the AU data output from the reconstruction unit 103 is the NAL size format. The conversion process in the storage format conversion means 302 is shown in the flowchart of FIG. First, in step 401, a field indicating the size of the NAL unit in the NAL size format is created based on the size input together with the NAL unit data. At this time, a predetermined value is used as the field length of the field indicating the size of the NAL unit. The field length may be set according to the maximum value of the NAL unit of the slice estimated from the profile or level of MPEG-4 AVC, the bit rate at the time of encoding, or the like, or switched for each IDR AU. Further, it may be updated in the AU in the middle of the video data. Next, in step 402, a field indicating the size of the NAL unit created in step 401 is added to the NAL unit data, and the NAL unit data is output. Here, zero_byte and the start code are removed in advance and only the NAL unit data is input to the storage format conversion means 302. However, it may be input with the zero_byte and the start code added. In this case, prior to step 401, zero_byte and the start code are removed. In addition, when the AU data in the NAL size format is input to the reconstruction unit 103, the conversion process in the storage format conversion unit 302 is not performed. Further, when only NAL unit data is stored in the AU as in the AU multiplexed in the RTP packet, a field indicating the size of the NAL unit may be added and output. Furthermore, when AU data having a storage format different from that of the PES packet or RTP packet is input, it may be converted into a NAL size format and output.

  Thus, by converting from the byte stream format to the NAL size format, even when packet data in which AU data is multiplexed in the byte stream format, such as a PES packet, is input, only the NAL size format is supported. The AU data output from the reconstruction unit 103 can be decoded by the decoding unit that is not.

  In the above description, the byte stream format is converted to the NAL size format, but the NAL size format may be converted to the byte stream format. For example, although MP4 stores AU data in the NAL size format, it can be output in a unified byte stream format that is a storage format in the PES packet. FIG. 17 is a flowchart showing a process of converting from the NAL size format to the byte stream format. Here, it is assumed that the NAL unit data, the size of the NAL unit, and flag information indicating whether the NAL unit is the head NAL unit in the AU are input. First, in step 601, it is determined whether the NAL unit is the head NAL unit of the AU. If it is the head, the process proceeds to step 603, and if it is not the head, the process proceeds to step 602. In step 602, the NAL unit type of the NAL unit is analyzed to determine whether or not a predetermined type of data is included. Specifically, it is determined whether or not the NAL unit stores SPS or PPS. If the NAL unit is a NAL unit storing SPS or PPS, the process proceeds to step 603, and if not, the process proceeds to step 604. In step 603, one or more zero_bytes are created and added to the head of the AU data. Finally, in step 604, a start code is added and NAL unit data is output. At this time, when zero_byte is added in step 603, a start code is added immediately after the added zero_byte, and when zero_byte is not added, a start code is added to the head of the AU data.

  Note that zero_byte may be added to NAL units other than the NAL unit determined to add zero_byte in step 601 or step 602. For example, zero_byte can be added for stuffing. The NAL unit data may be input with a field indicating the size of the NAL unit added. In this case, the field indicating the size of the NAL unit is removed by acquiring the field length of the field indicating the size of the NAL unit in advance. For example, in MP4, the field length of the field indicating the size of the NAL unit is stored in the header area of the MP4 file, and the value is variable in units of chunks. Therefore, the size of the NAL unit for each AU to be converted. It is necessary to obtain the field length of the field indicating. Here, the chunk indicates a unit in which one or more samples are grouped. Further, when only the NAL unit data is stored in the AU, like the AU multiplexed in the RTP packet, it may be converted into the byte stream format.

(Embodiment 3)
Next, a data reconstruction device according to Embodiment 3 of the present invention will be described. The data reconstruction device according to the third embodiment is common in the main components to the data reconstruction device according to the second embodiment, but the reconstruction unit 103 has a characteristic configuration. In this respect, it differs from the data reconstruction device according to the second embodiment. Hereinafter, this difference will be mainly described. In addition, about the same component as the said Embodiment 2, the same code | symbol is used and description is abbreviate | omitted.

  FIG. 12 is a block diagram showing a configuration of the reconfiguration unit 103 according to the third embodiment. The reconfiguration unit 103 has a function of adding a special NAL unit including additional information such as an AU display time to the data of the NAL unit in the input AU and outputting it. The AU header adding unit 401 is a NAL unit having a special NAL unit type that stores the AU display time based on the AU display time input from the PTS acquisition unit 203 and the AU data output from the AU separation unit 301. Is added to the input AU data and output. FIG. 13 is a flowchart showing the processing in the reconstruction unit 103. The operation flow is obtained by acquiring the AU display time in the operation flow of the second embodiment shown in FIG. 502 is added. In step 501, a NAL unit storing the AU display time (hereinafter, this NAL unit is referred to as an AU header NAL unit) is created. In step 502, the AU header NAL unit created in step 501 is added to the head of the AU and output.

  FIG. 14 shows an example of the AU data structure output from the AU header adding means 401. An AU header NAL unit is arranged at the head of the AU. The value of the NAL unit type in the AU header NAL unit is 24, and the AU display time is stored in the payload of the NAL unit. In the MPEG-4 AVC standard, NAL unit types up to 24-31 do not define a usage method as a user-defined area, and the usage method can be determined by an application. In the decoding means for decoding the AU data, the NAL unit having the NAL unit type value that cannot be interpreted is skipped or discarded. Therefore, the AU in which the AU header NAL unit is added in the decoding means that cannot interpret the AU header NAL unit. Even when data is input, the AU header NAL unit is skipped or discarded without affecting the decoding process. Note that the NAL unit type value of the AU header NAL unit is not limited to 24, and a value that is unused in the application may be selected from other values in the user-defined area. The AU display time is stored in the payload of the AU header NAL unit. The time scale of the display time is matched with the time scale of the display time in the PES header. In step 203 of the reconstruction unit 103, when a NAL unit having a NAL unit type value of 24 is detected in the input AU data, the data of the NAL unit is discarded. In step 203, when the NAL unit type of all NAL units in the input AU is not analyzed, a process for analyzing the NAL unit type of all NAL units in the input AU is added separately, and the NAL unit type value of 24 is determined. Detection may be performed, and when it is detected, it may be discarded. Here, the AU display time is indicated by the AU header NAL unit. However, the display time may be indicated by adding a header of the PES packet. At this time, the reconstructing means 103 inputs one or more AU data in the payload or a PES packet in which the AU data is divided and stored, reconstructs the input PES packet data, and data for one AU in the payload Is output as a PES packet in which is stored.

  Note that the AU header NAL unit may be stored as the last NAL unit of the AU or an NAL unit in the middle. Furthermore, not only the display time but also the decoding time may be stored in the AU header NAL unit, or a NAL unit for storing the decoding time may be newly defined and stored. Further, in the decoding means for decoding the AU data, the AU header NAL unit can be used as identification information indicating the head of the AU. Note that a preset value may be used as the time scale of the display time stored in the payload of the AU header NAL unit.

  Further, when transmitting a PES packet using a TS packet, if a TS packet is lost in the transmission path, a NAL unit (hereinafter referred to as an error notification NAL unit) for notifying the occurrence of a packet loss is used as AU data. It is good also as inserting in. Here, the error notification NAL unit is inserted immediately after the packet loss detection position. FIG. 18 shows an example of an error notification NAL unit insertion method. FIG. 18A shows AU data input to the reconstruction unit 103. In the input AU data, the second half of the slice NALU # 2, which is the second slice NAL unit data, is missing due to the loss of the TS packet. FIG. 18B shows AU data after the error notification NAL unit is inserted. Whether or not a TS packet loss has occurred can be detected by analyzing the header of the TS packet. When the AU data shown in FIG. 18A is input, the packet is transmitted in the middle of the slice NALU # 2. The occurrence of loss is detected. Therefore, an error notification NAL unit is inserted immediately after the received slice NAL # 2 data. After the error notification NAL unit is inserted, at least the head portion of the NAL unit data is stored as output AU data from the NAL unit that can be acquired. For this reason, in the example of FIG. 18B, when the head portion of the slice NALU # 3 cannot be acquired, the data of the slice NALU # 3 is not stored in the output AU data, and the head of the NAL unit is next stored. Storage starts from the NAL unit for which the part was acquired. The NAL unit type of the error notification NAL unit is 25, and error notification information is stored in the payload. The error notification information includes (1) the data of the NAL unit immediately before the error notification NAL unit is completely prepared, (2) the data of the NAL unit immediately before the error notification NAL unit is incomplete, (3) the error notification Three types of information can be included: it is unknown whether the data of the NAL unit immediately before the NAL unit is completely prepared. In a TS packet, for example, if one NAL unit is multiplexed into one PES packet, the NAL units immediately before the error notification NAL unit are always completely aligned. When MPEG-4 AVC AU data is transmitted using an RTP packet, it is determined whether all of the data of the NAL unit data or a part of the data is lost when the packet loss occurs. Since it can be determined from the information, the content of the error notification information can be switched based on the determination result. When it is determined that the data of the NAL unit immediately before the error notification NAL unit is incomplete, the data of the NAL unit immediately before the error notification NAL unit may not be output. Even when it is unknown whether the data is incomplete, the data of the immediately preceding NAL unit may not be output. Note that the error AU data inserted after detecting the packet loss may be output AU data for the current AU. At this time, all the NAL unit data acquired after the packet loss is detected in the NAL unit in the AU is discarded. After the packet loss is detected, the data may be output from the next detected IDR AU or non-IDR AU data, or may be output from the next detected IDR AU data.

  Note that information other than the above three may be added as error notification information. Further, information indicating that AU data is missing due to packet loss may be added in units of AUs instead of units of NAL units. Further, the NAL unit type value of the error notification NAL unit is not limited to 25, and a value that is not used in the application may be selected from other values in the user-defined area.

Data other than the display time, decoding time, or error notification information may be added to the AU data in the NAL unit data format.
In each of the above embodiments, only one of the PTS acquisition unit 203, the storage format conversion unit 302, or the AU header addition unit in the reconstruction unit 103 may be used, or a plurality of units may be combined. It may be used.

  The data reconstruction method according to the present invention can be applied to all devices that reproduce packet-multiplexed MPEG-4 AVC encoded data, and particularly for digital broadcast receivers, streaming data receivers, and the like. It is valid.

It is a block diagram which shows the whole structure of the data reconstruction apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the data structure of AU in MPEG-4 AVC. It is a figure which shows the example of arrangement | positioning of the NAL unit in AU. It is a block diagram which shows the structure of the reconstruction means in the data reconstruction apparatus which concerns on Embodiment 1 of this invention. It is a flowchart which shows the process of the reconstruction means in the data reconstruction apparatus which concerns on Embodiment 1 of this invention. It is a flowchart which shows the determination procedure of the display time acquisition method of AU in the reconstruction means which concerns on Embodiment 1 of this invention. It is a flowchart which shows the acquisition method determination procedure at the time of acquiring display time from AU data in the reconfiguration | reconstruction means which concerns on Embodiment 1 of this invention. It is a flowchart which shows the display time acquisition procedure of AU in the reconfiguration | reconstruction means which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the reconstruction means in the data reconstruction apparatus which concerns on Embodiment 2 of this invention. It is a flowchart which shows the process of the reconstruction means in the data reconstruction apparatus which concerns on Embodiment 2 of this invention. It is a flowchart which shows the conversion process of the storage system of AU data in the data reconstruction apparatus which concerns on Embodiment 2 of this invention. It is a block diagram which shows the structure of the reconstruction means in the data reconstruction apparatus which concerns on Embodiment 3 of this invention. It is a flowchart which shows the process of the reconstruction means in the data reconstruction apparatus which concerns on Embodiment 3 of this invention. It is a figure which shows the data structure of AU after adding the display time of AU in the reconstruction means in the data reconstruction apparatus which concerns on Embodiment 3 of this invention. It is a figure which shows the structure of the PES packet and TS packet in MPEG-2 system specification. It is a flowchart which shows the process of the reconstruction means in the conventional data reconstruction apparatus. It is a flowchart which shows the process at the time of converting the storage system of AU data from a NAL size format to a byte stream format in the data reconstruction apparatus which concerns on Embodiment 2 of this invention. It is a figure which shows the data structure of AU after adding error notification information in the data reconstruction apparatus which concerns on Embodiment 3 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Data I / O means 102 Memory 103 Reconfiguration means 201 PES separation means 202, 301 AU separation means 203 PTS acquisition means 302 Storage format conversion means 401 AU header addition means

Claims (6)

A data reconstruction device that separates and outputs each media data from data obtained by packet-encoding encoded media data such as moving images and voices,
Separating means for separating the header and payload of the packet;
Detection separation means for detecting and separating frame boundaries of the media data from the payload data;
Time information acquisition means for acquiring a decoding time of a frame of the separated media data or a display time;
An output means for outputting the separated frame data and the acquired time information;
The time information acquisition means switches a method for acquiring a decoding time or a display time based on identification information included in the media data, or parameter information related to the media data acquired separately from the media data. A data reconstruction device. The data reconstruction apparatus according to claim 1, wherein the time information acquisition unit determines whether to switch a time information acquisition method for each frame encoded in a specific encoding mode. The data reconstruction device according to claim 1, wherein
When the frame data is composed of one or more subframe units, and the size information of the subframe is stored at the head of the subframe unit, the size information is an identification code indicating the start of the subframe. A data reconstruction device further comprising storage format conversion means for converting the data into a storage format. When the frame data is composed of one or more subframe units,
2. The data reconstruction according to claim 1, wherein the output unit stores the time information acquired by the time information acquisition unit in a storage unit of the subframe, adds the frame information to the frame data, and outputs the frame data. apparatus. A data reconstruction method for separating and outputting each media data from data obtained by packet-encoding encoded media data such as moving images and voices,
A separation step of separating the header and payload of the packet;
A detection separation step of detecting and separating a frame boundary of the media data from the data of the payload;
A time information acquisition step of acquiring a decoding time of the frame of the separated media data or a display time;
Outputting the separated frame data and the acquired time information,
In the time information acquisition step, the acquisition method of the decoding time or the display time is switched based on the identification information included in the media data or the parameter information related to the media data acquired separately from the media data. A data reconstruction method characterized by the above. A program for a data reconstruction device that separates and outputs each media data from data in which encoded media data such as moving images and audio are packet-multiplexed,
A separation step of separating the header and payload of the packet;
A detection separation step of detecting and separating a frame boundary of the media data from the data of the payload;
A time information acquisition step of acquiring a decoding time of the frame of the separated media data or a display time;
Outputting the separated frame data and the acquired time information,
In the time information acquisition step, the acquisition method of the decoding time or the display time is switched based on the identification information included in the media data or the parameter information related to the media data acquired separately from the media data. A program characterized by causing a computer to execute each step characterized by the above.

Images