RU2825853C2 - Method for decoding video bitstream - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to a method for decoding a video bitstream. Method comprises obtaining a set of image synchronization parameters, including a coded image buffer deletion delay parameter for removing an access unit from the coded image buffer at the access unit level, wherein each access unit comprises a plurality of decoding units; obtaining data representing an encoded video image; storing data in a buffer of encoded images; obtaining a first flag indicating whether it is necessary to remove the access unit from the encoded picture buffer for decoding at the access unit level or whether it is necessary to remove the decoding unit from the encoded picture buffer for decoding at the fragment level, wherein the decoding unit is a subset of the access unit; determining, based on the first flag, that the access unit or the decoding unit is subject to removal from the encoded image buffer; and in response to the determination that the decoding unit is to be removed from the encoded picture buffer for decoding at the image fragment level, (i) obtaining a second flag indicating whether the bitstream contains a video parameter representing a total deletion delay for all decoding units in the access unit, wherein the total deletion delay is the delay between the current decoding unit and the immediately preceding decoding unit; (ii) in response to determining that the flag indicates that a parameter representing the total deletion delay for all decoding units is contained in the bit stream, obtaining a parameter representing the total deletion delay; (iii) in response to determining that the flag indicates that a parameter representing a total deletion delay for all decoding units is not contained in the bitstream, obtaining separate removal delay parameters for decoding units, each of which represents a delay between successive decoding units in an access unit, wherein at least two separate removal delay parameters for decoding units are different, (iv) removing the decoding unit from the encoded picture buffer based on either a parameter representing a total deletion delay or individual deletion delay parameters for the decoding units; (v) decoding units of decoding; (vi) extracting an image based on the decoded decoding units; and in response to the determination that the access unit is to be removed from the encoded picture buffer for decoding at the access unit level, (i) determining the removal delay for the access unit for the access unit in the data, (ii) removal of the access unit from the coded picture buffer based on the removal delay for the access unit, and (iii) entropy decoding of the remote access unit to generate a plurality of quantised transformed coefficients.

EFFECT: technical result consists in improvement of efficiency of video bit stream decoding.

1 cl, 79 dwg, 50 tbl

Description

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

[0001] The present disclosure relates generally to electronic devices. More particularly, the present disclosure relates to electronic devices for signaling hypothetical reference decoder parameters based on a sub-image and to systems and methods for hybrid operation of a decoded picture buffer (DPB).

LEVEL OF TECHNOLOGY

[0002] Electronic devices have become smaller and more powerful to meet consumer demands and improve portability and convenience. Consumers have become dependent on electronic devices and have come to expect increased functionality. Some examples of electronic devices include desktop computers, laptops, cell phones, smart phones, media players, integrated circuits, etc.

[0003] Some electronic devices are used to process and display digital media. For example, portable electronic devices now allow the use of digital media almost anywhere the consumer may be. In addition, some electronic devices may provide over-the-air downloading or streaming of digital media content for the consumer to use and enjoy.

[0004] The increasing popularity of digital media has presented several challenges. For example, efficiently representing high-quality digital media for storage, transmission, and rapid playback presents several difficult tasks. As can be seen from this discussion, systems and methods that represent digital media efficiently with improved performance may be useful.

[0005] The above and other objects, features and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention taken in conjunction with the accompanying drawings.

ESSENCE OF THE INVENTION

Solution to the invention problem

[0006] An aspect of the invention provides a method for decoding a video bitstream (data), comprising: (a) receiving a base bitstream representing an encoded video sequence; (b) receiving a plurality of enhancement bitstreams representing said encoded video sequences; (c) receiving a video parameter set comprising syntax elements that apply to said base bitstream and said plurality of enhancement bitstreams, wherein said video parameter set comprises a syntax element signaling an extension of the video parameter set; (d) receiving said extension of the video parameter set comprising syntax elements associated with at least one of said enhancement bitstreams; (e) receiving an output level set change message comprising information indicating a change of at least one output level set.

[0007] An aspect of the invention provides a method for decoding a video bitstream (data), comprising: (a) receiving a base bitstream representing said coded video sequence; (b) receiving a plurality of enhancement bitstreams representing said coded video sequences; (c) receiving a video parameter set comprising syntax elements that apply to said base bitstream and said plurality of enhancement bitstreams, wherein said video parameter set comprises a syntax element signaling an extension of the video parameter set; (d) receiving said extension of the video parameter set comprising syntax elements that includes parameters related to a decoded picture buffer, relative to a decoded picture buffer for at least one of said enhancement bitstreams.

[0008] An aspect of the invention provides a method for encoding video, comprising: initiating parsing of a first header of a (independently reconstructible) slice segment of a current picture; determining which steps performed by a decoded picture buffer (DPB) will be based on the picture and which steps will be based on the access unit (AU); performing deletion from the DPB; performing output of the picture from the DPB; performing decoding and storing the current decoded picture in the DPB; marking the current decoded picture in the DPB; and performing additional output of the picture from the DPB.

[0009] An aspect of the invention provides an electronic device configured to encode video, comprising: a processor; a memory in electronic communication with the processor, wherein instructions stored in the memory are executable to: begin parsing a first slice header of a current picture; determine which steps performed by a decoded picture buffer (DPB) will be based on the picture and which steps will be based on the access unit (AU); perform deletion from the DPB; perform output of the picture from the DPB; perform decoding and storage of the current decoded picture in the DPB; mark the current decoded picture in the DPB; and perform further output of the picture from the DPB.

BRIEF DESCRIPTION OF DRAWINGS

[0010] Fig. 1A is a block diagram illustrating an example of one or more electronic devices in which systems and methods for sending a message and buffering a bit stream may be implemented.

Fig. 1B is another block diagram illustrating an example of one or more electronic devices in which systems and methods for sending a message and buffering a bit stream may be implemented.

Fig. 2 is a flow chart illustrating one configuration of a method for sending a message.

Fig. 3 is a flow chart illustrating one configuration of a method for determining one or more deletion delays for decoding blocks in an access block.

Fig.4 is a flow chart illustrating one configuration of a method for buffering a bit stream;

Fig. 5 is a flow chart illustrating one configuration of a method for determining one or more deletion delays for decoding blocks in an access block,

Fig. 6A is a block diagram illustrating one configuration of an encoder 604 on an electronic device.

Fig. 6B is another block diagram illustrating one configuration of the encoder 604 on the electronic device.

Fig. 7A is a block diagram illustrating one configuration of a decoder on an electronic device.

Fig. 7B is another block diagram illustrating one configuration of a decoder on an electronic device.

Fig. 8 is an illustration of various components that may be used in a transmitting electronic device.

Fig. 9 is a block diagram illustrating various components that may be used in a receiving electronic device.

Fig. 10 is a block diagram illustrating one configuration of an electronic device in which systems and methods for sending a message may be implemented,

Fig. 11 is a block diagram illustrating one configuration of an electronic device in which systems and methods for buffering a bit stream may be implemented,

Fig. 12 is a flow chart illustrating one configuration of a method for operating a decoded image buffer.

Fig.13A is an illustration of another NAL unit header syntax.

Fig.13B is an illustration of another NAL unit header syntax,

Fig.13C is an illustration of another NAL unit header syntax,

Fig.14 - illustration of the general syntax of a NAL unit.

Fig.15 - illustration of an existing set of video parameters.

Fig.16 - illustration of existing types of scalability.

Fig.17 - illustration of an approximate set of video parameters.

Fig.18 - illustration of an example syntax of a scalability mapping map,

Fig.19 - illustration of an approximate set of video parameters.

Fig.20 - illustration of an existing set of video parameters.

Fig.21 - illustration of an existing dimension type, dimension identifier (id) syntax.

Fig.22 - illustration of an approximate set of video parameters.

Fig.23 - illustration of an example syntax of a scalability mapping map.

Fig.24 - illustration of an approximate set of video parameters.

Fig.25 - illustration of an approximate set of video parameters.

Fig.26 - illustration of an approximate set of video parameters.

Fig.27 - illustration of an approximate syntax of a scalability mask.

Fig.28 - illustration of an exemplary syntax for extending a set of video parameters.

Fig.29 - illustration of an example syntax of the video parameter set extension,

Fig.30 - illustration of an exemplary syntax for extending a set of video parameters.

Fig.31 - illustration of an exemplary syntax for extending a set of video parameters,

Fig.32 - illustration of an exemplary syntax for extending a set of video parameters.

Fig.33 - illustration of an example syntax for extending a set of video parameters,

Fig.34 - illustration of an approximate syntax of a set of video parameters.

Fig.35 - illustration of an exemplary syntax for extending a set of video parameters.

Fig.36 - illustration of an example syntax for changing output level sets.

Fig.37 - illustration of another example syntax for changing output level sets,

Fig. 38A is an illustration of an exemplary syntax for a video parameter extension.

Fig.38B - illustration of an exemplary syntax of the video parameters extension,

Fig.39A - Illustration of sample syntax of op_dpb_info_parameters(j).

Fig.39B - Illustration of sample syntax of op_dpb_info_parameters(j).

Fig.40 - illustration of another example syntax for video parameter extension.

Fig.41 - illustration of another example syntax of op_dpb_info_parameters(j).

Fig.42 - illustration of another example syntax of op_dpb_info_parameters(j).

Fig.43 - illustration of the approximate syntax of num_dpb_info_parameters.

Fig.44 - illustration of another example syntax of op_dpb_info_parameters(j).

Fig.45 - illustration of another example syntax of num_dpb_info_parameters.

Fig.46 - illustration of another example syntax of num_dpb_info_parameters.

Fig.47 - illustration of another example syntax of video parameters extension and layer_dpb_info(i).

Fig.48 - illustration of the approximate syntax of oop_dpb_info_parameters and layer_dpb_info(i).

Fig.49A - Illustration of another example of vps_extension() syntax.

Fig.49B - Illustration of another example of vps_extension() syntax.

Fig.50 - illustration of approximate syntax of oop_dpb_maxbuffering_parameters(i).

Fig.51 - illustration of an example layer_dpb_info_parameters(i).

Fig.52 - illustration of another example vps_extension().

Fig.53 - illustration of another example vps_extension().

Fig.54 - illustration of an example oop_dpb_maxbuffering_parameters(i,k).

Fig.55 - illustration of an example oop_dpb_maxbuffering_parameters(i,k).

Fig.56 - illustration of another example vps_extension().

Fig.57 - illustration of an example oop_dpb_maxbuffring_parameters(i,k).

Fig.58 - illustration of an example oop_dpb_maxbuffring_parameters(i,k).

Fig.59 - illustration of an example oop_dpb_maxbuffring_parameters(i,k).

Fig.60 - illustration of an example oop_dpb_maxbuffring_parameters(i,k).

Fig.61 - illustration of an example oop_dpb_maxbuffering_parameters(i,k).

Fig.62 - illustration of an example seq_parameter_set_rbsp().

Fig.63 is a block diagram illustrating video coding between multiple electronic devices.

Fig.64 is a flow chart of a method for hybrid operation of a decoded picture buffer (DPB).

Fig.65 is a block diagram of another method for hybrid operation of a decoded picture buffer (DPB).

Fig.66 is a block diagram illustrating one configuration of a decoder;

Fig. 67A is a block diagram illustrating the use of both an enhancement layer and a base layer for video coding using separate decoded picture buffers (DPBs) and separate hybrid DPB operation modules for the base layer and enhancement layer.

Fig. 67B is a block diagram illustrating the use of a shared decoded picture buffer (DPB) and a shared hybrid DPB operation module for a base layer and an enhancement layer.

Fig.68 is a timing diagram illustrating the hybrid operation of the decoded picture buffer (DPB).

Fig.69 is a block diagram illustrating the structure and timing diagram for the network abstraction layer (NAL) units for the coded picture layers and the access units (AU) when the second enhancement layer (EL2) has a lower image (processing/transmission) rate than the base layer (BL) and the first enhancement layer (EL1).

Fig. 70 is a block diagram illustrating the structure and timing diagram for network abstraction layer (NAL) units for coded picture layers and access units (AUs) when the base layer (BL) has a lower picture rate than the first enhancement layer (EL1) and the second enhancement layer (EL2).

Description of embodiments

Example 1

[0011] An electronic device for sending a message is described. The electronic device includes a processor and instructions stored in a memory that is in electronic communication with the processor. The electronic device determines, if a coded picture buffer (CPB) supports operation at the subpicture level, whether to include a common deletion delay parameter in the CPB of the decoding unit in the additional extended information (SEI) message about picture synchronization. The electronic device also generates, when the common deletion delay parameter in the CPB of the decoding unit is to be included in the SEI message about picture synchronization (or some other SEI message or some other set of parameters, such as a set of picture parameters or a set of sequence parameters or a set of video parameters or a set of adaptation parameters), a common deletion delay parameter in the CPB of the decoding unit, where the common deletion delay parameter in the CPB of the decoding unit is applicable to all decoding units in an access unit of the CPB. The electronic device also generates, if the common deletion delay parameter in the decoding unit CPB is not to be included in the SEI message about picture synchronization, a separate deletion delay parameter in the decoding unit CPB for each decoding unit in the access unit. The electronic device also sends the SEI message about picture synchronization with the common deletion delay parameter in the decoding unit CPB or the deletion delay parameters in the decoding unit CPB.

[0012] The general parameter of the removal delay in the CPB of the decoding block may describe the number of clock cycles for a picture fragment to wait after removal from the CPB of the immediately preceding decoding block before removal from the CPB of the current decoding block in the access block associated with the SEI message of the picture synchronization.

[0013] In addition, when the decoding block is the first decoding block in the access block, the common deletion delay parameter in the CPB of the decoding block may indicate the number of clock cycles of a sub-picture to wait after deleting from the CPB the last decoding block in the access block associated with the latest SEI message about the buffering period in the previous access block, before deleting from the CPB the first decoding block in the access block associated with the SEI message about the picture synchronization.

[0014] In contrast, when the decoding block is not the first decoding block in the access block, the common deletion delay parameter in the CPB of the decoding block may indicate the number of clock cycles of a sub-picture to wait after deleting from the CPB the previous decoding block in the access block associated with the SEI message about the picture synchronization, before deleting from the CPB the current decoding block in the access block associated with the SEI message about the picture synchronization.

[0015] The CPB deletion delay parameters of a decoding block may specify the number of picture fragment clocks to wait after the last decoding block is deleted from the CPB before deleting the i-th decoding block from the CPB in the access block associated with the picture synchronization SEI message.

[0016] The electronic device may calculate the deletion delay parameters in the CPB of the decoding block according to the value of the remainder of the modulo counter , where cpb_removal_delay_length_minus1+1 is the length of the common removal delay parameter in the CPB of the decoding block.

[0017] The electronic device may also generate, when the CPB supports operation at the access unit level, a picture synchronization SEI message containing a CPB deletion delay parameter that specifies how many clock ticks to wait after deleting from the CPB the access unit associated with the most recent buffering period SEI message in the previous access unit before deleting from the CPB the access unit data associated with the picture synchronization SEI message.

[0018] The electronic device may also determine whether the CPB supports operation at the image fragment level or at the access unit level. This may include determining an image synchronization flag that indicates whether the coded picture buffer (CPB) provides parameters that support operation at the image fragment level, based on the value of the image synchronization flag. The image synchronization flag may be included in the SEI image synchronization message.

[0019] Determining whether to include the common deletion delay parameter in the CPB of the decoding unit may include setting the common deletion delay flag in the CPB of the decoding unit to 1 when the common deletion delay parameter in the CPB of the decoding unit is to be included in the SEI message about the picture synchronization. This may also include setting the common deletion delay flag in the CPB of the decoding unit to 0 when the common deletion delay parameter in the CPB of the decoding unit is not to be included in the SEI message about the picture synchronization. The common deletion delay flag in the CPB of the decoding unit may be included in the SEI message about the picture synchronization.

[0020] The electronic device may also generate, if the CPB supports operation at the sub-picture level, individual parameters related to network abstraction layer (NAL) units that indicate the number, offset by one, of NAL units for each decoding unit in the access unit. Alternatively, or in addition, the electronic device may generate a common NAL parameter that indicates the number, offset by one, of NAL units common to each decoding unit in the access unit.

[0021] An electronic device for buffering a bit stream is also described. The electronic device includes a processor and instructions stored in a memory that is in electronic communication with the processor. The electronic device determines that the CPB signals parameters at the level of a fragment of a picture for an access unit. The electronic device also determines if the received additional extended information (SEI) message about picture synchronization contains a common flag for deleting a decoding block from a coded picture buffer (CPB), a common deletion delay parameter in the CPB of the decoding block applicable to all decoding blocks in the access unit. The electronic device also determines if the SEI message about picture synchronization does not contain a common flag for deleting a delay in the CPB of the decoding block, a separate deletion delay parameter in the CPB of the decoding block for each decoding block in the access unit. The electronic device also deletes decoding blocks from the CPB using the common deletion delay parameter in the CPB of the decoding block or separate deletion delay parameters in the CPB of the decoding block. The electronic device also decodes the decoding blocks in the access unit.

[0022] In one configuration, the electronic device determines that the image synchronization flag is set in the image synchronization SEI message. The electronic device may also set the removal delay parameters in the CPB, cpb_removal_delay, according to

[0023] [Mathematical expression l]

[0024] where du_cpb_removal_delay[i] are the parameters of the removal delay in the CPB of the decoding block, t _c is the clock cycle, t _c,sub is the clock cycle for the image fragment,

num_decoding_units_minus1 - the number of decoding units in the access block, offset by one, and i is the index.

[0025] Alternatively, the electronic device may set the removal delay parameter from the CPB, cpb_removal_delay and du_cpb_removal_delay[num_decoding_units_minus1], so as to satisfy the equation

[0026] [Mathematical expression 2]

[0027] where du_cpb_removal_delay[i] is the removal delay parameters in the CPB of the decoding block, t _c is the clock tick, t _c,sub is the tick for the image fragment, num_decoding_units_minus1 is the number of decoding units in the access block offset by one, and i is the index.

[0028] Alternatively, the electronic device may set the removal delay parameter in the CPB, cpb_removal_delay and du_cpb_removal_delay[num_decoding_units_minus1] according to cpb_removal_delay*t _c ⁼ du_cpb_removal_delay[num_decoding_units_minus1]*t _c,sub , where du_cpb_removal_delay[num_decoding_units_minus1] is the removal delay parameter in the decoding unit CPB for the num_decoding_units_minus1'th decoding unit, t _c is the clock tick, t _c,sub is the tick for the image fragment, num_decoding_units_minus1 is the number of decoding units in the access unit offset by one.

[0029] In one configuration, the electronic device determines that the image synchronization flag is set in the image synchronization SEI message. The electronic device may also set the removal delay parameters in the CPB, cpb_removal_delay and du_cpb_removal_delay[num_decoding_units_minus1], to satisfy the equation: -1<=(cpb_removal_delay*t _c - du_cpb_removal_delay[num_decoding_units_minus1]*t _c,sub )<=1, where du_cpb_removal_delay[num_decoding_units_minus1] is the removal delay parameter in the decoding block CPB for the num_decoding_units_minus1'th decoding block, t _c is the clock tick, t _c,sub is the tick for the image fragment, num_decoding_units_minus1 is the number of decoding blocks in the access block, offset by one.

[0030] The ClockDiff variable can be defined as ClockDiff=(num_units_in_tick - (num_units_in_sub_tick*(num_decoding_units_minus1+1))/time_scale), where num_units_in_tick is the number of ticks (time units) of the clock generator operating at a frequency of time_scale Hz that corresponds to one increment of the system clock tick counter, num_units_in_sub_tick is the number of ticks of the clock generator operating at a frequency of time_scale Hz that corresponds to one increment of the tick counter for an image fragment, num_decoding_units_minus1+1 is the number of decoding units in an access unit, and time_scale is the number of ticks that elapse in one second.

[0031] When the low delay hypothetical reference decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, the CPB operates at the sub-picture level, and ClockDiff is greater than zero, the removal time for decoding block m is determined according to: t _r (m) = t _r,n (m) + t _{c_sub} * Ceil ((t _af (m) - _{t r,n} (m)) / t _{c_sub} ) + ClockDiff, where t _r,n (m) is the nominal removal time of decoding block m, t _{c_sub} is the clock for the sub-picture, Ceil () is a rounding function, and t _af (m) is the final arrival time of decoding block m.

[0032] When the hypothetical low delay reference decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (n) < t _af (n), the picture synchronization flag is set to 1, the CPB operates at the access block level, and ClockDiff is greater than zero, the removal time for access block n, t _r (n), is determined according to: t _r (n) = t _r,n (n) + t _c * Ceil ((t _af (n) - _{t r,n} (n)) / t _c ) - ClockDiff, where t _r (n) is the nominal removal time of access block n, t _c is a tick of the system clock, Ceil () is a rounding function, and t _af (n) is the final arrival time of access block n.

[0033] When the Hypothetical Reference Decoder (HRD) Low Delay Flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the sub-picture level, the removal time for the last decoding block m in an access block, t _r (m), is according to: t _r (m) = t _r,n (m) + max ( ( t _{c_sub} * Ceil ( ( t _af (m) - _{t r,n} (m)) / t _{c_sub} )), ( t _c * Ceil ( ( t _af (n) - _{t r,n} (n)) / t _c ))), where t _r (m) is the nominal removal time of the last decoding block m, t _c,sub is the clock for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal time of removal of access block n, t _c is the system clock tick, and t _af (n) is the final time of arrival of access block n.

[0034] When the hypothetical reference decoder (HRD) low delay flag is set to 1, t _r,n (n) < t _af (n), the picture synchronization flag is set to 1, and the CPB operates at the access block level, the removal time for access block n, t _r (n), is according to: where t _r,n (n) is the nominal removal time of the last decoding block n, t _{c_sub} is the clock tick for the image sub-frame, Ceil() is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of the access block n, t _c is the clock tick, and t _af (n) is the final arrival time of the access block n.

[0035] When the low delay reference decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the subpicture level, the removal time for the last decoding block m in the access block, t _r (m), is according to: , where t _r,n (m) is the nominal removal time of the last decoding block m, t _{c_sub} is the clock tick for the image fragment, Ceil() is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of the access block n, t _c is the clock tick, and t _af (n) is the final arrival time of the access block n.

[0036] When the Hypothetical Reference Decoder (HRD) Low Delay Flag is set to 1, t _r,n (n) < t _af (n), the Picture Sync Flag is set to 1, and the CPB operates at the access block level, the removal time for access block n, t _r (n), is according to: t _r (n) = t _r,n (n) + min ( (t _{c_sub} * Ceil ( (t _af (m) - _{t r,n} (m)) / t _{c_sub} )), (t _c * Ceil ( (t _af (n) - t _r,n (n)) / t _c ))), where t _r,n (m) is the nominal removal time of the last decoding block n, t _{c_sub} is the clock cycle for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of access block n, the system clock cycle hours, and t _af (n) is the final time of arrival of access block n.

[0037] When the Hypothetical Reference Decoder (HRD) Low Delay Flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the sub-picture level, the removal time for the last decoding block m in an access block, t _r (m), is according to: t _r (m) = t _r,n (m) + (t _c * Ceil ((t _af (n) - _{t r,n} (n)) / t _c )), where t _r,n (m) is the nominal removal time of the last decoding block m, t _{c_sub} is the clock cycle for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of access block n, t _c is the clock cycle, and t _af (n) is final time of arrival of access block n.

[0038] When the hypothetical reference decoder (HRD) low-latency flag is set to 1, t _r,n (n) < t _af (n), the picture synchronization flag is set to 1, and the CPB operates at the access block level, the removal time for access block n, t _r (n), is according to: t _r (n) = t _r,n (n) + (t _c * Ceil ((t _af (n) - _{t r,n} (n)) / t _c )), where t _r,n (m) is the nominal removal time of the last decoding block n, t _{c_sub} is the clock cycle for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of access block n, t _c is the clock cycle, and t _af (n) is the final arrival time of access block n.

[0039] When the low delay hypothetical reference decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the sub-picture level, the removal time for a decoding block m that is not the last decoding block is set to t _r (m) = t _af (m), where t _af (m) is the final arrival time of the decoding block m. When the Hypothetical Low Delay Reference Decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the sub-picture level, the removal time for decoding block m, which is the last decoding block m in an access block, t _r (m) is according to: t _r (m) = t _r,n (m) + (t _{c_sub} * Ceil ((t _af (m) - _{t r,n} (m)) / t _{c_sub} )), where t _r,n (m) is the nominal removal time of the last decoding block m, t _{c_sub} is the tick for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of access block n, t _c is the tick of the system clock, t _af (n) is the final arrival time of access block n, and t _af (m) is the final arrival time of the last decoding block m in access block n.

[0040] When the low delay hypothetical reference decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the sub-picture level, the removal time for a decoding block m that is not the last decoding block is set to t _r (m) = t _af (m), where t _af (m) is the final arrival time of the decoding block m. When the Hypothetical Low Delay Reference Decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the sub-picture level, the removal time for decoding block m, which is the last decoding block m in an access block, t _r (m) is according to: t _r (m) = t _r,n (m) + (t _c * Ceil ((t _af (m) - _{t r,n} (m)) / t _c )), where t _r,n (m) is the nominal removal time of the last decoding block m, t _{c_sub} is the tick for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of access block n, t _c is the tick of the system clock, t _af (n) is the final arrival time of access block n, and t _af (m) is the final arrival time of the last decoding block m in access block n.

[0041] When the hypothetical low delay reference decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the sub-picture level, the removal time for decoding block m is set to t _r (m) = t _af (m), where t _r,n (m) is a nominal removal time of decoding block m, t _{c_sub} is a tick for the sub-picture, Ceil() is a rounding function, t _af (m) is a final arrival time of decoding block m, t _r,n (n) is a nominal removal time of access block n, t _c is a tick of the system clock; t _af (n) is a final arrival time of access block n, and t _af (m) is a final arrival time of decoding block m in access block n.

[0042] When the hypothetical reference decoder (HRD) low-latency flag is set to 1, t _r,n (n) < t _af (n), the picture synchronization flag is set to 1, and the CPB operates at the access block level, the removal time for access block n, t _r (n), is according to: t _r (n) = t _af (n), where t _r,n (m) is a nominal removal time of the last decoding block n, t _{c_sub} is a tick for a sub-picture, Ceil() is a rounding function, t _af (m) is a final arrival time of the last decoding block m, t _r,n (n) is a nominal removal time of access block n, t _c is a tick of the system clock, and t _af (n) is a final arrival time of access block n.

[0043] Additionally, in some cases, a flag may be sent in a portion of the bitstream to signal which of the above alternative equations are used to decide on the decoding block removal time and the access block removal time. In one case, the flag may be named du_au_cpb_alignment_mode_flag. If du_au_cpb_alignment_mode_flag is 1, then the equations above that adjust the operation of a CPB that operates in a sub-picture mode with a CPB that operates in an access unit mode are used. If du_au_cpb_alignment_mode_flag is 0, then the equations above that do not adjust the operation of a CPB that operates in a sub-picture mode with a CPB that operates in an access unit mode are used.

[0044] In one case, the du_au_cpb_alignment_mode_flag flag may be signaled in the video usability information (VUI). In another case, the du_au_cpb_alignment_mode_flag flag may be sent in the picture synchronization SEI message. In yet another case, the du_au_cpb_alignment_mode_flag flag may be sent in some other normative part of the bitstream. One example of modified syntax and semantics in accordance with the systems and methods disclosed in the description is given in Table (0), as set forth below.

[0045] [Table 0]

[0046] It should be noted that other symbols (names) than those used above for the various variables may be used. For example, t _r (n) of access block n may be called CpbRemovalTime(n), t _r (m) of decoding block n may be called CpbRemovalTime(m), t _{c_sub} may be called ClockSubTick, t _c may be called ClockTick, t _af (n) of access block m may be called FinalArrivalTime(n) of access block n, t _af (m) of decoding block m may be called FinalArrivalTime(m), t _r,n (n) may be called the NominalRemovalTime(n) time of access block n, t _r,n (m) may be called the NominalRemovalTime(m) time of decoding block m

[0047] A method for sending a message by an electronic device is also described. The method includes determining, when a coded picture buffer (CPB) supports operation at the subpicture level, whether to include a common deletion delay parameter in the CPB of a decoding unit in a supplementary extended information (SEI) message about picture synchronization. The method also includes generating, when the common deletion delay parameter in the CPB of the decoding unit is to be included in the SEI message about picture synchronization, a common deletion delay parameter in the CPB of the decoding unit, wherein the common deletion delay parameter in the CPB of the decoding unit is applicable to all decoding units in an access unit of the CPB. The method also includes generating, when the common deletion delay parameter in the CPB of the decoding unit is not to be included in the SEI message about picture synchronization, a separate deletion delay parameter in the CPB of the decoding unit for each decoding unit in the access unit. The method also includes sending the SEI message about picture synchronization with the common deletion delay parameter in the CPB of the decoding unit or the deletion delay parameters in the CPB of the decoding unit.

[0048] A method for buffering a bit stream by an electronic device is also described. The method includes determining that the CPB signals parameters at the level of a fragment of a picture for an access unit. The method also includes determining when a received additional extended information (SEI) message of picture synchronization contains a common deletion delay flag from a coded picture buffer (CPB) of a decoding unit, a common deletion delay parameter in the CPB of the decoding unit applicable to all decoding units in the access unit. The method also includes determining when the SEI message of picture synchronization does not contain a common deletion delay flag in the CPB of the decoding unit, a separate deletion delay parameter in the CPB of the decoding unit for each decoding unit in the access unit. The method also includes deleting decoding units from the CPB using the common deletion delay parameter in the CPB of the decoding unit or separate deletion delay parameters in the CPB of the decoding unit. The method also includes decoding decoding units in the access unit.

[0049] The systems and methods disclosed in the description describe electronic devices for sending a message and buffering a bit stream. For example, the systems and methods disclosed in the description describe buffering for bit streams starting from parameters of a sub-picture. In some configurations, the systems and methods disclosed in the description can describe signaling of a sub-picture based on parameters of a Hypothetical Reference Decoder (HRD). For example, the systems and methods disclosed in the description describe a modification of a supplementary enhanced information (SEI) message about picture synchronization. The systems and methods disclosed in the description (e.g., a modification of the HRD) can result in a more compact signaling of parameters when each picture detail is received and removed from the CPB periodically.

[0050] In addition, when the parameters of the CPB deletion delay are present at the sub-picture level, the coded picture buffer (CPB) may operate at the access unit level or the sub-picture level. The present systems and methods may also impose a constraint on the bitstream so that the operation of the CPB based on the sub-picture level and the operation of the CPB at the access unit level result in the same timing of the deletion of the decoding unit. Specifically, the timing of the deletion of the last decoding unit in the access unit when operating in the sub-picture mode and the timing of the deletion of the access unit when operating in the access unit mode will be the same.

[0051] It should be noted that although the term "hypothetical" is used when referring to the HRD, the HRD may be physically implemented. For example, "HRD" may be used to describe the implementation of an actual decoder. In some configurations, the HRD may be implemented to determine whether a bitstream complies with the specifications of the High Efficiency Video Coding (HEVC) standard. For example, the HRD may be used to determine whether Type I bitstreams and Type II bitstreams comply with the HEVC specifications. A Type I bitstream may contain only network abstraction layer (NAL) units for the Video Coding Layer (VCL) and NAL units with filler data. A Type II bitstream may contain additional other NAL units and syntax elements.

[0052] The Joint Video Coding Task Force (JCTVC) document JCTVC-I0333 includes a fragment-based HRD and supports SEI picture synchronization messages. This functionality was included in the High Efficiency Video Coding (HEVC) Committee Draft (JCTVC-I1003), which is incorporated herein by reference in its entirety. The document "High efficiency video coding (HEVC) text specification draft 10 (for DFIS & Last Call)" by B. Bros, W-J. Han, J-R. Ohm, G.J. Sullivan, Wang, and T. Wiegand, JCTVC-J10003_v34, Geneva, January 2013, is hereby incorporated herein by reference in its entirety.

[0053] One example of modified syntax and semantics in accordance with the systems and methods disclosed in the description is given in Table (1), as set forth below.

[0055] Examples regarding the semantics of a buffering period SEI message in accordance with the systems and methods disclosed in the specification are given as follows. In particular, additional detail regarding the semantics of the modified syntax elements is given as follows. When the NalHrdBpPresentFlag or VclHrdBpPresentFlag flags are equal to 1, a buffering period SEI message may be associated with any access unit in the bitstream, and a buffering period SEI message may be associated with each IDR access unit, with each CRA access unit, and with each access unit associated with a restore point SEI message. For some applications, frequent presence of a buffering period SEI message may be desirable. The buffering period is defined as a set of access units between two instances of the buffering period SEI message in decoding order.

[0056] The 'seq_parameter_set_id' element specifies a sequence parameter set that contains the HRD sequence attributes. The value of seq_parameter_set_id may be equal to the value of seq_parameter_set_id in the picture parameter set referenced by the primary coded picture associated with the buffering period SEI message. The value of seq_parameter_set_id may be in the range from 0 to 31, inclusive.

[0057] 'initial_cpb_removal_delay'[SchedSelIdx] specifies the delay of the SchedSelIdx-th CPB between the time the first bit of coded data associated with the access block associated with the SEI message for the buffering period arrives in the CPB and the time the coded data associated with the same access block is removed from the CPB for the first buffering period after HRD initialization. The syntax element has a length in bits specified by initial_cpb_removal_delay_length_minus1+1. It is expressed in units of the 90 kHz clock. The value of initial_cpb_removal_delay[SchedSelIdx] may not be 0 and may not exceed 90000 * (CpbSize[ SchedSelIdx ]/BitRate[ SchedSelIdx ]), equivalent in time to the CPB size for a 90 kHz clock.

[0058] The 'initial_cpb_removal_delay_offset'[SchedSelIdx] value is used for the SchedSelIdx-th CPB in combination with cpb_removal_delay to specify the initial delivery time of coded access blocks in the CPB. The value of initial_cpb_removal_delay_offset[SchedSelIdx] is expressed in units of the 90 kHz clock. The syntax element initial_cpb_removal_delay_offset[SchedSelIdx] is a fixed-length code whose length in bits is given by initial_cpb_removal_delay_length_minus1+1. This syntax element is not used by decoders and is only needed by the delivery scheduler (HSS) (e.g., as defined in JCTVC-I1003 Annex C).

[0059] Over the entire encoded video sequence, the sum of initial_cpb_removal_delay[SchedSelIdx] and initial_cpb_removal_delay_offset[SchedSelIdx] may be constant for each value of SchedSelIdx.

[0060] 'initial_du_cpb_removal_delay'[SchedSelIdx] specifies the delay of the SchedSelIdx-th CPB between the time the first bit of coded data associated with the first decoding block in the access block associated with the SEI message about the buffering period arrives in the CPB and the time the coded data associated with the same decoding block is removed from the CPB for the first buffering period after HRD initialization. The syntax element has a length in bits specified by initial_cpb_removal_delay_length_minus1+1. It is expressed in units of the 90 kHz clock. The value of initial_du_cpb_removal_delay[SchedSelIdx] may not be equal to 0 and may not exceed 90000 * ( CpbSize[ SchedSelIdx ]/BitRate[ SchedSelIdx ] ), equivalent in time to the CPB size in 90 kHz clock units.

[0061] The 'initial_du_cpb_removal_delay_offset'[SchedSelIdx] element is used for the SchedSelIdx-th CPB in combination with cpb_removal_delay to specify the initial delivery time of decoding blocks in the CPB.

[0062] The value of initial_cpb_removal_delay_offset[SchedSelIdx] is expressed in units of a 90 kHz clock. The syntax element initial_du_cpb_removal_delay_offset[SchedSelIdx] is a fixed-length code whose length in bits is given by initial_cpb_removal_delay_length_minus1+1. This syntax element is not used by decoders and is only required by the delivery scheduler (HSS) (e.g., as defined in Annex C of JCTVC-I1003).

[0063] Over the entire coded video sequence, the sum of initial_du_cpb_removal_delay[SchedSelIdx] and initial_du_cpb_removal_delay_offset[SchedSelIdx] may be constant for each value of SchedSelIdx.

[0064] Examples regarding the semantics of the SEI image synchronization message in accordance with the systems and methods disclosed in the description are given as follows. In particular, additional details regarding the semantics of the modified syntax elements are given as follows.

[0065] The syntax of a picture synchronization SEI message depends on the contents of the sequence parameter set that is active for the coded picture associated with the picture synchronization SEI message. However, unless the immediate decoding refresh (IDR) picture synchronization SEI message of an access unit is preceded by a buffering period SEI message within the same access unit, the activation of the associated sequence parameter set (and, for IDR pictures that are not the first picture in the bitstream, the determination that the coded picture is an IDR picture) does not occur until the decoding of the network abstraction layer (NAL) unit for the first coded slice of the block for the coded picture. Since the NAL unit of a coded slice of a coded picture follows the picture synchronization SEI message in NAL unit order, there may be cases in which a decoder needs to store the raw byte sequence payload (RBSP) containing the picture synchronization SEI message before determining the parameters for the sequence parameter that will be active for the coded picture, and then parse the picture synchronization SEI message.

[0066] The presence of a picture synchronization SEI message in the bitstream is specified as follows. If the CpbDpbDelaysPresentFlag flag is equal to 1, one picture synchronization SEI message may be present in each access unit of the coded video sequence. Otherwise (CpbDpbDelaysPresentFlag is equal to 0), no picture synchronization SEI messages may be present in any access unit of the coded video sequence.

[0067] 'cpb_removal_delay' specifies how many clock ticks (see JCTVC-11003 subclause E.2.1) to wait after removing from the CPB an access unit associated with the most recent buffering period SEI message in the preceding access unit before removing from the buffer the data corresponding to the access unit associated with the picture synchronization SEI message. This value is also used to calculate the earliest possible arrival time of the access unit data in the CPB for the HSS as defined in JCTVC-I1003 Appendix C. The syntax element is a fixed-length code whose length in bits is given by cpb_removal_delay_length_minus1+1. The value of cpb_removal_delay is the remainder of the counter modulo .

[0068] The cpb_removal_delay_length_minus1 value, which specifies the length (in bits) of the cpb_removal_delay syntax element, is the cpb_removal_delay_length_minus1 value encoded in the sequence parameter set that is active for the primary coded picture associated with the picture synchronization SEI message, although cpb_removal_delay specifies the number of clock ticks relative to the removal time of the preceding access unit containing the buffering period SEI message, which may be an access unit of another coded video sequence.

[0069] The 'dpb_output_delay' element is used to calculate the output time for a picture from the decoded picture buffer (DPB). It specifies how many clock ticks to wait after the last decoding block in an access block is removed from the CPB before the decoded picture is output from the DPB (see subclause C.2 of document JCTVC-I1003).

[0070] As for the DPB, a picture is not removed from the DPB at its output time when it is still marked as "used for short-term reference" or "used for long-term reference". Only one dpb_output_delay element is specified for a decoded picture. The length of the dpb_output_delay syntax element is given in bits according to dpb_output_delay_length_minus1+1. When max_dec_pic_buffering[max_temporal_layers_minus1] is 0, dpb_output_delay may be equal to 0.

[0071] The output time obtained from dpb_output_delay of any picture that is output from a decoder that meets the output timing conditions as specified in subclause C.2 of JCTVC-I1003 may precede the output time obtained from dpb_output_delay for all pictures in any subsequent coded video sequence in decoding order. The picture output order established according to the values of this syntax element may be the same order as established by the values of PicOrderCnt(), as specified by subclause. For pictures that are not output by the sub-item "drop" process because they precede the IDR picture in decoding order with no_output_of_prior_pics_flag equal to 1, or are implied to be equal to 1, the output times obtained from dpb_output_delay may be increasing in value of PicOrderCnt() relative to all pictures within the same encoded video sequence.

[0072] 'num_decoding_units_minus1' plus 1 specifies the number of decoding units in the picture sync access unit with which the SEI message is associated. The value of num_decoding_units_minus1 can be in the range 0 to PicWidthInCtbs * PicHeightInCtbs - 1, inclusive.

[0073] A 'common_du_cpb_removal_delay_flag' value of 1 indicates that the common_du_cpb_removal_delay syntax element is present.

[0074] A 'common_du_cpb_removal_delay_flag' value of 0 indicates that the common_du_cpb_removal_delay syntax element is not present.

[0075] The 'common_du_cpb_removal_delay' element specifies information as follows: If the decoding block is the first decoding block in an access unit associated with a picture synchronization SEI message, then common_du_cpb_removal_delay specifies how many picture fragment clocks (see JCTVC-I1003 subclause E.2.1) to wait after the last decoding block in the access unit associated with the most recent buffering period SEI message in the previous access unit is removed from the CPB before the first decoding block in the access unit associated with the picture synchronization SEI message is removed from the CPB.

[0076] Otherwise, common_du_cpb_removal_delay specifies how many picture fragment clocks (see JCTVC-I1003 subclause E.2.1) to wait after the previous decoding block in the access unit associated with the picture synchronization SEI message is removed from the CPB before the current decoding block in the access unit associated with the picture synchronization SEI message is removed from the CPB. This value is also used to calculate the earliest possible arrival time of a decoding block in the CPB for the HSS as defined in Annex C. The syntax element is a fixed-length code whose length in bits is given by cpb_removal_delay_length_minus1+1. common_du_cpb_removal_delay is the value of the counter modulo 2 ^{(cpb_removal_delay_length_minus1+1)} .

[0077] An alternative way to set 'common_du_cpb_removal_delay' is as follows.

[0078] The common_du_cpb_removal_delay element specifies how many picture fragment clock cycles (see JCTVC-I1003 subclause E.2.1) to wait after the last decoding block is removed from the CPB until the current decoding block is removed from the CPB in the access block associated with the picture synchronization SEI message. This value is also used to calculate the earliest possible arrival time of a decoding block in the CPB for the HSS as defined in Annex C. The syntax element is a fixed-length code whose length in bits is given by cpb_removal_delay_length_minus1+1. The value of common_du_cpb_removal_delay is the remainder of the counter modulo 2 ^{(cpb_removal_delay_length_minus1+1)} .

[0079] The value of cpb_removal_delay_length_minus1, which specifies the length (in bits) of the common_du_cpb_removal_delay syntax element, is the value of cpb_removal_delay_length_minus1 encoded in the sequence parameter set that is active for the coded picture associated with the picture synchronization SEI message, although common_du_cpb_removal_delay specifies the number of ticks for a picture fragment relative to the time of removal of the first decoding block in the preceding access unit containing the buffering period SEI message, which may be an access unit for another coded video sequence.

[0080] 'num_nalus_in_du_minus1[i]' plus 1 specifies the number of NAL units in the i-th decoding unit in the access unit with which the picture synchronization SEI message is associated. The value of num_nalus_in_du_minus1[i] can be in the range from 0 to PicWidthInCtbs * PicHeightInCtbs - 1, inclusive.

[0081] The first decoding unit of an access unit consists of the first num_nalus_in_du_minus1[0]+1 consecutive NAL units in decoding order in the access unit. The i-th (if i is greater than 0) decoding unit in the access unit consists of num_nalus_in_du_minus1[i]+1 consecutive NAL units immediately following the last NAL unit in the previous decoding unit in the access unit, in decoding order. There may be at least one VCL NAL unit in each decoding unit. All non-VCL NAL units associated with a VCL NAL unit may be included in the same decoding unit.

[0082] 'du_cpb_removal_delay[i]' specifies how many ticks for a fragment of a picture (see JCTVC-I1003 subclause E.2.1) to wait after the first decoding block in the access unit associated with the most recent buffering period SEI message in the previous access unit is removed from the CPB, before the i-th decoding block in the access unit associated with the picture synchronization SEI message is removed from the CPB. This value is also used to calculate the earliest possible arrival time of a decoding block in the CPB for the HSS (e.g., as defined in Annex C of JCTVC-I1003). The syntax element is a fixed-length code whose length in bits is given by cpb_removal_delay_length_minus1+1. du_cpb_removal_delay[i] is the value of the counter remainder modulo 2 ^{(cpb_removal_delay_length_minus1+1)} .

[0083] The value of cpb_removal_delay_length_minus1, which specifies the length (in bits) of the syntax element du_cpb_removal_delay[i], is the value of cpb_removal_delay_length_minus1 encoded in the sequence parameter set that is active for the coded picture associated with the picture synchronization SEI message, although du_cpb_removal_delay[i] specifies the number of ticks for a picture fragment relative to the time of removal of the first decoding block in the preceding access unit containing the buffering period SEI message, which may be an access unit for another coded video sequence.

[0084] In one configuration, synchronization of deletion of a decoding block and decoding of decoding blocks may be implemented as described below.

[0084] If SubPicCpbFlag is 0, the variable CpbRemovalDelay(m) is set to the value of cpb_removal_delay in the picture synchronization SEI message associated with the access unit, that is, decoding unit m, and the variable T _c is set to t _c . Otherwise, if SubPicCpbFlag is 1 and common_du_cpb_removal_delay_flag is 0, the variable CpbRemovalDelay(m) is set to the value of du_cpb_removal_delay[i] for decoding unit m (with m varying from 0 to num_decoding_units_minus1) in the picture synchronization SEI message associated with the access unit that contains decoding unit m, and the variable T _c is set to t _{c_sub} .

[0086] In some cases, otherwise if SubPicCpbFlag is 1 and common_du_cpb_removal_delay_flag is 0, the variable CpbRemovalDelay(m) is set to the value of (m+1)*du_cpb_removal_delay[i] for decoding unit m (with m varying from 0 to num_decoding_units_minus1) in the picture synchronization SEI message associated with the access unit that contains decoding unit m, and the variable T _c is set to t _{c_sub} .

[0087] Otherwise, if SubPicCpbFlag is 1 and common_du_cpb_removal_delay_flag is 1, the variable CpbRemovalDelay(m) is set to the value of common_du_cpb_removal_delay for decoding block m in the picture synchronization SEI message associated with the access block that contains decoding block m, and the variable T _c is set to t _{c_sub} .

[0088] When decoding block m is a decoding block with n equal to 0 (the first decoding block in the access block that initializes the HRD), the nominal time for removing the decoding block from the CPB is set according to t _r,n (0) = InitCpbRemovalDelay [SchedSelIdx] / 90000.

[0089] When decoding block m is the first decoding block of the first access block for a buffering period that does not initialize HRD, the nominal time of removing the decoding block from the CPB is set according to t _r,n (m) = t _r,n (m _b ) + T _c * CpbRemovalDelay(m), where t _r,n (m _b ) is the nominal time of removing the first decoding block for the previous buffering period.

[0090] When the decoding block m is the first decoding block in the buffering period, m _b is set equal to m at the removal time t _r,n (m) of the decoding block m. The nominal removal time t _r,n (m) of the decoding block m that is not the first decoding block in the buffering period is t _r,n (m)=t _r,n (m _b )+T _c *CpbRemovalDelay(m), where t _r,n (m _b ) is the nominal removal time of the first decoding block for the current buffering period.

[0091] The removal time of decoding block m is set as follows. If low_delay_hrd_flag is equal to 0 or t _r,n (m)>=t _af (m), the removal time of decoding block m t _r,n is set according to t _r (m)=t _r,n (m). Otherwise (low_delay_hrd_flag is equal to 1 and t _r,n (m)<t _af (m)), the removal time of decoding block m is set according to t _r (m)=t _r,n (m)+T _c *Ceil((t _af (m)-t _r,n (m))/T _c ). The latter case (low_delay_hrd_flag is equal to 1 and t _r,n (m)<t _af (m)) indicates that the size of decoding block m, b(m), is so large that it prevents removal within the nominal removal time.

[0092] In another case, the removal time of the decoding block m is set as follows. If low_delay_hrd_flag is 0 or t _r,n (m)>=t _af (m), the removal time of the decoding block m t _r,n is set according to t _r (m)=t _r,n (m). Otherwise (low_delay_hrd_flag is 1, and t _r,n (m)<t _af (m)), the removal time of the decoding block m, which is not the last decoding block in the access unit, is set according to t _r (m)=t _af (m), and the removal time of the decoding block m, which is the last decoding block in the access unit t _r (m)=t _r,n (m)+T _c * Ceil ((t _af (m) - t _r,n (m)) / t _c ). The last case (low_delay_hrd_flag is 1 and t _r,n (m) < t _af (m)) indicates that the decoding block size m, b(m), is so large that it prevents removal within the nominal removal time.

[0093] In another case, the removal time of the decoding block m is set as follows. If low_delay_hrd_flag is 0 or t _r,n (m)>=t _af (m), the removal time of the decoding block m is set according to t _r (m)=t _r,n (m). Otherwise (low_delay_hrd_flag is 1 and t _r,n (m)<t _af (m)), the removal time of the decoding block m, which is not the last decoding block in the access unit, is set according to t _r (m)=t _af (m), and the removal time of the decoding block m, which is the last decoding block in the access unit, t _r (m)=t _r,n (m)+t _c * Ceil ((t _af (m) - t _r,n (m)) / t _c ). The last case (low_delay_hrd_flag is 1 and t _r,n (m) < t _af (m)) indicates that the decoding block size m, b(m), is so large that it prevents removal within the nominal removal time.

[0094] In another case, the removal time of decoding block m is set as follows. If low_delay_hrd_flag is equal to 0 or t _r,n (m)>=t _af (m), the removal time of decoding block m is set according to t _r (m)=t _r,n (m). Otherwise (low_delay_hrd_flag is equal to 1 and t _r,n (m)<t _af (m)), the removal time of decoding block m is set according to t _r (m)=t _af (m). The latter case (low_delay_hrd_flag is equal to 1 and t _r,n (m)<t _af (m)) indicates that the size of decoding block m, b(m), is so large that it prevents removal within the nominal removal time.

[0095] When SubPicCpbFlag is 1, the nominal removal time from the CPB of access block n t _r,n (n) is set to the nominal removal time from the CPB of the last decoding block in access block n, the removal time from the CPB of access block n t _r (n) is set to the value of the removal time from the CPB of the last decoding block in access block n.

[0096] When SubPicCpbFlag is 0, each decoding block is an access block, therefore the nominal removal time in CPB and the removal time in CPB for access block n are the nominal removal time in CPB and the removal time in CPB for decoding block n.

[0097] At the time of deletion in CPB of decoding block m, the decoding block is decoded immediately.

[0098] Another example of modified syntax and semantics for an SEI image synchronization message in accordance with the systems and methods disclosed in the description is given in Table (2), as set forth below. Modifications in accordance with the systems and methods disclosed in the description are indicated in bold.

[0099] [Table 2]

[0100] The example illustrated in Table (2) includes a syntax element common_num_nalus_in_du_minus1, which can be used to determine how much data should be removed from the CPB when a decoding unit is removed. 'common_num_nalus_in_du_minus1 plus 1' specifies the number of NAL units in each decoding unit in the access unit with which the picture synchronization SEI message is associated. The value of common_num_nalus_in_du_minus1 can be in the range from 0 to PicWidthInCtbs * PicHeightInCtbs - 1, inclusive.

[0101] The first decoding unit in an access unit consists of the first common_num_nalus_in_du_minus1+1 consecutive NAL units in decoding order in the access unit. The i-th (if i is greater than 0) decoding unit in an access unit consists of the common_num_nalus_in_du_minus1+1 consecutive NAL units immediately following the last NAL unit in the previous decoding unit of the access unit, in decoding order. There may be at least one VCL NAL unit in each decoding unit. All non-VCL NAL units associated with a VCL NAL unit may be included in the same decoding unit.

[0102] Another example of modified syntax and semantics for an SEI image synchronization message in accordance with the systems and methods disclosed in the description is given in Table (3), as set forth below. Modifications in accordance with the systems and methods disclosed in the description are indicated in bold.

[0103] [Table 3]

[0104] The illustrated example in Table (3) includes a syntax element, common_num_nalus_in_du_flag, which, if equal to 1, specifies that the syntax element 'common_num_nalus_in_du_minus1' is present. 'common_num_nalus_in_du_flag' equal to 0 specifies that the syntax element 'common_num_nalus_in_du_minus1' is not present.

[0105] In another embodiment, the common_du_cpb_removal_delay_flag, common_num_nalus_in_du_minus1 flags may not be sent. Instead, the common_num_nalus_in_du_minus1 and common_du_cpb_removal_delay syntax elements may be sent every time. In this case, a value of 0 (or some other value) for these syntax elements may be used to indicate that these elements do not signal.

[0106] In addition to modifications to the syntax elements and semantics of the SEI picture synchronization message, the present systems and methods may also implement bitstream restriction such that operation of the CPB on a picture fragment basis and operation of the CPB at the access block level result in the same decoding block removal timing.

[0107] When sub_pic_cpb_params_present_flag is equal to 1, the CPB deletion delay parameters are present at the sub-picture level, and the CPB may operate at the access unit level or the sub-picture level. sub_pic_cpb_params_present_flag equal to 0 indicates that the CPB deletion delay parameters are not present at the sub-picture level, and the CPB operates at the access unit level. When sub_pic_cpb_params_present_flag is not present, its value is implied to be 0.

[0108] To support operation at both the access block level and the picture sub-block level, the following bitstream constraints may be used: If sub_pic_cpb_params_present_flag is 1, then the bitstream conformance requirement is that the following constraint is obeyed when signaling values for cpb_removal_delay and du_cpb_removal_delay[i] for all i:

[0109] [Mathematical expression 3]

[0110] where du_cpb_removal_delay[i] are the parameters of the removal delay in the CPB of the decoding block, t _c is the clock cycle, t _c,sub is the clock cycle for the image fragment, num_decoding_units_minus1 is the number of decoding units in the access block with an offset of one, and i is the index. In some embodiments, a tolerance parameter may be added to satisfy the above constraint.

[0111] To support operation at both the access block level and the image fragment level, bitstream constraints may be used as follows: let the variable T _du (k) be defined as:

[0112] [Mathematical expression 4]

[0113] where du_cpb_removal_delay_minus1 _k [i] and num_decoding_units_minus1 _k are the parameters for the i-th decoding unit of the k-th access unit (with k=0 for the access unit that initialized HRD, and T _du (k)=0 for k<l), and where du_cpb_removal_delay_minus1 _k [i] + l = du_cpb_removal_delay_minus1 _k [i] is the removal delay parameter in the CPB of the decoding unit for the i-th decoding unit of the k-th access unit, and num_decoding_units_minus1 _k is the number of decoding units in the k-th access unit, t _c is the clock tick, t _c,sub is the tick for the image fragment, and i and k are indices. Then, when the picture synchronization flag (e.g., sub_pic_cpb_params_present_flag) is set to 1, the following constraint shall be true: (au_cpb_removal_delay_minus1+1)*t _c == T _du (k), where (au_cpb_removal_delay_minus1+1) == cpb_removal_delay, the removal delay from the CPB. Thus, in this case, the CPB removal delay (au_cpb_removal_delay_minus1+1) is set so that the operation of the CPB based on the fragment of the picture and the operation of the CPB based on the access block result in the same timing of the removal of the access block and the removal of the last decoding block for the access block.

[0114] To support operation at both the access block level and the subpicture level, the following bitstream constraints may be used: If sub_pic_cpb_params_present_flag is 1, then the bitstream conformance requirement is that the following constraint is obeyed when signaling values for cpb_removal_delay and du_cpb_removal_delay[i] for all i:

[0115] [Mathematical expression 5]

[0116] where du_cpb_removal_delay[i] is the parameters of the CPB removal delay of the decoding block, t _c is the clock cycle, t _c,sub is the clock cycle for the image fragment, num_decoding_units_minus1 is the number of decoding units in the access block with an offset of one, and i is the index.

[0117] To support operation at both the access unit level and the sub-picture level, the following bitstream constraints may be used: If sub_pic_cpb_params_present_flag is 1, then the bitstream conformance requirement is that the following constraint is obeyed when signaling values for cpb_removal_delay and du_cpb_removal_delay[num_decoding_units_minus1]: cpb_removal_delay*t _c =du_cpb_removal_delay[num_decoding_units_minus1]*t _c,sub , where du_cpb_removal_delay[num_decoding_units_minus1] is the removal delay parameter in the decoding unit CPB for the num_decoding_units_minus1-th decoding unit, t _c is the system clock tick, t _c,sub is the tick for the sub-picture, num_decoding_units_minus1 - the number of decoding units in the access unit offset by one. In some embodiments, a tolerance parameter may be added to satisfy the above constraint.

[0118] To support operation at both the access unit level and the subpicture level, the following bitstream constraints may be used: If sub_pic_cpb_params_present_flag is 1, then the requirement for the bitstream to comply with the constraint is that the following constraint is obeyed when signaling values for cpb_removal_delay and du_cpb_removal_delay[i] for all i: -1<=(cpb_removal_delay*t _c - du_cpb_removal_delay[num_decoding_units_minus1]*t _c,sub )<=1, where du_cpb_removal_delay[num_decoding_units_minus1] is the removal delay parameter in the decoding unit CPB for the num_decoding_units_minus1-th decoding unit, t _c is the system clock tick, t _c,sub is the tick for the subpicture, num_decoding_units_minus1 - the number of decoding units in an access block offset by one.

[0119] In addition, the present systems and methods can modify the timing of the removal of the decoding block. When the parameters of the removal delay from the CPB are present at the image sub-section level, the time of removal of the decoding block for "large pictures" (when low_delay_hrd_flag is 1, and t _r,n (m) < t _af (m)) can be changed to compensate for the difference that may arise due to the clock counter of the system clock and the clock counter for the image sub-section.

[0120] When sub_pic_cpb_params_present_flag is 1, the CPB deletion delay parameters at the sub-picture level are present, and the CPB can operate at the access unit level or the sub-picture level. sub_pic_cpb_params_present_flag equal to 0 indicates that the CPB deletion delay parameters at the sub-picture level are not present, and the CPB operates at the access unit level. When sub_pic_cpb_params_present_flag is not present, its value is implied to be 0.

[0121] Specifically, one example of implementing the synchronization of deleting a decoding block and decoding a decoding block is as follows. A variable SubPicCpbPreferredFlag is either set by an external means, or if not set by an external means, is set to 0. The variable SubPicCpbFlag is obtained as follows: SubPicCpbFlag=SubPicCpbPreferredFlag && sub_pic_cpb_params_present_flag. If SubPicCpbFlag is 0, the CPB operates at the access block level, and each decoding block is an access block. Otherwise, the CPB operates at the sub-picture level, and each decoding block is a subset of an access block.

[0122] If SubPicCpbFlag is 0, the variable CpbRemovalDelay(m) is set to the value cpb_removal_delay in the picture synchronization SEI message associated with the access unit that is decoding unit m, and the variable T _c is set to t _c . Otherwise, the variable CpbRemovalDelay(m) is set to the value du_cpb_removal_delay[i] for decoding unit m in the picture synchronization SEI message associated with the access unit that contains decoding unit m, and the variable T _c is set to t _{c_sub} .

[0123] When decoding block m is a decoding block with n equal to 0 (the first decoding block in the access block that initializes the HRD), the nominal time for removing the decoding block from the CPB is set according to t _r,n (0) = InitCpbRemovalDelay [SchedSelIdx] / 90000.

[0124] When decoding block m is the first decoding block in the first access block for a buffering period that does not initialize HRD, the nominal time of removing the decoding block from the CPB is set according to t _r,n (m) = t _r,n (m _b ) + T _c * CpbRemovalDelay(m), where t _r,n (m _b ) is the nominal time of removing the first decoding block for the previous buffering period.

[0125] When decoding block m is the first decoding block in the buffering period, m _b is set equal to m at the time of deleting t _r,n (m) of decoding block m.

[0126] The nominal time t _r,n (m) for removing a decoding block m that is not the first decoding block in the buffering period is given by t _r,n (m)=t _r,n (m _b )+T _c *CpbRemovalDelay(m), where t _r,n (m _b ) is the nominal time for removing the first decoding block for the current buffering period.

[0127] The decoding block m removal time is specified as follows. The variable ClockDiff is defined as ClockDiff=(num_units_in_tick-(num_units_in_sub_tick*(num_decoding_units_minus1+1))/time_scale). In some case, there may be a bitstream compliance requirement that the parameters num_units_in_tick, num_units_in_sub_tick, num_decoding_units_minus1 signal such that the following equation is satisfied

(num_units_in_tick-(num_units_in_sub_tick*(num_decoding_units_minus1+1)))>=0

[0128] In some other case, there may be a bitstream conformance requirement that the parameters num_units_in_tick, num_units_in_sub_tick, num_decoding_units_minus1 may be signaled such that the following equation is satisfied (num_units_in_tick-(num_units_in_sub_tick*(num_decoding_units_minus1+1)))<=0 If low_delay_hrd_flag is 0 or t _r,n (m)>=t _af (m), the removal time of decoding block m is given by t _r (m)=t _r,n (m).

[0129] Otherwise (low_delay_hrd_flag is 1 and t _r,n (m) < t _af (m)), and when sub_pic_cpb_params_present_flag is 1 and CPB operates at the sub-picture level, and if ClockDiff is greater than zero, the removal time for decoding block m when it is the last decoding block of access block n is set according to t _r (m) = t _r,n (m) + T _c * Ceil ( (t _af (m) - _{t r,n} (m)) / T _c ) + ClockDiff.

[0130] Otherwise (low_delay_hrd_flag is 1 and t _r,n (m) < t _af (m)), and when sub_pic_cpb_params_present_flag is 1 and CPB operates at the access block level, and if ClockDiff is less than zero, the removal time of access block n is given by t _r (m) = t _r,n (m) + t _c * Ceil ((t _af (m) - t _r,n (m)) / t _c ) - ClockDiff.

[0131] Otherwise (low_delay_hrd_flag is 1 and t _r,n (m) < t _af (m)), the deletion time of decoding block m is set according to t _r (m) = t _r,n (m) + T _c * Ceil ((t _af (m) - t _r,n (m)) / T _c ). The latter case (low_delay_hrd_flag is 1 and t _r,n (m) < t _af (m)) means that the size of decoding block m, b(m), is large enough that it prevents deletion within the nominal deletion time.

[0132] Otherwise (low_delay_hrd_flag is 1 and t _r,n (m) < t _af (m)) and when the picture synchronization flag is set to 1 and CPB operates at the sub-picture level, the removal time for the last decoding block m in an access block, t _r (m), is given by: t _r (m) = t _r,n (m) + min ( (t _{c_sub} * Ceil ( (t _af (m) - _{t r,n} (m)) / t _{c_sub} )), (t _c * Ceil ( (t _af (n) - _{t r,n} (n)) / t _c ))), where t _r,n (m) is the nominal removal time of the last decoding block m, t _{c_sub} is the clock for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of the access block n, t _c - the system clock cycle and t _af (n) - the final time of arrival of the nth access block.

[0133] Otherwise (low_delay_hrd_flag is 1 and t _r,n (n) < t _af (n)) and when the picture synchronization flag is set to 1 and the CPB operates at the access block level, the removal time for access block n, t _r (n), is given by: t _r (n) = t _r,n (n) + min ( (t _{c_sub} * Ceil ( (t _af (m) - _{t r,n} (m)) / t _{c_sub} )), (t _c * Ceil ( (t _af (n) - _{t r,n} (n)) / t _c ))), where t _r,n (m) is the nominal removal time of the last decoding block n, t _{c_sub} is the clock tick for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of access block n, t _c is the clock tick of the system clock, and t _af (n) - final time of arrival of access block n.

[0134] Otherwise (low_delay_hrd_flag is 1 and t _r,n (m) < t _af (m)) and the picture synchronization flag is set to 1, and CPB operates at the sub-picture level, the removal time for the last decoding block m in an access block, t _r (m), is given by: t _r (m) = t _r,n (m) + (t _c * Ceil ((n) - _{t r,n} (n)) / t _c )), where t _r,n (m) is the nominal removal time of the last decoding block m, t _{c_sub} is the clock cycle for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of access block n, t _c is the clock cycle, and t _af (n) is the final arrival time of access block n.

[0135] Otherwise (low_delay_hrd_flag is 1 and t _r,n (n) < t _af (n)) and the picture synchronization flag is set to 1, and the CPB operates at the access block level, the removal time for access block n, t _r (n), is given by: t _r (n) = t _r,n (n) + (t _c * Ceil ((t _af (n) - _{t r,n} (n)) / t _c )), where t _r,n (m) is the nominal removal time of the last decoding block n, t _{c_sub} is the clock cycle for the sub-picture, Ceil() is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of access block n, t _c is the clock cycle, and t _af (n) is the final arrival time of access block n.

[0136] Otherwise (low_delay_hrd_flag is 1 and t _r,n (m) < t _af (m)) and the picture synchronization flag is set to 1 and the CPB operates at the sub-picture level, the removal time for a decoding block that is not the last decoding block in an access block is set to t _r (m) = t _af (m), where t _af (m) is the final arrival time of the mth decoding block and the removal time of the last mth decoding block in an access block, t _r (m) is set according to: t _r (m) = t _r,n (m) + (t _{c_sub} * Ceil ((t _af (m) - _{t r,n} (m)) / t _{c_sub} )), where t _r,n (m) is the nominal removal time of the last mth decoding block, t _{c_sub} is the clock for a sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last mth block decoding, t _r,n (n) is the nominal time of removal of access block n, t _c is the system clock tick, and t _af (n) is the final time of arrival of access block n, and t _af (m) is the final time of arrival of the last decoding block m in access block n.

[0137] Otherwise (low_delay_hrd_flag is 1 and t _r,n (m) < t _af (m)) and the picture synchronization flag is set to 1 and CPB operates at the sub-picture level, the removal time for a decoding block that is not the last decoding block for an access block is set as t _r (m) = t _af (m), where t _af (m) is the final arrival time of decoding block m and the removal time of the last decoding block m in the access block, t _r (m) is set according to: t _r (m) = t _r,n (m) + (t _c * Ceil ((t _af (m) - _{t r,n} (m)) / t _c )), where t _r,n (m) is the nominal removal time of the last decoding block m, t _{c_sub} is the clock for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal time of removal of access block n, t _c is the system clock tick, and t _af (n) is the final arrival time of access block n, and t _af (m) is the final arrival time of the last decoding block m in access block n.

[0138] Otherwise (low_delay_hrd_flag is 1 and t _r,n (m) < t _af (m)) and the picture synchronization flag is set to 1, and CPB operates at the sub-picture level, the removal time for the decoding block is set as t _r (m) = t _af (m), where t _r,n (m) is the nominal removal time of decoding block m, t _{c_sub} is the clock cycle for the sub-picture, Ceil() is a rounding function _, t _af (m) is the final arrival time of decoding block m, t _r,n (n) is the nominal removal time of access block n, t _c is the clock cycle, and t _af (n) is the final arrival time of access block n in access block n.

[0139] Otherwise (low_delay_hrd_flag is 1, and t _r,n (n) < t _af (n)) and the picture synchronization flag is set to 1, and the CPB operates at the access block level, the removal time for access block n, t _r (n), is given by: t _r (n) = t _af (n), where t _r,n (m) is the nominal removal time of the last decoding block n, t _{c_sub} is the clock tick for the picture sub-frame, Ceil() is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of access block n, t _c is the clock tick, and t _af (n) is the final arrival time of access block n.

[0140] When SubPicCpbFlag is 1, the nominal removal time in the CPB of access block n, t _r,n (n), is set to the nominal removal time in the CPB of the last decoding block in access block n, the removal time in the CPB of access block n, t _r (n) is set to the removal time in the CPB of the last decoding block in access block n.

[0141] When SubPicCpbFlag is 0, each decoding block is an access block, therefore, the nominal CPB removal time and the CPB removal time for access block n are the nominal CPB removal time and the CPB removal time for decoding block n. At the CPB removal time of decoding block m, the decoding block is immediately decoded.

[0142] As illustrated by the foregoing, the systems and methods disclosed in the description provide syntax and semantics that modify bitstreams of a picture synchronization SEI message carrying parameters based on a portion of a picture. In some configurations, the systems and methods disclosed in the description may be applied to High Efficiency Video Coding (HEVC) standard specifications.

[0143] For convenience, several definitions are as follows, which may be applied to the systems and methods disclosed in the description. A random access point may be any point in a data stream (e.g., a bitstream), where decoding the bitstream does not require access to any point in the bitstream preceding the random access point in order to decode the current picture and all pictures following said current picture in output order.

[0144] The buffering period may be defined as a set of access units between two instances of the buffering period SEI message in decoding order. Supplemental extended information (SEI) may contain information that is not necessary to decode coded picture samples from the VCL NAL units. SEI messages may assist in procedures related to decoding, display, or other purposes. Compliant decoders may not be required to process this information for output order compliance with the HEVC specifications (Annex C of the HEVC specifications (JCTVC-I1003) includes specifications for compliance, for example). Some SEI message information may be used to verify bitstream compliance and for decoder output timing compliance.

[0145] The buffering period SEI message may be an SEI message associated with a buffering period. The picture synchronization SEI message may be an SEI message associated with the CPB removal synchronization. These messages may define syntax and semantics that specify the bitstream arrival synchronization and the coded picture removal synchronization.

[0146] The coded picture buffer (CPB) may be a first-in-first-out buffer containing access units in decoding order specified in a hypothetical reference decoder (HRD). An access unit may be a set of network abstraction layer (NAL) units that are sequential in decoding order and contain exactly one coded picture. In addition to the NAL units of coded slices in the coded picture, the access unit may also contain other NAL units that do not contain coded picture slices. Decoding an access unit always results in a decoded picture. A NAL unit may be a syntactic structure containing an indicator of the type of data to follow and bytes containing this data in the form of a raw byte sequence payload, with emulation prevention bytes inserted as needed.

[0147] As used in the description, the term "common" generally refers to a syntax element or variable that is applicable to more than one entity. For example, in the context of syntax elements in a picture synchronization SEI message, the term "common" may mean that the syntax element (e.g., common_du_cpb_removal_delay) is applicable to all decoding units in an access unit associated with the picture synchronization SEI message. In addition, data units that are described in terms of "n" and "m" generally refer to access units and decoding units, respectively.

[0148] Various configurations are now described with reference to the figures of the drawings, wherein like reference numerals may indicate functionally similar elements. The systems and methods, as generally described and illustrated in the figures of the drawings in the description, can be arranged and designed in a wide variety of different configurations. Thus, the following more detailed description of several configurations as shown in the figures of the drawings is not intended to limit the scope as claimed in the claims, but merely to illustrate the systems and methods.

[0149] Figure 1A is a block diagram illustrating an example of one or more electronic devices 102 in which systems and methods for sending a message and buffering a bit stream may be implemented. In this example, electronic device A 102a and electronic device B 102b are illustrated. However, it should be noted that one or more of the features and functionality described with respect to electronic device A 102a and electronic device B 102b may be combined into a single electronic device in some configurations.

[0150] The electronic device A 102a includes an encoder 104. The encoder 104 includes a message generation unit 108. Each of the components included in the electronic device A 102a (e.g., the encoder 104 and the message generation unit 108) may be implemented in hardware, software, or a combination thereof.

[0151] The electronic device A 102a may receive one or more input images 106. In some configurations, the input image(s) 106 may be obtained by recording on the electronic device A 102a using an image input sensor, may be retrieved from memory, and/or may be received from another electronic device.

[0152] Encoder 104 may encode input image(s) 106 to create encoded data. For example, encoder 104 may encode a series of input images 106 (e.g., video). In one configuration, encoder 104 may be an HEVC encoder. The encoded data may be digital data (e.g., part of bitstream 114). Encoder 104 may generate service signaling based on the input signal.

[0153] The message generation unit 108 may generate one or more messages. For example, the message generation unit 108 may generate one or more SEI messages or other messages. With respect to the CPB that supports operation at the sub-picture level, the electronic device 102 may send sub-picture parameters (for example, a removal delay parameter in the CPB). Specifically, the electronic device 102 (for example, the encoder 104) may determine whether to include a common removal delay parameter in the CPB of the decoding unit in the SEI message about the picture synchronization. For example, the electronic device may set a flag (for example, common_du_cpb_removal_delay_flag) to one when the encoder 104 includes a common removal delay parameter in the CPB of the decoding unit (for example, common_du_cpb_removal_delay) in the SEI message about the picture synchronization. When the common deletion delay parameter in the CPB of the decoding unit is enabled, the electronic device can form a common deletion delay parameter in the CPB of the decoding unit that is applicable to all decoding units in the access unit. In other words, instead of enabling the deletion delay parameter in the CPB of the decoding unit for each decoding unit in the access unit, the common parameter can be applied to all decoding units in the access unit with which the SEI message about image synchronization is associated.

[0154] In contrast, when the common parameter of the deletion delay in the CPB of the decoding unit is not to be included in the SEI message about the picture synchronization, the electronic device 102 can generate a separate deletion delay CPB of the decoding unit for each decoding unit in the access unit with which the SEI message about the picture synchronization is associated. The message generating unit 108 can perform one or more of the procedures described in connection with Fig. 2 and Fig. 3 below.

[0155] In some configurations, the electronic device A 102a may send a message to the electronic device B 102b as part of the bitstream 114. In some configurations, the electronic device A 102a may send a message to the electronic device B 102b in a separate transmission 110. For example, the separate transmission may not be part of the bitstream 114. For example, the image synchronization SEI message or another message may be sent using some out-of-band mechanism. It should be noted that in some configurations, the other message may include one or more characteristics of the image synchronization SEI message described above. In addition, the other message, in one or more aspects, may be used similarly to the SEI message described above.

[0156] The encoder 104 (and the message generation module 108, for example) can create a bitstream 114. The bitstream 114 can include coded image data based on the input image(s) 106. In some configurations, the bitstream 114 can also include service data, such as a picture synchronization SEI message or another message, a slice header(s), a picture parameter set (PPS), etc. Since the additional input images 106 are coded, the bitstream 114 can include one or more coded images. For example, the bitstream 114 can include one or more coded images with corresponding service data (e.g., a picture synchronization SEI message or another message).

[0157] The bitstream 114 may be provided to the decoder 112. In one example, the bitstream 114 may be transmitted to the electronic device B 102b using a wired or wireless communication link. In some cases, this may be done over a network, such as the Internet or a Local Area Network (LAN). As illustrated in Fig. 1A, the decoder 112 may be implemented on the electronic device B 102b separately from the encoder 104 on the electronic device A 102a. However, it should be noted that the encoder 104 and the decoder 112 may be implemented on the same electronic device in some configurations. In an implementation where the encoder 104 and the decoder 112 are implemented on the same electronic device, for example, the bitstream 114 may be provided over a bus to the decoder 112 or stored in memory for retrieval by the decoder 112.

[0158] The decoder 112 may be implemented in hardware, software, or a combination thereof. In one configuration, the decoder 112 may be a HEVC decoder. The decoder 112 may receive (e.g., obtain) the bitstream 114. The decoder 112 may generate one or more decoded images 118 based on the bitstream 114. The decoded image(s) 118 may be displayed, played, stored in memory, and/or transmitted to another device, etc.

[0159] The decoder 112 may include a CPB 120. The CPB 120 may temporarily store coded pictures. The CPB 120 may use parameters detected in the SEI picture synchronization message to determine when to discard data. When the CPB 120 supports operation at the subpicture level, individual decoding blocks may be discarded rather than all access blocks at once. The decoder 112 may include a decoded picture buffer (DPB) 122. Each decoded picture is placed in the DPB 122 for reference during the decoding process, as well as for output and cropping. A decoded picture is removed from the DPB later than the time the DPB is output or the time it is no longer needed for reference in inter-picture prediction.

[0160] The decoder 112 may receive a message (e.g., a picture synchronization SEI message or another message). The decoder 112 may also determine whether the received message includes a common removal delay parameter in the decoding unit CPB (e.g., common_du_cpb_removal_delay). This may include an identification of a flag (e.g., common_du_cpb_removal_delay_flag) that is set when the common parameter is present in the picture synchronization SEI message. If the common parameter is present, the decoder 112 may determine the common removal delay parameter in the decoding unit CPB that applies to all decoding units in the access unit. If the common parameter is not present, the decoder 112 may determine a separate removal delay parameter in the decoding unit CPB for each decoding unit in the access unit. Decoder 112 may also remove decoding blocks from CPB 120 using either a common removal delay parameter in the decoding block CPB or individual removal delay parameters in the decoding block CPB. CPB 120 may perform one or more of the procedures described in connection with Fig. 4 and Fig. 5 below.

[0161] The HRD described above may be one example of the decoder 112 illustrated in Fig. 1A. Thus, the electronic device 102 may operate in accordance with the HRD and the CPB 120 and DPB 122 described above in some configurations.

[0162] It should be noted that one or more elements or parts thereof included in the electronic device(s) 102 may be implemented in hardware. For example, one or more of these elements or parts thereof may be implemented in the form of a microcircuit, circuit components or hardware, etc. It should also be noted that one or more of the functions or methods described herein may be implemented in hardware and/or performed using them. For example, one or more of the methods described herein may be implemented within and/or implemented using a microprocessor set, an application-specific integrated circuit (ASIC), a large-scale integrated circuit (LSI), or an integrated circuit, etc.

[0163] Figure 1B is a block diagram illustrating another example of an encoder 1908 and a decoder 1972. In this example, electronic device A 1902 and electronic device B 1970 are illustrated. However, it should be noted that the features and functionality described with respect to electronic device A 1902 and electronic device B 1970 may be combined into a single electronic device in some configurations.

[0164] Electronic device A 1902 includes an encoder 1908. Encoder 1908 may include base layer encoder 1910 and enhancement layer encoder 1920. Video encoder 1908 is suitable for scalable video coding and multi-view (multi-aspect) video coding, as described below. Encoder 1908 may be implemented as hardware, software, or a combination of both. In one configuration, encoder 1908 may be a high efficiency video coding (HEVC) encoder, including scalable and/or multi-view. Other encoders may be used in a similar manner. Electronic device A 1902 may receive source video 1906. In some configurations, source video 1906 may be obtained by recording on electronic device A 1902 using an image input sensor, retrieved from memory, or received from another electronic device.

[0165] Encoder 1908 may encode source video 1906 to create base layer bitstream 1934 and enhancement layer bitstream 1936. For example, encoder 1908 may encode a series of images (e.g., video) in source video 1906. In particular, for scalable video coding for SNR scalability, also known as quality scalability, the same source video 1906 may be provided to the base layer and enhancement layer encoder. In particular, for scalable video coding for spatial scalability, subsampled source video may be used for the base layer encoder. In particular, for multi-view coding, source video of another view may be used for the base layer encoder and the enhancement layer encoder. Encoder 1908 may be similar to encoder 1782, described below in connection with Fig. 6B.

[0166] The bitstreams 1934, 1936 may include coded picture data based on the source video 1906. In some configurations, the bitstreams 1934, 1936 may also include service data such as slice header information, PPS information, etc. Since additional pictures in the source video 1906 are coded, the bitstreams 1934, 1936 may include one or more coded pictures.

[0167] The bitstreams 1934, 1936 may be provided to the decoder 1972. The decoder 1972 may include the base layer decoder 1980 and the enhancement layer decoder 1990. The video decoder 1972 is suitable for scalable video decoding and multi-view video decoding. In one example, the bitstreams 1934, 1936 may be transmitted to the electronic device B 1970 using a wired or wireless communication link. In some cases, this may be done over a network, such as the Internet or a Local Area Network (LAN). As illustrated in Fig. 1B, the decoder 1972 may be implemented on the electronic device B 1970 separately from the encoder 1908 on the electronic device A 1902. However, it should be noted that in some configurations, the encoder 1908 and the decoder 1972 may be implemented on the same electronic device. In an embodiment where the encoder 1908 and the decoder 1972 are implemented on the same electronic device, for example, the bit streams 1934, 1936 may be provided over a bus to the decoder 1972 or stored in memory for retrieval by the decoder 1972. The decoder 1972 may provide image(s) of the decoded base layer 1992 and the decoded enhancement layer 1994 as an output.

[0168] Decoder 1972 may be implemented as hardware, software, or a combination of both. In one configuration, decoder 1972 may be a high efficiency video coding (HEVC) decoder, including scalable and/or multi-view. Other decoders may be used in a similar manner. Decoder 1972 may be similar to decoder 1812, described further in connection with Fig. 7B. In addition, the base layer encoder and/or the enhancement layer encoder may each include a message generation module, such as described in connection with Fig. 1A. In addition, the base layer decoder and/or the enhancement layer decoder may include a coded picture buffer and/or a decoded picture buffer, such as described in connection with Fig. 1A. In addition, the electronic devices of Fig. 1B may operate in accordance with the functions of the electronic devices of Fig. 1A, depending on the conditions.

[0169] Figure Fig. 2 is a flow chart illustrating one configuration of a method 200 for sending a message. The method 200 can be performed by the encoder 104 or one of its subsystems (for example, the message generation unit 108). The encoder 104 can determine 202 a picture synchronization flag (for example, sub_pic_cpb_params_present_flag), which indicates whether the CPB 120 supports operation at the sub-picture level. For example, when the picture synchronization flag is set to 1, the CPB 120 can operate at the access block level or the sub-picture level. It should be noted that, even when the picture synchronization flag is set to 1, the decision whether to actually operate at the sub-picture level is left to the decoder 112 itself.

[0170] The encoder 104 may also determine 204 one or more removal delays for the decoding units in the access unit. For example, the encoder 104 may determine a single common removal delay parameter in the CPB of the decoding unit (e.g., common_du_cpb_removal_delay), which is applicable to all decoding units in the access unit, from the CPB 120. Alternatively, the encoder 104 may determine a separate removal delay of the CPB of the decoding unit (e.g., du_cpb_removal_delay[i]) for each decoding unit in the access unit.

[0171] The encoder 104 may also determine 206 one or more NAL parameters that indicate the number, with an offset of one, of NAL units in each decoding unit in the access point. For example, the encoder 104 may determine a single common NAL parameter (e.g., common_num_nalus_in_du_minus1) that is applicable to all decoding units in the access unit from the side of the CPB 120. Alternatively, the encoder 104 may determine a separate CPB removal delay of a decoding unit (e.g., num_nalus_in_du_minus1[i]) for each decoding unit in the access unit.

[0172] The encoder 104 may also send 208 a picture synchronization SEI message that includes a picture synchronization flag, removal delays, and NAL parameters. The picture synchronization SEI message may also include other parameters (e.g., cpb_removal_delay, dpb_output_delay, etc.). For example, the electronic device 102 may transmit the message via one or more of a wireless transmission, a wired transmission, a device bus, a network, etc. For example, the electronic device A 102a may transmit the message to the electronic device B 102b. The message may be part of the bitstream 114, for example. In some configurations, the electronic device A 102a may send 208 the message to the electronic device B 102b in a separate transmission 110 (which is not part of the bitstream 114).

[0173] For example, the message can be sent using some out-of-band mechanism. In some case, the information indicated in 204, 206 can be sent in an SEI message other than the image synchronization SEI message. In another case, the information indicated in 204, 206 can be sent in a parameter set, such as a video parameter set and/or a sequence parameter set and/or a picture parameter set and/or an adaptation parameter set and/or a slice header.

[0174] Figure 3 is a flow chart illustrating one configuration of a method 300 for determining one or more removal delays for decoding units in an access unit. In other words, the method 300 illustrated in Figure 3 may further illustrate step 204 in the method 200 illustrated in Figure 2. The method 300 may be performed by an encoder 104. The encoder 104 may determine 302 whether to include a common removal delay parameter in the CPB of a decoding unit (e.g., common_du_cpb_removal_delay).

[0175] This may include determining whether a common flag for removing delay from the CPB of a decoding block (e.g., common_du_cpb_removal_delay_flag) is set. The encoder 104 may send this common parameter in the event that decoding blocks are removed from the CPB at regular intervals. This may be the case, for example, when each decoding block corresponds to a certain number of lines of an image or has some other regular structure.

[0176] For example, the common flag of the removal delay in the CPB of the decoding unit may be set to 1 when the common parameter of the removal delay in the CPB of the decoding unit is to be included in the SEI message about the picture synchronization, and to 0 when it is not to be included. If "yes" (e.g., the flag is set to 1), the encoder 104 may determine 304 a common parameter of the removal delay in the CPB of the decoding unit (e.g., common_du_cpb_removal_delay), which is applicable to all decoding units in the access unit. If "no" (e.g., the flag is set to 0), the encoder 104 may determine 306 separate parameters of the removal delay in the CPB of the decoding unit (e.g., du_cpb_removal_delay[i]) for each decoding unit in the access unit.

[0177] If the general parameter of the deletion delay in the CPB of the decoding block is present in the SEI message of the picture synchronization, it may specify the number of clocks of the image fragment to wait after the deletion from the CPB 120 of the immediately preceding decoding block until the deletion from the CPB 120 of the current decoding block in the access block associated with the SEI message of the picture synchronization.

[0178] For example, when the decoding unit is the first decoding unit in the access unit, the common deletion delay parameter in the CPB 120 of the decoding unit may specify the number of clock cycles of a sub-picture to wait after the last decoding unit in the access unit associated with the most recent SEI message about the buffering period in the previous access unit is deleted from the CPB 120 until the first decoding unit in the access unit associated with the SEI message about the picture synchronization is deleted from the CPB 120.

[0179] When the decoding block is not the first decoding block in the access block, the common deletion delay parameter in the CPB of the decoding block may indicate the number of clock cycles of a sub-picture to wait after deleting from the CPB 120 the previous decoding block in the access block associated with the SEI message about the picture synchronization, before deleting from the CPB the current decoding block in the access block associated with the SEI message about the picture synchronization.

[0180] In contrast, when a common decoding block CPB removal delay parameter (e.g., common_du_cpb_removal_delay) is not sent in the picture synchronization SEI message, individual decoding block CPB removal delay parameters (e.g., du_cpb_removal_delay[i]) may be included in the picture synchronization SEI message for each decoding block in the access unit. The decoding block CPB removal delay parameters (e.g., du_cpb_removal_delay[i]) may specify the number of image fragment clocks to wait after the last decoding block is removed from CPB 120 until the i-th decoding block is removed from CPB 120 in the access unit associated with the picture synchronization SEI message. The decoding block CPB removal delay parameters may be calculated based on the value of the remainder of the counter modulo , where cpb_removal_delay_length_minus1+1 is the length of the common removal delay parameter in the CPB of the decoding block.

[0181] Figure Fig. 4 is a flow chart illustrating one configuration of a method 400 for buffering a bitstream. The method 400 can be performed by a decoder 112 in the electronic device 102 (e.g., the electronic device B 102b), which can receive a message (e.g., an SEI message about image synchronization or another message) at 402. For example, the electronic device 102 can receive the message at 402 via one or more of a wireless transmission, a wired transmission, a device bus, a network, etc. For example, the electronic device B 102b can receive a message from the electronic device A 102a at 402. The message can be part of the bitstream 114, for example. In another example, the electronic device B 102b can receive the message from the electronic device A 102a in a separate transmission 110 (which is not part of the bitstream 114, for example). For example, the SEI message about image synchronization can be received using some out-of-band mechanism. In some configurations, the message may include one or more picture synchronization flags, one or more removal delays for decoding units in the access unit, and one or more NAL parameters. Thus, receiving the message at step 402 may include receiving one or more of a picture synchronization flag, one or more removal delays for decoding units in the access unit, and one or more NAL parameters.

[0182] The decoder 112 may determine at step 404 whether the CPB 120 operates at the access unit level or at the sub-picture level. For example, the decoder 112 may decide to operate on a sub-picture basis if it desires to achieve low delay. Alternatively, the decision may be based on whether the decoder 112 has sufficient resources to support sub-picture based operation. If the CPB 120 operates at the sub-picture level, the decoder may determine at step 406 one or more removal delays for the decoding units in the access unit. For example, the decoder 112 may determine a single common removal delay parameter in the CPB of the decoding unit (e.g., common_du_cpb_removal_delay), which is applicable to all decoding units in the access unit. Alternatively, the decoder 112 may determine a separate CPB removal delay of the decoding unit (e.g., du_cpb_removal_delay[i]) for each decoding unit in the access unit. In other words, the SEI message about the picture synchronization may include a common parameter applicable to all decoding units in the access unit, or separate parameters for each decoding unit.

[0183] The decoder 112 may also remove decoding blocks at step 408 based on removal delays for the decoding blocks, that is, using either a common parameter applicable to all decoding blocks in the access block, or separate parameters for each decoding block. The decoder 112 may also decode decoding blocks at step 410.

[0184] The decoder 112 may use the variable ClockDiff when determining the erasure time determined from various signaled parameters. Specifically, ClockDiff may be determined according to ClockDiff=(num_units_in_tick-(num_units_in_sub_tick*(num_decoding_units_minus1+1))/time_scale), where num_units_in_tick is the number of ticks of the clock generator operating at a frequency of time_scale Hz that corresponds to one increment of the tick counter of the system clock, num_units_in_sub_tick is the number of ticks of the clock generator operating at a frequency of time_scale Hz that corresponds to one increment of the tick counter for the image fragment, num_decoding_units_minus1+1 is the number of decoding units in the access unit, and time_scale is the number of ticks that occur in one second.

[0185] When the hypothetical low delay reference decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, the CPB operates at the sub-picture level, and ClockDiff is greater than zero, the removal time for decoding block m, t _r (m), is determined according to: t _r (m) = t _r,n (m) + t _{c_sub} * Ceil ((t _af (m) - _{t r,n} (m)) / t _{c_sub} ) + ClockDiff, where t _r,n (m) is the nominal removal time of decoding block m, t _{c_sub} is the clock for the sub-picture, Ceil () is a rounding function, and t _af (m) is the final arrival time of decoding block m.

[0186] When the hypothetical low delay reference decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (n) < t _af (n), the picture synchronization flag is set to 1, the CPB operates at the access block level, and ClockDiff is greater than zero, the removal time for access block n, t _r (n), is determined according to: t _r (n) = t _r,n (n) + t _c * Ceil ((t _af (n) - _{t r,n} (n)) / t _c ) - ClockDiff, where t _r,n (n) is the nominal removal time of access block n, t _c is the system clock tick, Ceil () is a rounding function, and t _af (n) is the final arrival time of access block n.

[0187] When the Hypothetical Low Delay Reference Decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the sub-picture level, the removal time for the last decoding block m in an access block, t _r (m), is: t _r (m) = t _r,n (m) + max ( ( t _{c_sub} * Ceil ( ( t _af (m) - _{t r,n} (m)) / t _{c_sub} )), ( t _c * Ceil ( ( t _af (n) - _{t r,n} (n)) / t _c ))), where t _r,n (m) is the nominal removal time of the last decoding block m, t _{c_sub} is the clock for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal time of removal of access block n, t _c is the system clock tick, and t _af (n) is the final time of arrival of access block n.

[0188] When the Hypothetical Reference Decoder (HRD) Low Delay Flag is set to 1, t _r,n (n) < t _af (n), the Picture Sync Flag is set to 1, and the CPB operates at the access block level, the removal time for access block n, t _r (n), is set according to: t _r (n) = t _r,n (n) + max ( (t _{c_sub} * Ceil ( (t _af (m) - _{t r,n} (m)) / t _{c_sub} )), (t _c * Ceil ( (t _af (n) - t _r,n (n)) / t _c ))), where t _r,n (m) is the nominal removal time of the last decoding block n, t _{c_sub} is the clock for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of access block n, t _c - the system clock cycle, and tr _af (n) - the final time of arrival of the nth access block.

[0189] When the Hypothetical Reference Decoder (HRD) Low Delay Flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the Picture Sync Flag is set to 1, and the CPB operates at the sub-picture level, the removal time for the last decoding block m in an access block, t _r (m), is set according to: t _r (n) = t _r,n (n) + min ( ( t _{c_sub} * Ceil ( ( t _af (m) - _{t r,n} (m)) / t _{c_sub} )), ( t _c * Ceil ( ( t _af (n) - _{t r,n} (n)) / t _c ))), where t _r,n (m) is the nominal removal time of the last decoding block m, t _{c_sub} is the clock for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last block m decoding, t _r,n (n) is the nominal time of removal of access block n, t _c is the system clock tick, and t _af (n) is the final time of arrival of access block n.

[0190] When the Hypothetical Reference Decoder (HRD) flag is set to 1, t _r,n (n) < t _af (n), the picture synchronization flag is set to 1, and the CPB operates at the access block level, the removal time for access block n, tr (n), is set according to: t _r (n) = t _r,n (n) + min ( (t _{c_sub} * Ceil ( (t _af (m) - _{t r,n} (m) / t _{c_sub} )), (t _c * Ceil ( (t _af (n) - _{t r,n} (n)) / t _c ))), where t _r,n (m) is the nominal removal time of the last decoding block n, t _{c_sub} is the clock for the sub-picture, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of access block n, t _c is is the system clock cycle, and t _af (n) is the final time of arrival of the nth access block.

[0191] When the hypothetical low delay reference decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the sub-picture level, the removal time for the last decoding block m in an access block, t _r (m), is defined according to: t _r (m) = t _r,n (m) + (t _c * Ceil ((t _af (n) - _{t r,n} (n)) / t _c )), where t _r,n (m) is the nominal removal time of the last decoding block m, t _{c_sub} is the clock cycle for the sub-picture, Ceil () is a rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of the access block n, t _c is the clock cycle, and t _af (n) - final time of arrival of access block n.

[0192] When the hypothetical reference decoder (HRD) low delay flag is set to 1, t _r,n (n) < t _af (n), the picture synchronization flag is set to 1, and the CPB operates at the access block level, the removal time for access block n, t _r (n), is set according to: t _r (n) = t _r,n (n) + (t _c * Ceil ((t _af (n) - _{t r,n} (n)) / t _c )), where t _r,n (m) is a nominal removal time of the last decoding block n, t _{c_sub} is a clock cycle for a sub-picture, Ceil () is a rounding function, t _af (m) is a final arrival time of the last decoding block m, t _r,n (n) is a nominal removal time of access block n, t _c is a clock cycle, and t _af (n) is a final arrival time of access block n.

[0193] When the low delay hypothetical reference decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the subpicture level, the removal time for a decoding block that is not the last decoding block in an access block is set to t _r (m) = t _af (m), where t _af (m) is the final arrival time of decoding block m. And the removal time for the last decoding block m in an access block, t _r (m), is given by: t _r (m) = t _r,n (m) + (t _{c_sub} * Ceil ((t _af (m) - _{t r,n} (m)) / t _{c_sub} )), where t _r,n (m) is the nominal removal time of the last decoding block m, t _{c_sub} is the clock cycle for the image fragment, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of the access block n, t _c is the clock cycle of the system clock; t _af (n) is the final arrival time of the access block n, and t _af (m) is the final arrival time of the last decoding block in the access block n.

[0194] When the low delay hypothetical reference decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the subpicture level, the removal time for a decoding block that is not the last decoding block in an access block is set to t _r (m) = t _af (m), where t _af (m) is the final arrival time of decoding block m. And the removal time for the last decoding block m in an access block, t _r (m), is given by: t _r (m) = t _r,n (m) + (t _c * Ceil ((t _af (m) - _{t r,n} (m)) / t _c )), where t _r,n (m) is the nominal removal time of the last decoding block m, t _{c_sub} is the clock cycle for the image fragment, Ceil () is the rounding function, t _af (m) is the final arrival time of the last decoding block m, t _r,n (n) is the nominal removal time of the access block n, t _c is the clock cycle of the system clock; t _af (n) is the final arrival time of the access block n, and t _af (m) is the final arrival time of the last decoding block in the access block n.

[0195] When the hypothetical low delay reference decoder (HRD) flag (e.g., low_delay_hrd_flag) is set to 1, t _r,n (m) < t _af (m), the picture synchronization flag is set to 1, and the CPB operates at the sub-picture level, the removal time for a decoding block is set as t _r (m) = t _af (m), where t _r,n (m) is a nominal removal time of decoding block m, t _{c_sub} is a clock cycle for a sub-picture, Ceil() is a rounding function, t _af (m) is a final arrival time of decoding block m, t _r,n (n) is a nominal removal time of access block n, t _c is a clock cycle; t _af (n) is a final arrival time of access block n, and t _af (m) is a final arrival time of a decoding block in access block n.

[0196] When the hypothetical reference decoder (HRD) low delay flag is set to 1, t _r,n (n) < t _af (n), the picture synchronization flag is set to 1, and the CPB operates at the access block level, the removal time for access block n, t _r (n), is set according to: t _r (n) = t _af (n), where t _r,n (m) is a nominal removal time of the last decoding block n, t _{c_sub} is a clock cycle for a sub-picture, Ceil() is a rounding function, t _af (m) is a final arrival time of the last decoding block m, t _r,n (n) is a nominal removal time of access block n, t _c is a clock cycle, and t _af (n) is a final arrival time of access block n.

[0197] If the CPB operates at the access block level, the decoder 112 may determine a removal delay parameter in the CPB at step 412. This may be included in the received SEI message about picture synchronization (e.g., cpb_removal_delay). The decoder 112 may also remove an access block at step 414 based on the removal delay parameter in the CPB and decode the access block at step 416. In other words, the decoder 112 may decode all access blocks at once, instead of decoding blocks in an access block.

[0198] Figure 5 is a flow chart illustrating one configuration of a method 500 for determining one or more removal delays for decoding units in an access unit. In other words, the method 500 illustrated in Figure 5 may further illustrate step 406 in the method 400 illustrated in Figure 4. The method 500 may be performed by a decoder 112. The decoder 112 may determine 502 whether a received SEI message about picture synchronization includes a common removal delay parameter in the CPB of the decoding unit. This may include determining whether a common removal delay flag from the CPB of the decoding unit (e.g., common_du_cpb_removal_delay_flag) is set. If "yes", the decoder 112 may determine 504 a common removal delay parameter in the CPB of the decoding unit (e.g., common_du_cpb_removal_delay), which is applicable to all decoding units in the access unit. If "no", the decoder 112 may determine 506 individual removal delay parameters in the CPB of the decoding unit (e.g., du_cpb_removal_delay[i]) for each decoding unit in the access unit.

[0199] In addition to modifying the semantics of the SEI message about picture synchronization, the present systems and methods may also impose a constraint on the bitstream such that the operation of the CPB based on a picture fragment and the operation of the CPB based on an access block result in the same decoding block removal timing. Specifically, when the picture synchronization flag (e.g., sub_pic_cpb_params_present_flag) is set to 1, the removal delay in the CPB may be set according to

[0200] [Mathematical expression 6]

[0201] where du_cpb_removal_delay[i] are the removal delay parameters in the CPB of the decoding block, t _c is the clock tick, t _{c_sub} is the tick for the image fragment, num_decoding_units_minus1 is the number of decoding units in the access block offset by one, and i is the index.

[0202] Alternatively, the CPB removal delay may be set as follows: Let the variable T _du (k) be defined as:

[0203] [Mathematical expression 7]

[0204] where du_cpb_removal_delay_minus1 _k [i] and num_decoding_units_minus1 _k are the parameters for the i-th decoding unit of the k-th access unit (with k=0 for the access unit that initialized HRD, and T _du (k)=0 for k<1), and where du_cpb_removal_delay_minus1 _k [i]+1=du_cpb_removal_delay_minus1 _k [i] is the removal delay parameter in the CPB of the decoding unit for the i-th decoding unit of the k-th access unit, and num_decoding_units_minus1 _k is the number of decoding units in the k-th access unit, t _c is the clock tick, t _{c_sub} is the tick for the image fragment, and i and k are indices. Then, when the image sync flag (eg sub_pic_cpb_params_present_flag) is set to 1, the following condition must be true:

[0205] (au_cpb_removal_delay_minus1+1)*t _c ==T _du (k), where (au_cpb_removal_delay_minus1+1)=cpb_removal_delay, CPB removal delay. Thus, in this case, the CPB removal delay (au_cpb_removal_delay_minus1+1) is set such that the operation of the CPB based on the image fragment and the operation of the CPB based on the access block result in the same synchronization of the removal of the access block and the removal of the last decoding block in the access block.

[0206] Alternatively, the CPB removal delay may be set according to

[0207] [Mathematical expression 8]

[0208] where du_cpb_removal_delay[i] are the removal delay parameters in the CPB of the decoding block, t _c is the clock tick, t _{c_sub} is the tick for the image fragment, num_decoding_units_minus1 is the number of decoding units in the access block, offset by one, and i is the index.

[0209] Alternatively, cpb_removal_delay and du_cpb_removal_delay[num_decoding_units_minus1] may be set as follows: cpb_removal_delay*t _c = du_cpb_removal_delay[num_decoding_units_minus1]*t _c,sub , where du_cpb_removal_delay[num_decoding_units_minus1] is the removal delay parameter in the decoding unit CPB for the num_decoding_units_minus1-th decoding unit, t _c is the system clock tick, t _{c_sub} is the tick for the image fragment, num_decoding_units_minus1 is the number of decoding units in the access unit offset by one.

[0210] In addition to modifying the semantics of the SEI picture synchronization message, the present systems and methods may also impose a bitstream constraint such that operation of the CPB based on a picture fragment and operation of the CPB based on an access block result in the same decoding block removal synchronization. Specifically, when the picture synchronization flag (e.g., sub_pic_cpb_params_present_flag) is set to 1, the values for cpb_removal_delay and du_cpb_removal_delay[num_decoding_units_minus1] may be set to satisfy: -1<=(cpb_removal_delay*t _c - du_cpb_removal_delay[num_decoding_units_minus1]*t _{c_sub} )<=1, where du_cpb_removal_delay[num_decoding_units_minus1] is the removal delay parameter in the decoding unit CPB for the num_decoding_units_minus1-th decoding unit, t _c is the system clock tick, t _{c_sub} is the tick for the image fragment, num_decoding_units_minus1 is the number of decoding units in the access unit, with an offset of one.

[0211] Figure 6A is a block diagram illustrating one configuration of the encoder 604 on the electronic device 602. It should be noted that one or more elements illustrated as included in the electronic device 602 may be implemented in hardware, software, or a combination of both. For example, the electronic device 602 includes the encoder 604, which may be implemented in hardware, software, or a combination of both. For example, the encoder 604 may be implemented as a circuit, an integrated circuit, an application-specific integrated circuit (ASIC), a processor in electronic communication with a memory with executable instructions, firmware, a field programmable gate array (FPGA), etc., or a combination thereof. In some configurations, the encoder 604 may be a HEVC encoder.

[0212] The electronic device 602 may include a video source 622. The video source 622 may provide an image or video data (e.g., video) to the encoder 604 as one or more input images 606. Examples of the video source 622 may include image sensors, memory, communication interfaces, network interfaces, wireless receivers, ports, etc.

[0213] One or more input images 606 may be provided to the intra-frame prediction module 624 and the reconstruction buffer. The input image 606 may also be provided to the motion estimation and motion compensation module 646 and the subtraction module 628.

[0214] The intra-picture prediction unit 624 and the reconstruction buffer may generate intra-picture mode information 640 and an intra-picture coded signal 626 based on one or more input pictures 606 and the reconstructed data 660. The motion estimation and motion compensation unit 646 may generate inter-picture mode information 648 and an inter-picture coded signal 644 based on one or more input pictures 606 and a reference picture 678 from the decoded picture buffer 676. In some configurations, the decoded picture buffer 676 may include data from one or more reference pictures located in the decoded picture buffer 676.

[0215] The encoder 604 may select between the intra-frame coded signal 626 and the inter-frame coded signal 644 according to the mode. The intra-frame coded signal 626 may be used to apply spatial characteristics in an image in the intra-frame coded mode. The inter-frame coded signal 644 may be used to apply temporal characteristics between images in the inter-frame coded mode. When in the intra-frame coded mode, the intra-frame coded signal 626 may be provided to the subtractor 628, and the intra-frame coded mode information 640 may be provided to the entropy coding unit 642. When in the inter-frame coding mode, the inter-frame coding signal 644 may be provided to the subtractor 628, and the inter-frame mode information 648 may be provided to the entropy coding module 642.

[0216] Either the intra-frame coded signal 626 or the inter-frame coded signal 644 (depending on the mode) is subtracted from the input image 606 in a subtraction unit 628 to create a residual prediction signal 630. The residual prediction signal 630 is provided to a transform unit 632. The transform unit 632 may compress the residual prediction signal 630 to create a transformed signal 634, which is provided to a quantization unit 636. The quantization unit 636 quantizes the transformed signal 634 to create transformed and quantized coefficients (TQC) 638.

[0217] The TQC coefficients 638 are provided to an entropy encoding module 642 and an inverse quantization module 650. The inverse quantization module 650 performs inverse quantization on the TQC coefficients 638 to create an inversely quantized signal 652, which is provided to an inverse transform module 654. The inverse transform module 654 decompresses (restores to an uncompressed state) the inversely quantized signal 652 to create a decompressed signal 656, which is provided to a restore module 658.

[0218] The restoration module 658 may create restored data 660 based on the decompressed signal 656. For example, the restoration module 658 may re-create (modified) images. The restored data 660 may be provided to the deblocking filter 662 and to the intra-frame prediction module and restoration buffer 624. The deblocking filter 662 may output a filtered signal 664 based on the restored data 660.

[0219] The filtered signal 664 may be provided to a sample adaptive offset module (SAO filter) 666. The SAO module 666 may create SAO information 668, which is provided to an entropy encoding module 642, and a SAO signal 670, which is provided to an adaptive feedback filter (ALF) 672. The ALF 672 produces an ALF signal 674, which is provided to a decoded picture buffer 676. The ALF signal 674 may include data from one or more pictures that may be used as reference pictures.

[0220] Entropy encoding unit 642 may encode the coefficients of TQC 638 to create bitstream A 614a (e.g., coded image data). For example, entropy encoding unit 642 may encode instances of TQC 638 using context-adaptive variable run-length coding (CAVLC) or context-adaptive binary arithmetic coding (CABAC). In particular, entropy encoding unit 642 may encode instances of TQC 638 based on one or more of intra-frame mode information 640, inter-frame mode information 648, and SAO information 668. Bitstream A 614a (e.g., coded image data) may be provided to message generation unit 608. Message generation unit 608 may be configured similar to message generation unit 108 described in connection with FIG. Additionally or alternatively, message generation module 608 may perform one or more of the procedures described in connection with Fig. 2 and Fig. 3.

[0221] For example, the message generation module 608 may generate a message (e.g., a picture synchronization SEI message or another message) that includes parameters of a sub-picture. The parameters of a sub-picture may include one or more removal delays for decoding units (e.g., common_du_cpb_removal_delay or du_cpb_removal_delay[i]) and one or more NAL parameters (e.g., common_num_nalus_in_du_minus1 or num_nalus_in_du_minus1[i]). In some configurations, the message may be inserted into bitstream A 614a to create bitstream B 614b. Thus, the message may be generated after the entire bitstream A 614a has been generated (e.g., after most of bitstream B 614b has been generated), for example. In other configurations, the message may not be inserted into bitstream A 614a (in which case bitstream B 614b may be the same as bitstream A 614a), but may be provided in a separate transmission 610.

[0222] In some configurations, the electronic device 602 sends the bit stream 614 to another electronic device. For example, the bit stream 614 can be provided to a communication interface, a network interface, a wireless transmitter, a port, etc. For example, the bit stream 614 can be transmitted to another electronic device via a LAN, the Internet, a cellular telephone base station, etc. The bit stream 614 can additionally or alternatively be stored in a memory or another component in the electronic device 602.

[0223] Figure 6B is a block diagram illustrating one configuration of a video encoder 1782 on an electronic device 1702. The video encoder 1782 may include an enhancement layer encoder 1706, a base layer encoder 1709, a resolution upscaling (changing) unit 1770, and an output interface 1780. The video encoder of Figure 6B, for example, is suitable for scalable video coding and multi-view video coding, as described herein.

[0224] The enhancement layer encoder 1706 may include a video input unit 1781 that receives an input image 1704. The output of the video input unit 1781 may be provided to an adder/subtractor 1783 that receives an output of the prediction mode selection 1750. The output of the adder/subtractor 1783 may be provided to a transform and quantize unit 1752. The output of the transform and quantize unit 1752 may be provided to an entropy encoding unit 1748 and a scaling and inverse transform unit 1772. After performing the entropy encoding 1748, the output of the entropy encoding unit 1748 may be provided to an output interface 1780. The output interface 1780 may output both the encoded base layer video bitstream 1707 and the encoded enhancement layer video bitstream 1710.

[0225] The output of the inverse transform and scaling unit 1772 may be provided to a summer 1779. The summer 1779 may also receive the output of the prediction mode selection 1750. The output of the summer 1779 may be provided to a deblocking unit 1751. The output of the deblocking unit 1751 may be provided to a reference picture buffer 1794. The output of the reference buffer 1794 may be provided to a motion compensation unit 1754. The output of the motion compensation unit 1754 may be provided to the prediction mode selection 1750. The output of the reference picture buffer 1794 may also be provided to an intra-picture predictor 1756. The output of the intra-picture predictor 1756 may be provided to the prediction mode selection 1750. The prediction mode selection 1750 may also receive the output of the resolution upscaling unit 1770.

[0226] The base layer encoder 1709 may include a video input unit 1762 that receives a subsampled input image, or other image content suitable for combining (comb filtering?) with another image, or an alternative view input image, or the same input image 1703 (that is, the same as the input image 1704 received by the enhancement layer encoder 1706). The output of the video input unit 1762 may be provided to an encoding prediction circuit 1764. Entropy encoding 1766 may be provided on the output of the encoding prediction circuit 1764. The output of the encoding prediction circuit 1764 may also be provided to a reference picture buffer 1768. The reference picture buffer 1768 may provide feedback for the encoding prediction circuit 1764. The output of the reference picture buffer 1768 may also be provided to the resolution increase unit 1770. Once the entropy coding 1766 has been performed, the output may be provided to the output interface 1780. The coded base layer video bitstream 1707 and/or the coded enhancement layer video bitstream 1710 may be provided to one or more message generation units, as needed.

[0227] Figure 7A is a block diagram illustrating one configuration of a decoder 712 on an electronic device 702. The decoder 712 may be included in the electronic device 702. For example, the decoder 712 may be a HEVC decoder. The decoder 712 and one or more of the elements illustrated as included in the decoder 712 may be implemented as hardware, software, or a combination of both. The decoder 712 may receive a bitstream 714 (e.g., one or more coded pictures and service data included in the bitstream 714) for decoding. In some configurations, the received bitstream 714 may include received service data, such as a message (e.g., a picture synchronization SEI message or other message), a slice header, a PPS, etc. In some configurations, the decoder 712 may additionally receive a separate transmission 710. The separate transmission 710 may include a message (e.g., an SEI message about picture synchronization or another message). For example, the SEI message about picture synchronization or another message may be received in a separate transmission 710, and not in the bitstream 714. However, it should be noted that the separate transmission 710 may be optional and may not be used in some configurations.

[0228] The decoder 712 includes a CPB 720. The CPB 720 may be configured similarly to the CPB 120 described in connection with Fig. 1 above. In addition or alternatively, the decoder 712 may perform one or more of the procedures described in connection with Fig. 4 and Fig. 5. For example, the decoder 712 may receive a message (e.g., a picture synchronization SEI message or another message) with parameters of a sub-picture, and remove and decode decoding units in an access unit based on the parameters of the sub-picture. It should be noted that one or more access units may be included in a bitstream and may include one or more of coded picture data and service data.

[0229] The coded picture buffer (CPB) 720 may provide coded picture data to the entropy decoding unit 701. The coded picture data may be entropy decoded by the entropy decoding unit 701, thereby creating a motion data signal 703 and quantized, scaled and/or transformed coefficients 705.

[0230] The motion data signal 703 may be combined with a portion of the reference frame signal 798 from the decoded picture buffer 709 in the motion compensation unit 780, which may create an inter-picture prediction signal 782. The quantized, descaled, and/or transformed coefficients 705 may be inversely quantized, scaled, and inversely transformed by the inverse (transform/scaling/quantization) unit 707, thereby outputting a decoded residual signal 784. The decoded residual signal 784 may be summed with the prediction signal 792 to create a combined signal 786. The prediction signal 792 may be a signal selected from either the inter-picture prediction signal 782 output by the motion compensation unit 780 or the intra-picture prediction signal 790 output by the intra-picture prediction unit 788. In some configurations, this signal selection may be based on (e.g. controlled by) the bitstream 714.

[0231] The intra-frame prediction signal 790 may be predicted based on previously decoded information from the combined signal 786 (in the current frame, for example). The combined signal 786 may also be filtered by a deblocking filter 794. The resulting filtered signal 796 may be written to the decoded picture buffer 709. The resulting filtered signal 796 may include a decoded picture. The decoded picture buffer 709 may provide a decoded picture, which may be output 718. In some cases, 709 may be considered as a frame memory.

[0232] Figure 7B is a block diagram illustrating one configuration of a video decoder 1812 in an electronic device 1802. The video decoder 1812 may include an enhancement layer decoder 1815 and a base layer decoder 1813. The video decoder 1812 may also include an interface 1889 and a resolution increase unit 1870. The video decoder of Figure 7B, for example, is suitable for scalable video decoding and multi-view video decoding, as described herein.

[0233] Interface 1889 may receive coded video stream 1885. Coded video stream 1885 may consist of coded base layer video stream and coded enhancement layer video stream. These two streams may be sent separately or together. Interface 1889 may provide some or all of coded video stream 1885 to entropy decoding unit 1886 in base layer decoder 1813. Output of entropy decoding unit 1886 may be provided to decoding prediction circuit 1887. Output of decoding prediction circuit 1887 may be provided to reference picture buffer 1888. Reference picture buffer 1888 may provide feedback to decoding prediction circuit 1887. Reference picture buffer 1888 may also output decoded base layer video stream 1884.

[0234] Interface 1889 may also provide some or all of the encoded video stream 1885 to entropy decoding unit 1890 in enhancement layer decoder 1815. The output of entropy decoding unit 1890 may be provided to an inverse quantization unit. The output of inverse quantization unit 1891 may be provided to a summer. Summer 1892 may sum the output of inverse quantization unit 1891 and the output of prediction mode selection unit 1895. The output of summer 1892 may be provided to deblocking unit 1893. The output of deblocking unit 1893 may be provided to reference picture buffer 1894. Reference picture buffer 1894 may output decoded enhancement layer video stream 1882. The output of the reference picture buffer 1894 may also be provided to the intra-frame predictor 1897. The enhancement layer decoder 1815 may include a motion compensation unit 1896. The motion compensation 1896 may be performed after increasing the resolution 1870. The prediction mode selection unit 1895 may receive the output of the intra-frame predictor 1897 and the output of the motion compensation 1896. In addition, the decoder may include one or more coded picture buffers as needed, for example, together with the interface 1889.

[0235] Figure 8 illustrates various components that may be used in the transmitting electronic device 802. One or more of the electronic devices 102, 602, 702 described herein may be implemented in accordance with the transmitting electronic device 802 illustrated in Figure 8.

[0236] The transmitting electronic device 802 includes a processor 817 that controls the operation of the electronic device 802. The processor 817 may also be referred to as a central processing unit (CPU). Memory 811, which may include both read-only memory (ROM) and random access memory (RAM), or any type of device that can store information, provides instructions 813a (e.g., executable instructions) and data 815a to the processor 817. A portion of the memory 811 may also include non-volatile random access memory (NVRAM). The memory 811 may be in electronic communication with the processor 817.

[0237] The instructions 813b and the data 815b may also reside on the processor 817. The instructions 813b and/or the data 815b loaded into the processor 817 may also include the instructions 813a and/or the data 815a from the memory 811 that were loaded for execution or processing by the processor 817. The instructions 813b may be executed by the processor 817 to implement the systems and methods disclosed in the description. For example, the instructions 813b may be executable to perform one or more of the methods 200, 300, 400, 500 described above.

[0238] The transmitting electronic device 802 may include one or more communication interfaces 819 for communicating with other electronic devices (e.g., a receiving electronic device). The communication interfaces 819 may be based on a wired communication technology, a wireless communication technology, or both. Examples of the communication interface 819 include a serial port, a parallel port, a universal serial bus (USB), an Ethernet adapter, an IEEE 1394 bus interface, a small computer system interface (SCSI) bus interface, an infrared (IR) communication port, a Bluetooth wireless communication adapter, a wireless transceiver in accordance with the 3rd Generation Partnership Project (3GPP) specifications, etc.

[0239] The transmitting electronic device 802 may include one or more output devices 823 and one or more input devices 821. Examples of output devices 823 include a speaker, a printer, etc. One type of output device that may be included in the electronic device 802 is a display device 825. The display devices 825 used with the configurations disclosed in the description may use any suitable image projection technology, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), gas discharge plasma, electroluminescence, and the like. A display controller 827 may be provided for converting data stored in the memory 811 into text, graphics, and/or moving images (as needed) displayed on the display device 825. Examples of input devices 821 include a keyboard, a mouse, a microphone, a remote control, a button, a joystick, a trackball, a touch pad, a touch screen, a light pen, etc.

[0240] The various components of the transmitting electronic device 802 are connected together by a bus system 829, which may include a power bus, a control signal bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, the various buses are illustrated in Fig. 8 as a bus system 829. The transmitting electronic device 802 illustrated in Fig. 8 is a functional block diagram and not a listing of specific components.

[0241] Figure 9 is a block diagram illustrating various components that may be used in the receiving electronic device 902. One or more of the electronic devices 102, 602, 702 described herein may be implemented in accordance with the receiving electronic device 902 illustrated in Figure 9.

[0242] The receiving electronic device 902 includes a processor 917 that controls the operation of the electronic device 902. The processor 917 may also be referred to as a CPU. Memory 911, which may include both read-only memory (ROM) and random access memory (RAM), or any type of device that can store information, provides instructions 913a (e.g., executable instructions) and data 915a to the processor 917. A portion of the memory 911 may also include non-volatile random access memory (NVRAM). The memory 911 may be in electronic communication with the processor 917.

[0243] The instructions 913b and the data 915b may also reside on the processor 917. The instructions 913b and/or the data 915b loaded into the processor 917 may also include the instructions 913a and/or the data 915a from the memory 911 that were loaded for execution or processing by the processor 917. The instructions 913b may be executed by the processor 917 to implement the systems and methods disclosed in the description. For example, the instructions 913b may be executable to perform one or more of the methods 200, 300, 400, 500 described above.

[0244] The receiving electronic device 902 may include one or more communication interfaces 919 for communicating with other electronic devices (e.g., a transmitting electronic device). The communication interface 919 may be based on a wired communication technology, a wireless communication technology, or both. Examples of communication interfaces 919 include a serial port, a parallel port, a universal serial bus (USB), an Ethernet adapter, an IEEE 1394 bus interface, a small computer system interface (SCSI) bus interface, an infrared (IR) communication port, a Bluetooth wireless communication adapter, a wireless transceiver in accordance with the 3rd Generation Communications Partnership Project (3GPP), etc.

[0245] The receiving electronic device 902 may include one or more output devices 923 and one or more input devices 921. Examples of output devices 923 include a speaker, a printer, etc. One type of output device that may be included in the electronic device 902 is a display device 925. The display devices 925 used with the configurations disclosed in the description may use any suitable image projection technology, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), gas discharge plasma, electroluminescence, and the like. A display controller 927 may be provided for converting data stored in the memory 911 into text, graphics, and/or moving images (as needed) displayed on the display device 925. Examples of input devices 921 include a keyboard, a mouse, a microphone, a remote control, a button, a joystick, a trackball, a touch pad, a touch screen, a light pen, etc.

[0246] The various components of the receiving electronic device 902 are connected by a bus system 929, which may include a power bus, a control signal bus, and a status signal bus, in addition to a data bus. However, for the sake of clarity, the various buses are illustrated in Fig. 9 as a bus system 929. The receiving electronic device 902 illustrated in Fig. 9 is a functional block diagram and not a listing of specific components.

[0247] Figure Fig. 10 is a block diagram illustrating one configuration of an electronic device 1002 in which systems and methods for sending a message may be implemented. The electronic device 1002 includes encoding means 1031 and transmitting means 1033. Encoding means 1031 and transmitting means 1033 may be configured to perform one or more of the functions described in connection with one or more of Fig. 1, Fig. 2, Fig. 3, Fig. 6, and Fig. 8 above. For example, encoding means 1031 and transmitting means 1033 may generate a bit stream 1014. Figure Fig. 8 above illustrates one example of a specific hardware design of Fig. 10. Other various structures may be implemented to implement one or more of the functions of Fig. 1, Fig. 2, Fig. 3, Fig. 6, and Fig. 8. For example, the DSP may be implemented in software.

[0248] Figure 11 is a block diagram illustrating one configuration of an electronic device 1102 in which systems and methods for buffering a bitstream 1114 may be implemented. The electronic device 1102 may include a receiving means 1135 and a decoding means 1137. The receiving means 1135 and the decoding means 1137 may be configured to perform one or more of the functions described in connection with one or more of Figures 1, 4, 5, 7, and 9 above. For example, the receiving means 1135 and the decoding means 1137 may receive the bitstream 1114. Figure 9 above illustrates one example of the specific hardware design of Figure 11. Other various structures may be implemented to implement one or more of the functions of Figures 1, 4, 5, 7, and 9. For example, the DSP may be implemented in software.

[0249] Figure 12 is a flow chart illustrating one configuration of a method 1200 for operating a decoded picture buffer (DPB). The method 1200 may be performed by the encoder 104 or one of its subsystems (e.g., the decoded picture buffer module 676). The method 1200 may be performed by the decoder 112 in the electronic device 102 (e.g., the electronic device B 102b). In addition or alternatively, the method 1200 may be performed by the decoder 712 or one of its subsystems (e.g., the decoded picture buffer module 709). The decoder may parse the first slice header of the picture 1202. The output and removal of pictures from the DPB before decoding the current picture (but after parsing the slice header for the first slice of the current picture) occurs immediately when the first decoding block of the access block containing the current picture is removed from the CPB and continues as described below.

[0250] - Starting a decoding process for a reference picture set (RPS). A reference picture set is a set of reference pictures associated with a picture, consisting of all reference pictures that are before the associated picture in decoding order, which can be used for inter-picture prediction of the associated picture or any picture following the associated picture in decoding order.

[0251] - A video bitstream may include a syntax structure that is placed into logical data packets, commonly referred to as network abstraction layer (NAL) units. Each NAL unit includes a NAL unit header, such as a two-byte NAL unit header (e.g., 16 bits), to identify the destination of the associated data payload. For example, each coded slice (and/or frame) may be coded in one or more slice (and/or frame) NAL units. Other NAL units may be included for other categories of data, such as, for example, additional extension information, a coded slice of a TSA frame, a coded slice of a STSA frame, a coded slice of a non-TSA frame, a non-STSA trailer frame, a coded slice of a broken link access frame, a coded slice of an immediate refresh of a decoded frame, a coded slice of a random access frame, a coded slice of a decoded head frame, a coded slice of a marked-for-drop frame, a video parameter set, a sequence parameter set, a picture parameter set, an access unit delimiter, an end of a sequence, an end of a bitstream, filler data, and/or a sequence extension information message. Table (4) illustrates one example of NAL unit codes and NAL unit type classes. Other NAL unit types may be included as needed. It should also be understood that the NAL unit type values for the NAL units shown in Table (4) may be rearranged and reassigned. In addition, additional NAL unit types may be added. Also, some NAL unit types may be removed.

[0252] An intra-frame random access point (IRAP) picture (frame) is a coded picture for which each video coding layer NAL unit has a nal_unit_type in the range from BLA_W_LP to RSV_IRAP_VCL23, inclusive, as shown in Table (4). An IRAP picture contains only intra-frame coded (I) slices. An instantaneous decoding refresh (IDR) picture is an IRAP picture for which each video coding layer NAL unit has a nal_unit_type equal to IDR_W_RADL or IDR_N_LP, as shown in Table (4). An instantaneous decoding refresh (IDR) picture contains only I slices, and may be the first picture in the bitstream in decoding order, or may appear later in the bitstream. Each IDR picture is the first picture in a coded video sequence (CVS) in decoding order. A broken link access (BLA) picture is an IRAP picture for which each NAL unit of the video coding layer has a nal_unit_type equal to BLA_W_LP, BLA_W_RADL, or BLA_N_LP, as shown in Table (4). A BLA picture contains only I slices, and may be the first picture in the bitstream in decoding order, or may appear in the bitstream subsequently. Each BLA picture begins a new coded video sequence, and has the same effect on the decoding process as an IDR picture. However, a BLA picture contains syntax elements that specify a non-empty set of reference pictures.

[0253]

Table 4 nal_unit_type Block type name nal NAL unit content and raw byte sequence payload (RBSP) syntax structure NAL Unit Type Class 0
1 TRAIL_N
TRAIL_R Encoded segment of non-TSA slice, non-STSA tail image slice_segment_layer_rbsp() Video Coding Layer (VCL) 2
3 TSA_N
TSA_R Encoded slice segment in a TSA image slice_segment_layer_rbsp() VCL 4
5 STSA_N
STSA_R Coded image slice segment with step-wise temporal sublayer access (STSA) slice_segment_layer rbsp() VCL 6
7 RADL_N
RADL_R Coded Random Access Decodable Head (RADL) Slice Segment slice_segment_layer_rbsp() VCL 8
9 RASL_N
RASL_R Coded random access skipped leading (RASL) slice_segment_layer_rbsp() VCL 10
12
14 RSV_VCL_N10 RSV_VCL_ N12 RSV_VCL_ N14 Reserved VCL NAL Unit Types for IRAP without Sublayer Reference VCL 11
13
15 RSV_VCL_R11 RSV_VCL_R13 RSV_VCL_R15 Reserved VCL NAL unit types for non-IRAP with sublayer reference VCL 16
17
18 BLA_W_LP BLA_W_RADL BLA_N_LP Coded image slice segment with broken link access (BLA) slice_segment_layer rbsp() VCL 19
20 IDR_W_RADL IDR_N_LP Coded image slice segment with instantaneous decoding refresh (IDR)
slice_segment_layer_rbsp() VCL 21 CRA_NUT Coded Random Access (CRA) Image Slice Segment slice_segment_layer_rbsp() VCL 22
23 RSV_IRAP_VCL22 RSV_IRAP_VCL23 Reserved VCL NAL Unit Types for IRAP VCL 24..31 RSV_VCL24..
RSV_VCL31 Reserved VCL NAL Unit Types for Non-IRAP VCL 32 VPS_NUT Video Parameter Set video_parameter_set_rbsp() Non-Video Coding Layer (non-VCL) 33 SPS_NUT Sequence parameter set seq_parameter_set_rbsp() non-VCL 34 PPS_NUT Image parameter set pic_parameter_set rbsp() non-VCL 35 AUD_NUT Access Unit Delimiter access_unit_delimiter_rbsp() non-VCL 36 EOS_NUT End of sequence end_of_ seq_rbsp() non-VCL 37 EOB_NUT End of bitstream end_of_bitstream_rbsp() non-VCL 38 FD_NUT Filler data filler_data_rbsp() non-VCL 39
40 PREFIX_SEI_NUT
SUFFIX_SEI_NUT Additional extended information sei_rbsp() non-VCL 41..47 RSV_NVCL41..RSV_NVCL47 Reserved non-VCL 48..63 UNSPEC48..
UNSPEC63 Not defined non-VCL

Table (4)

[0254] With reference to Table (5), the NAL unit header syntax may include two bytes of data, namely, 16 bits. The first bit is the "forbidden_zero_bit", which is always set to zero at the beginning of the NAL unit. The next six bits are the "nal_unit_type", which specifies the type of the raw byte sequence payload ("RBSP") data structure contained in the NAL unit, as shown in Table (4). The next 6 bits are the "nuh_layer_id", which specifies the layer identifier. In some cases, these six bits may be specified as "nuh_reserved_zero_6bits" instead. The value of nuh_reserved_zero_6bits may be equal to 0 in the base specification of the standard. In scalable video coding and/or syntax extensions, nuh_layer_id may indicate that this particular NAL unit belongs to the layer identified by the value of these 6 bits. The next syntax element is "nuh_temporal_id_plus1". The value nuh_temporal_id_plus1 minus 1 can specify a temporal identifier for a NAL unit. The variable TemporalId can be defined as TemporalId=nuh_temporal_id_plus1-1. The TemporalId is used to identify the temporal sublayer. The variable HighestTid identifies the highest temporal sublayer to be decoded.

[0255] [Table 5]

[0256] Table (6) shows the syntactic structure of an exemplary sequence parameter set (SPS).

[0257] The pic_width_in_luma_samples value specifies the width of each decoded image in units of luma samples. The pic_width_in_luma_samples value must not be 0.

[0258] The pic_height_in_luma_samples value specifies the height of each decoded image in units of luma samples. The pic_height_in_luma_samples value must not be equal to 0.

[0259] The value of sps_max_sub_layers_minus1 plus 1 specifies the maximum number of temporary sublayers that can be present in each CVS that references the SPS. The value of sps_max_sub_layers_minus1 must be in the range 0 to 6, inclusive.

[0260] An sps_sub_layer_ordering_info_present_flag value of 1 specifies that the sps_max_dec_pic_buffering_minus1[i], sps_max_num_reorder_pics[i], and sps_max_latency_increase_plus1[i] syntax elements are present for sps_max_sub_layers_minus1+1 sublayers. An sps_sub_layer_ordering_info_present_flag value of 0 specifies that the sps_max_dec_pic_buffering_minus1[sps_max_sub_layers_minus1], sps_max_num_reorder_pics [sps_max_sub_layers_minus1], and sps_max_latency_increase_plus1 [sps_max_sub_layers_minus1] syntax elements apply to all sublayers.

[0261] The value of sps_max_dec_pic_buffering_minus1[i] plus 1 specifies the maximum required decoded picture buffer size for CVS in units of picture storage buffers when HighestTid is i. The value of sps_max_dec_pic_buffering_minus1[i] will be in the range from 0 to MaxDpbSize - 1, inclusive, where MaxDpbSize specifies the maximum decoded picture buffer size in units of picture storage buffers. When i is greater than 0, sps_max_dec_pic_buffering_minus1[i] will be greater than or equal to sps_max_dec_pic_buffering_minus1[i-1]. When sps_max_dec_pic_buffering_minus1[i] is not present for i in the range 0 to sps_max_sub_layers_minus1-1, inclusive, due to sps_sub_layer_ordering_info_present_flag being 0, it is implied to be equal to sps_max_dec_pic_buffering_minus1[sps_max_sub_layers_minus1].

[0262] The sps_max_num_reorder_pics[i] value specifies the maximum number of images that may precede any image in CVS in decoding order and follow that image in output order when HighestTid is i. The sps_max_num_reorder_pics[i] value will be in the range from 0 to sps_max_dec_pic_buffering_minus1[i], inclusive. When i is greater than 0, sps_max_num_reorder_pics[i] will be greater than or equal to sps_max_num_reorder_pics[i-1]. When sps_max_num_reorder_pics[i] is not present for i in the range 0 to sps_max_sub_layers_minus1-1, inclusive, due to sps_sub_layer_ordering_info_present_flag being 0, it is implied to be equal to sps_max_num_reorder_pics[sps_max_sub_layers_minus1].

[0263] A non-zero value of sps_max_latency_increase_plus1[i] is used to calculate the value of SpsMaxLatencyPictures[i], which specifies the maximum number of pictures that can precede any CVS image in output order and follow that image in decoding order when HighestTid is i.

[0264] When sps_max_latency_increase_plus1[i] is not 0, the value of SpsMaxLatencyPictures[i] is set as follows:

[0265] SpsMaxLatencyPictures[i]=sps_max_num_reorder_pics[i]+sps_max_latency_increase_plus1[i]-1.

[0266] When sps_max_latency_increase_plus1[i] is 0, the corresponding limit is not expressed.

[0267] The value of sps_max_latency_increase_plus1[i] will be in the range 0 to 2 ³² - 2, inclusive. When sps_max_latency_increase_plus1[i] is not present for i in the range 0 to sps_max_sub_layers_minus1-1, inclusive, due to sps_sub_layer_ordering_info_present_flag being 0, it is implied to be equal to sps_max_latency_increase_plus1[sps_max_sub_layers_minus1].

[0268] [Table 6]

[0269] When the current image is an IRAP image, the following applies:

[0270] - If the current picture is an IDR picture, a BLA picture, the first picture in the bitstream in decoding order, or the first picture that follows the end of a NAL unit sequence in decoding order, the NoRaslOutputFlag variable is set to 1.

[0271] - Otherwise, if some external means is available to set the HandleCraAsBlaflag variable to the value for the current image, the HandleCraAsBlaFlag variable is set to the value provided by that external means, and the NoRaslOutputFlag variable is set to HandleCraAsBlaflag.

[0272] - Otherwise, the HandleCraAsBlaflag variable is set to 0, and the NoRaslOutputFlag variable is set to 0.

[0273] If the current image is an IRAP image with NoRaslOutputFlag equal to 1 that is not image 0, the following ordered steps are applied:

[0274] 1. The NoOutputOfPriorPicsFlag variable is obtained for the decoder under test as follows:

[0275] - If the current image is a CRA image, NoOutputOfPriorPicsFlag is set to 1 (regardless of the value of no_output_of_prior_pics_flag).

[0276] - Otherwise, if the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_pic_buffering_minus1[HighestTid] received from the active SPS differs from the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_pic_buffering_minus1[HighestTid], respectively, received from the SPS active for the preceding picture, the NoOutputOfPriorPicsFlag flag may (but need not) be set to 1 by the decoder under test regardless of the value of no_output_of_prior_pics_flag.

[0277] - Otherwise NoOutputOfPriorPicsFlag is set to no_output_of_prior_pics_flag.

[0278] 2. The NoOutputOfPriorPicsFlag value obtained for the decoder under test is applied to HRD as follows:

[0279] - If NoOutputOfPriorPicsFlag is 1, all image storage buffers in the DPB are cleared without outputting the images they contain, and the DPB fullness is set to 0.

[0280] - Otherwise (NoOutputOfPriorPicsFlag is 0), all image storage buffers containing an image that is marked as "not required for output" and "not used for reference" are cleared (no output), and all non-empty image storage buffers in the DPB are cleared by repeatedly initiating the "flushing" process 1204, and the DPB fullness is set to 0.

[0281] - Otherwise (the current image is not an IRAP image with NoRaslOutputFlag equal to 1), all image storage buffers containing an image that are marked as "not required for output" and "not used for reference" are flushed (no output). For each image storage buffer that is flushed, the DPB occupancy is decreased by one. When one or more of the following conditions are true, the "flushing" process 1204 is called iteratively, further decreasing the DPB occupancy by one for each additional image storage buffer that is flushed, until none of the following conditions are true:

[0282] 1. The number of images with this particular nuh_layer_id value in the DPB that are marked as "needed for output" is greater than sps_max_num_reorder_pics[HighestTid] from the active sequence parameter set (when this particular nuh_layer_id value is 0), or from the active layer sequence parameter set for this particular nuh_layer_id value.

[0283] 2. If sps_max_latency_increase_plus1[HighestTid] from the active sequence parameter set (when that particular nuh_layer_id value is 0) or from the active layer sequence parameter set for that particular nuh_layer_id value is not equal to 0, and there is at least one picture with that particular nuh_layer_id value in the DPB that is marked as "required for output" for which the associated PicLatencyCount variable is greater than or equal to SpsMaxLatencyPictures[HighestTid] for that particular nuh_layer_id value.

[0284] 3. The number of images with this particular nuh_layer_id value in the DPB is greater than or equal to sps_max_dec_pic_buffering[HighestTid]+1 from the active sequence parameter set (when this particular nuh_layer_id value is 0) or from the active layer sequence parameter set for this particular nuh_layer_id value.

[0285] The process of decoding the image in step 1206 (decoding and marking the image) occurs immediately when the last decoding block in the access block containing the current image is removed from the CPB.

[0286] For each image with a nuh_layer_id value equal to the nuh_layer_id value of the current image in the DPB that is marked as "required for output", the associated PicLatencyCount variable is set to PicLatencyCount+1.

[0287] The current picture is considered decoded after the last decoding block in the picture is decoded. The current decoded picture is stored in the DPB in an empty picture storage buffer and the following applies:

[0288] - If the PicOutputFlag of the current decoded image is 1, it is marked as "required for output" and its associated PicLatencyCount variable is set to 0.

[0289] - Otherwise (PicOutputFlag of the current decoded image is 0), it is marked as "not required for output".

[0290] The current decoded image is marked as "used for short-term reference".

[0291] When one or more of the following conditions are true, the additional "drop" process 1208 is called repeatedly until none of the following conditions are true:

[0292] - The number of images with nuh_layer_id equal to the current image's nuh_layer_id in the DPB that are marked as "needed for output" is greater than sps_max_num_reorder_pics[HighestTid] from the active sequence parameter set (when the current image's nuh_layer_id is 0) or from the active layer sequence parameter set for the current image's nuh_layer_id.

[0293] - sps_max_latency_increase_plus1[HighestTid] from the active sequence parameter set (when the current picture's nuh_layer_id value is 0) or from the active layer sequence parameter set for the current picture's nuh_layer_id value is not equal to 0, and there is at least one picture with that particular nuh_layer_id value in the DPB that is marked as "required for output" for which the associated PicLatencyCount variable is greater than or equal to SpsMaxLatencyPictures[HighestTid] for that particular nuh_layer_id value.

[0294] The "drop" process 1204 and the additional drop process 1208 are identical in terms of steps, and consist of the following ordered steps: The pictures that are first for output are selected as those with the smallest value of the picture order counter (PicOrderCntVal) of all the pictures in the DPB marked as "required for output". The picture order counter is a variable that is associated with each picture, uniquely identifies the linked picture among all the pictures in the CVS, and, when the linked picture is to be output from the decoded picture buffer, indicates the position of the linked picture in the output order relative to the output order positions of other pictures in the same CVS that are to be output from the decoded picture buffer.

[0295] - These images are cropped using the cropping condition matching window specified in the active sequence parameter set for the image with nuh_layer_id equal to 0, or in the active layer sequence parameter set for the nuh_layer_id value equal to that of the image, the cropped images are output in ascending nuh_layer_id order, and the images are marked as "not required for output".

[0296] - Each image storage buffer that contains an image marked as "not used for reference" and that included one of the images that was cropped and output is cleared.

[0297] Referring to Fig. 13A, as previously described, the syntax of a NAL unit header may include two bytes of data, namely, 16 bits. The first bit is a "forbidden_zero_bit", which is always set to zero at the beginning of a NAL unit. The next six bits are a "nal_unit_type", which specifies the type of the raw byte sequence payload ("RBSP") data structure contained in the NAL unit. The next 6 bits are "nuh_reserved_zero_6bits". nuh_reserved_zero_6bits may be equal to 0 in the base specification of the standard. Other values of nuh_reserved_zero_6bits may be defined as needed. Decoders may ignore (that is, remove from the bitstream and discard) all NAL units with nuh_reserved_zero_6bits values not equal to 0 when processing the stream based on the base specification of the standard. In a scalable or other extension, nuh_reserved_zero_6bits may specify other values to signal scalable video coding and/or syntax extensions. In some cases, the nuh_reserved_zero_6bits syntax element may be named reserved_zero_6bits. In some cases, the nuh_reserved_zero_6bits syntax element may be named layer_id_plus1 or layer_id, as illustrated in Fig. 13B and Fig. 13C. In this case, the layer_id element will be layer_id_plus1 minus 1. In this case, it can be used to signal information related to a scalable coded video layer. The next syntax element is "nuh_temporal_id_plus1". The value nuh_temporal_id_plus1 minus 1 may specify a temporal identifier for a NAL unit. The "temporary identifier" variable TemporalId can be defined as TemporalId=nuh_temporal_id_plus1-1.

[0298] With reference to Fig. 14, the general syntax structure of a NAL unit is illustrated. The syntax of the two bytes of the NAL unit header of Fig. 13 is included in the reference to nal_unit_header() of Fig. 14. The remainder of the syntax of the NAL unit generally relates to RBSP.

[0299] One existing method for using "nuh_reserved_zero_6bits" is to signal scalable video coding information by splitting the 6 bits of nuh_reserved_zero_6bits into separate bit fields, namely, one or more fields of a dependency identifier (ID), a quality ID, a view ID, and a depth flag, each of which is related to identifying a different level of the scalable coded video. Accordingly, the 6 bits indicate the level for the scalable coding method to which this particular NAL unit belongs. Then, in the data payload, such as the syntax ("scalability_type") of the video parameter set ("VPS") extension, as illustrated in Fig. 15, the level information is defined. The VPS extension syntax of Fig. 15 includes 4 bits for a scalability type (scalability_type syntax element), which describes the scalability types in use in the coded video sequence and the measurements signaled by layer_id_plus1 (or layer_id) in the NAL unit header. When the scalability type is 0, the coded video sequence complies with the base specification, so layer_id_plus1 for all NAL units is 0 and there are no NAL units belonging to the layer or enhancement type. Higher values of the scalability type are interpreted as illustrated in Fig. 16.

[0300] The layer_id_dim_len[i] value specifies the length, in bits, of the ith scalability dimension identifier. The sum of the layer_id_dim_len[i] values for all values of i in the range 0 through 7 is less than or equal to 6. The vps_extension_byte_alignment_reserved_zero_bit value is zero. The vps_layer_id[i] value specifies the layer_id value of the ith layer to which the subsequent layer dependency information is applied. The num_direct_ref_layers[i] value specifies the number of layers on which the ith layer directly depends. The ref_layer_id[i][j] value identifies the jth layer on which the ith layer directly depends.

[0301] Thus, the existing method signals scalability identifiers in a NAL unit and in a video parameter set to distribute bits among the scalability types listed in Figure 16. Then, for each scalability type, Figure 16 specifies how many dimensions are supported. For example, scalability type 1 has 2 dimensions (i.e., spatial and quality). For each of the dimensions, layer_id_dim_len[i] specifies the number of bits distributed to each of these two dimensions, where the total sum of all layer_id_dim_len[i] values is less than or equal to 6, which is the number of bits in the nuh_reserved_zero_6bits field in the NAL unit header. Thus, in total, the method identifies which scalability types are used and how the 6 bits of the NAL unit header are distributed among the scalability.

[0302] Although such a fixed combination of different scalability dimensions as illustrated in Fig. 16 is suitable for many applications, there are desirable combinations that are not included. Referring to Fig. 17, the modified video parameter set extension syntax describes a scalability type for each bit in the nuh_reserved_zero_6bits syntax element. The vps_extension_byte_alignment_reserved_zero_bit value is set to 0. The max_num_layers_minus1_bits value indicates the total number of bits used for the syntax element in the first two bytes of the NAL unit header in Fig. 13, called layer_id_plus1 or nuh_reserved_zero_6bits. The scalability_map[i] value specifies the scalability type for each bit in the layer_id_plus1 syntax element. In some cases, the layer_id_plus1 syntax element may instead be referred to as the nuh_reserved_zero_6bits or reserved_zero_6bits syntax element. The scalability map for all bits of the layer_id_plus1 syntax element collectively specifies the scalability in use in the coded video sequence. The actual identifier value for each of the scalability types is signaled by these corresponding bits in the layer_id_plus1 (nuh_reserved_zero_6bits) field in the NAL unit header. When scalability_map[i] is 0 for all values of i, the coded video sequence conforms to the base specification, so the layer_id_plus1 value of the NAL units is 0 and there are no NAL units belonging to the enhancement layer or view. The vps_layer_id[i] element specifies the layer_id value for the i-th layer to which the subsequent layer dependency information applies. The value of num_direct_ref_layers[i] specifies the number of layers on which the ith layer directly depends. The value of ref_layer_id[i][j] identifies the jth layer on which the ith layer directly depends.

[0303] The higher values of scalability_map[i] are interpreted as shown in Fig. 18. scalability_map[i] includes scalability dimensions of (0) none; (1) spatial; (2) quality; (3) depth; (4) multiview; (5) undefined; (6) reserved; and (7) reserved.

[0304] Therefore, each bit in the NAL unit header is interpreted based on 3 bits in the video parameter set regarding what the scalability dimension is (e.g., none, spatial, quality, depth, multiview, unspecified, reserved). For example, to signal that all bits in layer_id_plus1 correspond to spatial scalability, the scalability_map values in the VPS may be encoded as 001 001 001 001 001 001 for 6 bits of the NAL unit header. Also, for example, to signal that 3 bits in layer_id_plus1 correspond to spatial scalability and 3 bits correspond to quality scalability, the scalability_map values in the VPS may be encoded as 001 001 001 010 010 010 for 6 bits of the NAL unit header.

[0305] Referring to FIG. 19, another design includes a video parameter set that signals a number of scalability dimensions in 6 bits of a NAL unit header using num_scalability_dimensions_minus1. The value of num_scalability_dimensions_minus1 plus 1 indicates the number of scalability dimensions signaled by layer_id_plus1; nuh_reserved_zero_6bits; and/or the reserved_zero_6bits syntax elements. The scalability_map[i] element has the same semantics as described above with respect to FIG. 17. The value of num_bits_for_scalability_map[i] specifies the length in bits for the i-th scalability dimension. The sum of all num_bits_for_scalability_map[i] for i=0..num_scalability_dimensions_minus1 is equal to six (or otherwise equal to the number of bits used for layer_id_plus1; vps_reserved_zero_6bits; max_num_layers_minus1; reserved_zero_6bits; nuh_reserved_zero_6bits syntax elements).

[0306] With respect to Fig. 17 and Fig. 19, other embodiments may be used, if desired. In one embodiment, for example, scalability_map[i] may be signaled with u(4) (or u(n) with n> 3 or n<3). In this case, the higher values of scalability_map[i] may be defined as reserved for bitstreams corresponding to a particular video technology profile. For example, values 6..15 of the scalability map may be defined as 'reserved' when signaling scalability_map[i] with u(4). In another embodiment, for example, scalability_map[i] may be signaled with ue(v) or some other coding scheme. In another design, for example, the constraint may be defined such that the values of scalability_map[i] are arranged in monotonically non-decreasing (or non-increasing) order. This results in the scalability dimension boundary fields in the layer_id_plus1 field in the NAL unit header.

[0307] Another existing method for signaling scalable video coding using the syntax element "layer_id_plus1" or "nuh_reserved_zero_6bits" is to map layer_id_plus1 in the NAL unit header to layer identification information by signaling a common look-up table in a video parameter set. Referring to Fig. 20, the existing method includes a video parameter set that specifies a number of dimension types and dimension identifications for the i-th layer in a look-up table. In particular, vps_extension_byte_alignment_reserved_zero_bit is zero. The value num_dimensions_minus1[i] plus 1 specifies the number of dimension types (dimension_type[i][j]) and dimension identifiers (dimension_id[i][j]) for the i-th layer. The value of dimension_type[i][j] specifies the j-th scalability dimension type of the i-th layer whose layer_id or layer_id_plus1 is equal to i, as defined in Fig. 31. As illustrated in Fig. 21, the dimensions that are identified include (0) the index (idx) of the presentation order; (1) the depth flag; (2) the dependency ID; (3) the quality ID; (4) - (15) reserved. The value of dimension_id[i][j] specifies the identifier of the j-th scalability dimension type of the i-th layer, which, if not present, is assumed to be 0. The value of num_direct_ref_layers[i] specifies the number of layers on which the i-th layer directly depends. The value of ref_layer_id[i][j] identifies the j-th layer on which the i-th layer directly depends. Unfortunately, the proposed implementation illustrated in Fig. 20 results in a cumbersome lookup table.

[0308] Referring to Fig. 22, the modified video parameter set extension includes a scalability mask that is used in combination with a scalability dimension. The scalability_mask signals a combination of 0 and 1 bits, where each bit corresponds to one scalability dimension, as indicated by the scalability map syntax of Fig. 23. A value of 1 for a particular scalability dimension specifies that this scalability dimension is present at this level (the I-th level). A value of 0 for a particular scalability dimension specifies that this scalability dimension is not present at this level (the I-th level). For example, the bit set 00100000 refers to quality scalability. The actual identifier value of the particular scalability dimension that is present is indicated by the signaled value of scalability_id[j]. The values of num_scalability_types[i] are equal to the sum of the number of bits in scalability_mask that have a value of 1. Thus,

[0309] [Mathematical expression 9]

[0310] The scalability_id[j] value specifies the identifier value of the j-th scalability dimension for the type of scalability values that are signaled by the scalability_mask value.

[0311] With reference to Fig. 24, a modification of Fig. 22 includes a scalability mask signaled outside the loop. This results in one common mask for each level identification. With reference to Fig. 25, in this modification, a corresponding example set of video parameters may include a scalable identification with a scalability mask that is not included. In this case, the scalable_id[j] syntax element has the same interpretation as the scalability_id[j] syntax element in Fig. 22.

[0312] With reference to FIG. 26 , a modification of FIG. 22 includes a scalability mask (scalability_mask) signaled outside the loop. This results in one common mask for each layer identification. scalability_mask signals a combination of 0 and 1 bits, where each bit corresponds to one scalability dimension, as indicated by the scalability map syntax of FIG. 27 . A value of 1 for a particular scalability dimension specifies that this scalability dimension is present in this layer (the i-th layer). A value of 0 for a particular scalability dimension specifies that this scalability dimension is not present in this layer (the i-th layer). For example, the bit set 00100000 refers to quality scalability. The actual identifier value of the particular scalability dimension that is present is indicated by the signaled value of scalability_id[j]. The values of num_scalability_types[i] are equal to the sum of the number of bits in scalability_mask that have a value of 1. Thus,

[0313] [Mathematical expression 10]

[0314] In this case, the variable scalability_id[j] may instead be referred to as the variable dimension_id[i][j]. The element dimension_id[i][j] specifies the identifier of the j-th scalability dimension for the i-th level. The variable Scalabilityld[i][j] is then obtained as follows.

[0315] [Table 7]

[0316] Where ScalabilityId[i][k] signals the measurement ID for the corresponding scalability type, as described below.

[0317] [Table 8]

[0318] Where DependencyId[i][1] is the dependency ID for the spatial scalability measurement for the i-th level, QualityId[i][2] is the quality ID for the quality scalability measurement for the i-th level, depthFlag[i][3] is the depth flag/depth ID for the depth scalability measurement for the i-th level, and ViewId[i][4] is the view ID for the multi-view scalability measurement for the i-th level.

[0319] Also in Fig. 26, the avc_base_codec_flag flag equal to 1 specifies that the base layer complies with Rec. ITU-T H.264 | ISO/IEC 14496-10, and the avc_base_codec_flag flag equal to 1 specifies HEVC compliance. The vps_nuh_layer_id_presnet_flag flag indicates whether the layer_id_in_nuh[i] variable, which signals the layer_id value in the NAL unit header, is signaled.

[0320] In another design, one or more of the scalability_mask[i], scalability_mask, scalability_id[j] syntax elements may be signaled using a number of bits other than u(8). For example, they may be signaled using u(16) (or u(n) when n>8 or n<8). In another design, one or more of these syntax elements may be signaled using ue(v). In another design, scalability_mask may be signaled in the NAL unit header in the syntax elements layer_id_plus1; vps_reserved_zero_6bits; max_num_layers_minus1; reserved_zero_6bits; and/or nuh_reserved_zero_6bits. In some embodiments, the system may do this only for VPS NAL units, or only for NAL units that do not correspond to the VPS, or for all NAL units. In another implementation, the scalability_mask may be signaled per picture somewhere in the bitstream. For example, it may be signaled in a slice header, a picture parameter set, a video parameter set, or any other parameter set or any other normative part of the bitstream.

It should be noted that the figures of Figs. 13, 15, 18, 20, 21, 22, 23 and the corresponding description refer to 6 bits because the syntax element nuh_reserved_zero_6bits or layer_id_plus1 in the NAL unit header of Fig. 13 occupies 6 bits. However, all of the above description can be suitably modified if this syntax element used a number of bits other than 6 bits. For example, if this syntax element (nuh_reserved_zero_6bits or layer_id_plus1) used 9 bits instead, then in Fig. 17, the value of max_num_layer_minus1 in bits would be 9, and scalability_map[i] would be signaled for each of the 9 bits instead of 6 bits.

[0322] With reference to Fig. 24, a modification of Fig. 22 provides a syntax for signaling layer dependency information. A new syntax element layer_dependency_information_pattern is defined.

[0323] The layer_dependency_information_pattern element signals a combination of 0 and 1 bits having a length equal to vps_max_layers_minus1. A value of 0 for the i-th bit specifies that the layer with layer_id(i+1) is an independent layer. A value of 1 for the i-th bit specifies that the layer with layer_id(i+1) is a dependent layer that depends on one or more other layers.

[0324] The value of NumDepLayers is equal to the sum of the number of bits in layer_dependency_information_pattern that have a value of 1. Thus

[0325] [Mathematical expression 11]

[0326] With reference to FIG. 29, a modification of FIG. 26 provides a syntax for signaling layer dependency information. A new syntax element layer_dependency_flag[i] is defined. The element layer_dependency_flag[i] signals whether a layer depends on other layers. A value of 0 for the flag specifies that the layer with layer_id (corresponding to) i is an independent layer. A value of 1 for the i-th bit specifies that the layer with layer_id i is a dependent layer.

[0327] With reference to FIG. 30, a modification of FIG. 26 provides a syntax for signaling layer dependency information. A new syntax element layer_dependency_map[i] is defined. The element layer_dependency_map[i] signals a combination of 0 and 1 bits having a length equal to vps_max_layers_minus1. A value of 0 for the k-th bit of layer_dependency_map[i] specifies that layer i does not depend on layer with layer_id(k+1). A value of 1 for the k-th bit of layer_dependency_map[i] specifies that layer i depends on layer with layer_id(k+1).

[0328] With reference to Fig. 31, the modification of Fig. 26 provides a syntax for signaling layer dependency information. A new syntax element layer_dependency_information_pattern is defined.

[0329] The layer_dependency_information_pattern element signals a combination of 0 and 1 bits having length vps_max_layers_minus1. A value of 0 for the i-th bit specifies that the layer with layer_id(i+1) is an independent layer. A value of 1 for the i-th bit specifies that the layer with layer_id(i+1) is a dependent layer that depends on one or more other layers. The value of NumDepLayers is equal to the sum of the number of bits in layer_dependency_information_pattern that have a value of 1. Thus

[0330] [Mathematical expression 12]

[0331] The element layer_dependency_map[i] signals a combination of 0 and 1 bits having length vps_max_layers_minus1. A value of 0 for the k-th bit of layer_dependency_map[i] specifies that layer i does not depend on the layer with layer_id(k+1). A value of 1 for the k-th bit of layer_dependency_map[i] specifies that layer i depends on the layer with layer_id(k+1).

[0332] With reference to Fig. 32, a modification of Fig. 26 provides a syntax for signaling layer dependency information. Fig. 28 provides another syntax based on the syntax of Fig. 27. A new syntax element layer_dependency_information_pattern is defined.

[0333] The layer_dependency_information_pattern element signals a combination of 0 and 1 bits having length vps_max_layers_minus1. A value of 0 for the i-th bit specifies that the layer with layer_id(i+1) is an independent layer. A value of 1 for the i-th bit specifies that the layer with layer_id(i+1) is a dependent layer that depends on one or more other layers.

[0334] The value of NumDepLayers is equal to the sum of the number of bits in layer_dependency_information_pattern that have the value 1.

[0335] [Mathematical expression 13]

[0336] The syntax elements num_direct_ref_layers[i] and ref_layer_id[i][j] signal only if layer_dependency_information_pattern(i) has the value 1. Where layer_depdndency_information_pattern(i) is the i-th bit of the layer_dependency_pattern syntax element.

[0337] With reference to Fig. 33, a modification of Fig. 26 provides a syntax for signaling layer dependency information. The figure of Fig. 29 is another syntax based on the syntax of Fig. 31. A new syntax element layer_dependency_information_pattern is defined.

[0338] The layer_dependency_information_pattern element signals a combination of 0 and 1 bits having length vps_max_layers_minus1. A value of 0 for the i-th bit specifies that the layer with layer_id(i+1) is an independent layer. A value of 1 for the i-th bit specifies that the layer with layer_id(i+1) is a dependent layer that depends on one or more other layers.

[0339] The value of NumDepLayers is equal to the sum of the number of bits in layer_dependency_information_pattern that have a value of 1. Thus

[0340] [Mathematical expression 14]

[0341] The element layer_dependency_map[i] signals a combination of 0 and 1 bits having length vps_max_layers_minus1. A value of 0 for the k-th bit of layer_dependency_map[i] specifies that layer i does not depend on the layer with layer_id(k+1). A value of 1 for the k-th bit of layer_dependency_map[i] specifies that layer i depends on the layer with layer_id(k+1). Syntax elements layer_dependency_map[i] only signal if layer_dependency_information_pattern(i) has the value 1. Where layer_depdndency_information_pattern(i) is the i-th bit of the syntax element layer_dependency_pattern.

[0342] In another implementation, the layer_dependency_information_pattern syntax element may be signaled as a set of 1-bit flag values. In this case, a total of vps_max_layers_minus1 1-bit values would be signaled as:

[0343] [Table 9]

[0344] In another implementation, the layer_dependency_map[i] syntax element may be signaled as a set of 1-bit flag values. In this case, the total number of vps_max_layers_minus1 1-bit values would be signaled as:

[0345] [Table 10]

[0346] In another implementation, one or more elements of the layer_dependency_information_pattern, layer_dependency_map syntax elements may be signaled using a known constant number of bits instead of u(v). For example, they may be signaled using u(64).

[0347] In another implementation, one or more elements of the multiple syntax elements layer_dependency_information_pattern, layer_dependency_map may be signaled using ue(v) or some other encoding scheme.

[0348] In another implementation, the names of various syntactic elements and their semantics may be changed by adding plus1 or plus2 or subtracting minus1 or minus2 compared to the described syntax and semantics.

[0349] In another implementation, various syntax elements such as layer_dependency_information_pattern, layer_dependency_map, layer_dependency_flag[i], etc. may be signaled on a per-picture basis somewhere in the bitstream. For example, this may be signaled in a slice header, pps/ sps/ vps/ aps, or any other set of parameters or other normative part of the bitstream.

[0350] As previously described, scalable video coding is a method of coding a video bitstream that also includes one or more subset bitstreams. A subset of a video bitstream may be obtained by discarding packets from a larger video data to reduce the bandwidth required for the subset bitstream. The subset bitstream may represent a video signal of lower spatial resolution (smaller screen), lower temporal resolution (lower frame rate), or lower quality. For example, a video bitstream may include 5 subset bitstreams, where each of the subset bitstreams adds additional content to the base bitstream. The document by Hannuksela, et al., "Test Model for Scalable Extensions of High Efficiency Video Coding (HEVC)" in JCTVC-L0453, Shanghai, October 2012, is hereby incorporated by reference in its entirety. The paper by Chen, et al., “SHVC Draft Text 1,” in JCTVC-L1008, Geneva, March 2013, is hereby incorporated by reference in its entirety. The paper by Wang, et al., “AHG9: On VPS and SPS in HEVC 3DV and scalable extensions,” in JCTVC-M0268, Incheon, April 2013, is hereby incorporated by reference in its entirety.

[0351] As previously described, multi-view video coding is a method of coding a video bitstream that also contains one or more other bitstreams representing alternative views. For example, the multiple views may be a view pair for stereoscopic video. For example, the multiple views may represent multiple views of the same scene from different viewpoints. Multiple views typically contain a large number of inter-view statistical dependencies since the images are the same scene from different viewpoints. Therefore, combined temporal and inter-view prediction can provide efficient multi-view coding. For example, a frame can be efficiently predicted not only from temporally related frames, but also from frames from neighboring viewpoints. The document by Hannuksela, et al., “Common specification text for scalable and multi-view extensions,” JCTVC-L0452, Geneva, January 2013, is hereby incorporated by reference in its entirety. The document Tech et al., “MV-HEVC Draft Text 3 (ISO/IEC 23008-2:201x/PDAM2)”, JCT3V-C1004_d3, Geneva, January 2013, is hereby incorporated by reference in its entirety.

[0352] Chen, et al., “SHVC Draft Text 1,” JCTVC-L1008, Geneva, January 2013; Hannuksela, et al., “Test Model for Scalable Extensions of High Efficiency Video Coding (HEVC),” JCTVC-L0453-spec-text, Shanghai, October 2012; and Hannuksela, “Draft Text for Multiview Extension of High Efficiency Video Coding (HEVC),” JCTVC-L0452-spec-text-rl, Shanghai, October 2012; each of which is incorporated by reference in its entirety. JCTVC-L0452 and JCTVC-L0453 each have the concept of output operation points. This was later changed to the concept of output level sets in JCTVC-L1008. With respect to syntax elements, the num_output_operation_points, output_op_point_index[], and output_layer_flag[][] elements are defined for output operation points in JCTVC-L0453 and LCTVC-L0452, and modified into num_output_layer_sets, output_layer_set_idx[], and output_layer_flag[][] in JCTVC-L1008. Each of these documents defines a layer dependency change SEI message. The SEI message allows signaling changes in layer dependency information starting with the current access block containing the SEI message.

[0353] In Chen, et al., "SHVC Draft Text 1", JCTVC-L1008, Geneva, January 2013; Hannuksela, et al. "Test Model for Scalable Extensions of High Efficiency Video Coding (HEVC)", JCTVC-L0453-spec-text, Shanghai, October 2012; and Hannuksela, “Draft Text for Multiview Extension of High Efficiency Video Coding (HEVC)”, JCTVC-L0452-spec-text-rl, Shanghai, October 2012; each of which is incorporated by reference in its entirety, each describes a decoded picture buffer (DPB) in output order that operates by using the syntax elements sps_max_num_reorder_pics[HighestTid], sps_max_latency_increase_plus1[HighestTid], and sps_max_dec_pic_buffering[HighestTid] to output and discard images 0 from the DPB. This information is signaled in the video parameter set for the base layer, which provides buffering information for the video content, including enhancement levels, if any.

[0354] With reference to Fig. 34, another example of a video parameter set is illustrated. In particular, the video parameter set of Fig. 34 includes a reference to an associated video parameter set extension (vps_extension). The vps_extension element can be signaled using vps_extension_flag, vps_extension2_flag, and/or vps_extension_data_flag.

[0355] With reference to Fig. 35, another example of a video parameter set extension is illustrated. In particular, the video parameter set extension of Fig. 35 includes a reference to a num_output_layer_sets element that provides a definition of those layers from a set of bitstream layers that can be output by a decoder to a viewer. The num_output_layer_sets element specifies the number of layer sets for which output layers are specified using output_layer_set_index[i] and output_layer_flag[lsIdx][j]. If not present, the value of num_output_layer_sets is implied to be 0. A layer set describing the layers to be output is an output layer set.

[0356] The output_layer_set_idx[i] element specifies the index lsIdx of the layer set for which output_layer_flag[lsIdx][j] is present.

[0357] An output_layer_flag[lsIdx][j] of 1 specifies that the layer with nuh_layer_id equal to j is the target output layer for the lsIdxth layer set. An output_layer_flag[lsIdx][j] of 0 specifies that the layer with nuh_layer_id equal to j is not the target output layer for the lsIdxth layer set.

[0358] With num_output_layer_sets defined in vps_extension as described, num_output_layer_sets can be updated by sending a new video parameter set together with the corresponding video parameter set extension. Unfortunately, sending a new video parameter set together with the corresponding video parameter set extension results in a significant reduction in the available bandwidth. In addition, the new video parameter set can only be activated in some image types, such as intra-frame random access point image types.

[0359] As an example, a bitstream may include a base layer 0 and four enhancement layers, namely, enhancement layer 1, enhancement layer 2, enhancement layer 3, and enhancement layer 4. The first set of layers may be a base layer 0 and enhancement layer 1. The second set of layers may be a base layer 0, enhancement layer 1, and enhancement layer 2. The third set of layers may be a base layer 0, enhancement layer 1, and enhancement layer 3. The fourth set of layers may be a base layer 0 and enhancement layer 4. The sets of output layers define specific layers in the set of layers that may be provided as output layers. For example, set 1 of output layers may be that layer 0 is outputtable and layer 1 is not outputtable. For example, set 2 of output layers may be that layer 0 is outputtable, layer 1 is outputtable, and layer 2 is not outputtable. For example, a set of 3 output levels may be that level 0 is not outputtable, level 1 is not outputtable, and level 3 is outputtable. For example, a set of 4 output levels may be that level 0 is outputtable and level 4 is not outputtable. A limited number of output levels is useful to accommodate limited decoding capabilities. In some cases, it is desirable to prohibit some dependencies between different levels to ensure decoding of other levels and/or decoder capabilities. When some level dependencies change, it may similarly be desirable to enable, disable, and/or add other levels as output levels. As an example, a change in level dependency may require that fewer levels be decoded in order to decode a particular target level (e.g., a view in MV-HEVC). In this way, a decoder can decode and output a level (e.g., and an additional view) after a level dependency change that it previously could not due to a hardware limitation. There is currently no mechanism to allow signaling of changes in output level sets without sending a new set of video parameters.

[0360] Referring to Fig. 36, an SEI message syntax may be used to facilitate a change of output layer sets, namely, output_layer_sets_change(payloadSize). This SEI message indicates that output layer set information is changed starting with the current access unit containing the SEI message and is interpreted with respect to the active video parameter set. More than one video parameter set may be included with the bitstream, and the active video parameter set is changed as needed. When present, the output layer set change SEI message is applied to the target set of access units, which consists of the current access unit and all subsequent access units in decoding order, until any subsequent output layer set change SEI message or the end of the coded video sequence is earlier in decoding order. The output layer set change SEI messages preferably do not have a cumulative effect.

[0361] The 'active_vps_id' element identifies the active video parameter set that contains the output level set information to which the modification or addition was applied. The value of active_vps_id is equal to the value of video_parameter_set_id of the active video parameter set for the VCL NAL units in the access unit containing the SEI message.

[0362] The 'num_changed_output_layer_sets' element specifies the number of changed output layer sets for which output layers are specified in the vps extension section of the active video parameter set identified by active_vps_id. The value of num_changed_output_layer_sets must be in the range from 0 to num_output_layer_sets, inclusive. If not present, the value of num_changed_output_layer_sets is implied to be 0.

[0363] The 'changed_output_layer_sets_idx_entry'[i] element identifies the index entry in the list of output layer set entries in the vps extension identified by active_vps_id for which the change in output_layer_flag[clsIdx][j] applies. The value of changed_output_layer_set_idx_entry[i] must be in the range 0 to num_output_layer_sets, inclusive. changed_output_layer_sets_idx_entry[i] is not equal to changed_output_layer_sets_idx_entry[j] for any j not equal to i.

[0364] An 'output_layer_flag'[clsIdx][j] of 1 specifies that the layer with nuh_layer_id equal to j is the target output layer of the clsIdxth layer set. An output_layer_flag[clsIdx][j] of 0 specifies that the layer with nuh_layer_id equal to j is not the target output layer for the clsIdxth layer set.

[0365] If output_layer_flag[clsIdx][j] is not present for clsIdx in the range 0 to vps_num_layer_sets_minus1, inclusive, and for j in the range 0 to 63, inclusive, output_layer_flag[clsIdx][j] is implied to be equal to (j == LayerSetLayerIdList[clsIdx] [NumLayersInIdList[clsIdx] - 1]).

[0366] The 'num_addl_output_layer_sets' member specifies the number of additional layer sets for which output layers are specified using addl_output_layer_set_idx[i] and output_layer_flag[addllsIdx][j]. If not present, the value of num_addl_output_layer_sets is implied to be 0. The 'addl_output_layer_sets_idx'[i] member identifies the addllsIdx index of the layer set for which output_layer_flag[addllsIdx][j] is present.

[0367] An 'output_layer_flag'[addllsIdx][j] of 1 specifies that the layer with nuh_layer_id equal to j is the target output layer of the addllsIdxth layer set. An output_layer_flag[addllsIdx][j] of 0 specifies that the layer with nuh_layer_id equal to j is not the target output layer of the addllsIdxth layer set.

[0368] When output_layer_flag[addllsIdx][j] is not present for addllsIdx in the range 0 to vps_num_layer_sets_minus1, inclusive, and for j in the range 0 to 63, inclusive, output_layer_flag[addllsIdx][j] is implied to be equal to (j == LayerSetLayerIdList[addllsIdx][NumLayersInIdList[addllsIdx]- 1]).

[0369] When output_layer_flag[addllsIdx][j] is not present for addllsIdx in the range 0 to vps_num_layer_sets_minus1, inclusive, and for j in the range 0 to 63, inclusive, and when output_layer_flag[clsIdx][j] is not present for clsIdx in the range 0 to vps_num_layer_sets_minus1, inclusive, and for j in the range 0 to 63, inclusive, output_layer_flag[addllsIdx][j] is implied to be equal to (j == LayerSetLayerIdList[addllsIdx][NumLayersInIdList[addllsIdx]-1]).

[0370] When output_layer_flag[addllsIdx][j] is not present for addllsIdx in the range 0 to vps_num_layer_sets_minus1, inclusive, and for j in the range 0 to 63, inclusive, and when output_layer_flag[clsIdx][j] is not present for clsIdx in the range 0 to vps_num_layer_sets_minus1, inclusive, and for j in the range 0 to 63, inclusive, output_layer_flag[clsIdx][j] is implied to be equal to (j == LayerSetLayerIdList[clsIdx][NumLayersInIdList[clsIdx] - 1]).

[0371] A target list of output layers may be obtained if desired. The target list of output layer identifiers, targetOpLayerIdList, specifies a list of nuh_layer_id values, in ascending order, of the nuh_layer_id values that are output layers for the selected set of output layers, may be obtained as follows:

[0372] For a selected set of output layers, oLsIdx, the target list of output layer identifiers, targetOpLayerIdList, is obtained as:

[0373] [Table 11]

[0374] The target set of output layers may be identified by a linked list of target output layer identifiers, targetOpLayerIdList.

[0375] The selected (index) oLsIdx of the output layer set is between 0 and num_output_layer_sets if there are no SEI messages about changing output layer sets.

[0376] The selected output layer set oLsIdx is between 0 and num_output_layer_sets+num_addl_output_layer_sets when an output layer set change SEI message is present.

[0377] A target list of decoded layers may be obtained if desired. A target list of decoded layer identifiers, targetDLayerIdList specifies a list of nuh_layer_id values to be decoded for the selected set of output layers, may be obtained as follows:

[0378] For the oLsIdx selected output layer set, the target list of output layer identifiers, targetOpLayerIdList, is obtained as:

[0379] [Table 12]

[0380] The target list of decoded layer identifiers targetDLayerIdList can be derived as:

[0381] [Table 13]

[0382] In some case, the final targetDLayerIdList can be obtained by sorting the above targetDLayerIdList in ascending order of nuh_layer_id values.

[0383] The target set of decoded layers is identified by an associated target list of decoded layer identifiers targetDLayerIdList. In some cases, targetDLayerIdList may be the same as the list of layer identifiers TargetDecLayerIdList from the JCTVC-L1008 document, which describes a list of nuh_layer_id values in ascending order of the nuh_layer_id values for the NAL units to be decoded:

[0384] The oLsIdx value of the selected output layer set is between 0 and num_output_layer_sets if there are no SEI messages about changing output layer sets.

[0385] The oLsIdx value of the selected output layer set is between 0 and num_output_layer_sets+num_addl_output_layer_sets when an output layer set change SEI message is present.

[0386] Referring to Fig. 37, another SEI message syntax may be used to facilitate changing output layer sets, namely, output_layer_sets_change(payloadSize).

[0387] The 'num_deleted_output_layer_sets' element specifies the number of output layer sets that have been deleted and are therefore no longer present. The value of num_deleted_output_layer_sets is in the range from 0 to num_output_layer_sets, inclusive. If not present, the value of num_deleted_output_layer_sets is assumed to be 0.

[0388] The 'deleted_output_layer_sets_idx_entry'[i] element identifies the entry at the index in the output layer set list that is specified to be deleted and is therefore no longer present. The value of deleted_output_layer_set_idx_entry[i] must be in the range 0 to num_output_layer_sets, inclusive. The value of deleted_output_layer_sets_idx_entry[i] is not equal to deleted_output_layer_sets_idx_entry[j] for any j not equal to i.

[0389] The 'num_changed_output_layer_sets' element specifies the number of changed output layer sets for which output layers are specified in the vps extension section of the active video parameter set identified by active_vps_id. The value of num_changed_output_layer_sets must be in the range from 0 to num_output_layer_sets - num_deleted_output_layer_sets, inclusive. If not present, the value of num_changed_output_layer_sets is implied to be 0.

[0390] A target list of output layers may be obtained if desired. The target list of output layer identifiers, targetOpLayerIdList, specifies a list of nuh_layer_id values, in ascending order, of the nuh_layer_id values that are output layers for the selected set of output layers, may be obtained as follows:

[0391] For a selected set of output layers, oLsIdx, the target list of output layer identifiers targetOpLayerIdList is obtained as:

[0392] [Table 14]

[0393] The target set of output layers is identified by a linked list of target output layer identifiers targetOpLayerIdList.

[0394] The oLsIdx value of the selected output layer set is between 0 and num_output_layer_sets if there are no SEI messages about changing output layer sets.

[0395] The oLsIdx value of the selected output layer set is between 0 and num_output_layer_sets - num_deleted_output_layer_sets+num_addl_output_layer_sets, when an output layer set change SEI message is present.

[0396] A target list of decoded levels may be obtained if desired.

[0397] The target list of decodable layer identifiers, TargetDLayerIdList, specifies a list of nuh_layer_id values to be decoded for the selected set of output layers, and can be obtained as follows:

[0398] For the oLsIdx selected output layer set, the target list of output layer identifiers, targetOpLayerIdList, can be obtained as:

[0399] [Table 15]

[0400] Then the target list of decoded layer identifiers targetDLayerIdList can be obtained as:

[0401] [Table 16]

[0402] In some case, the final targetDLayerIdList can be obtained by sorting the above targetDLayerIdList in ascending order of nuh_layer_id values.

[0403] The target set of decoded layers is identified by an associated target list of decoded layer identifiers targetDLayerIdList. In some case, targetDLayerIdList may be the same as the list of layer identifiers TargetDecLayerIdList from the JCTVC-L1008 document, which describes a list of nuh_layer_id values, in ascending order of the nuh_layer_id values, for the NAL units to be decoded:

[0404] The oLsIdx value of the selected output layer set is between 0 and num_output_layer_sets if there are no SEI messages about changing output layer sets.

[0405] The oLsIdx value of the selected output layer set is between 0 and num_output_layer_sets - num_deleted_output_layer_sets+num_addl_output_layer_sets, when an output layer set change SEI message is present.

[0406] The output layer set change SEI message is defined such that its effect is not cumulative. In another design, the syntax and/or semantics of this message may be defined such that they are cumulative. In this case, the decoder (and encoder) may keep track of all previous values of num_output_layer_sets (from the vps extension) and/or num_deleted_output_layer_sets and/or num_changed_output_layer_sets and/or num_addl_output_layer_sets and may accumulate these changes each time a new SEI message is signaled.

[0407] In another implementation, one or more of the syntax elements may be signaled using a known fixed number of bits instead of u(v) instead of ue(v). For example, they may be signaled using u(8) or u(16) or u(32) or u(64), etc.

[0408] In another implementation, one or more of these syntactic elements may be signaled using ue(v) or some other encoding scheme instead of a fixed number of bits, such as u(v) encoding.

[0409] In another implementation, the names of various syntactic elements and their semantics may be changed by adding plus1 or plus2 or subtracting minus1 or minus2 compared to the described syntax and semantics.

[0410] In another embodiment, the various syntax elements included in the output level set SEI message may be signaled elsewhere in the bitstream on a per-picture basis or at another frequency. For example, they may be signaled in a slice segment header, a pps/sps/vps/adaptation parameter set, or any other parameter set or other normative part of the bitstream.

[0411] In another embodiment, various syntax elements may be signaled elsewhere in the bitstream per picture or at another frequency. For example, they may be signaled in a slice segment header, a pps/sps/vps/adaptation parameter set, or any other parameter set or other normative part of the bitstream.

[0412] In yet other embodiments, all concepts defined in this invention relating to output level sets may be applied to output operating points as defined in documents JCTVC-L0452 and JCTVC-L0453 and/or to operating points as defined in document JCTVC-L1003.

Example 2

[0413] Another example of the present invention is described below. It should be noted that, for convenience, elements having functions identical to those of corresponding elements illustrated in the First Example are given corresponding identical reference numerals, and description of these elements is omitted here.

[0414] It has been found that signaling the order of output of the decoded picture buffer (DPB) based on the use of the syntax elements sps_max_num_reorder_pics[HighestTid], sps_max_latency_increase_plus1[HighestTid], and sps_max_dec_pic_buffering[HighestTid] for outputting and removing pictures from the DPB does not take into account characteristics of the buffer that may be caused by scalable video coding, such as when different numbers of enhancement levels are used, which tend to change after the content has been encoded based on user viewing preferences, and multi-view enhancement levels, which tend to change after the content has been encoded based on user viewing preferences. It has also been determined that signaling the output order of the decoded picture buffer (DPB) based on the use of the sps_max_num_reorder_pics[HighestTid], sps_max_latency_increase_plus1[HighestTid], and sps_max_dec_pic_buffering[HighestTid] syntax elements for outputting and removing pictures from the DPB may not be optimal in terms of DPB memory usage when the decoder is operating at a certain operating point and/or outputting a selected set of output levels. To accommodate such differences in viewing preferences, the output order of the decoded picture buffer (DPB) may additionally and/or alternatively be based on such syntax elements included with the video parameter set extension (VPS extension) to provide syntax elements for one or more levels of the enhancement levels. In this way, syntactic elements can be selected to be particularly appropriate for a particular workpoint or set of output levels, which is intended to match the user's viewing preferences.

[0415] The DPB buffering related parameters vps_max_dec_pic_buffering_minus1, vps_max_num_reorder_pics, vps_max_latency_increase_plus1 may be signaled for sublayers for CVS for one or more operation points and/or for output layer sets in the VPS extension. Similarly, the system may determine the operating mode and flushing process for the output queue DPB to use the above signaled DPB buffering parameters from the VPS extension if they are signaled for the operation point being tested or for the selected output layer set. Otherwise, the corresponding SPS layer parameters from the active SPS (when the currLayerId that corresponds to the nuh_layer_id of the current picture is 0) or from the SPS of the active layer are used depending on the layer_id of the current layer. In some cases, the parameters vps_max_dec_pic_buffering_minus1, vps_max_num_reorder_pics, vps_max_latency_increase_plus1 may be called max_vps_dec_pic_buffering_minus1, max_vps_num_reorder_pics, max_vps_latency_increase_plus1.

[0416] In another case, the DPB buffering-related parameter max_vps_dec_pic_buffering_minus1 is signaled for sublayers for CVS for one or more sets of output layers in the VPS extension. The sets of output layers may correspond to operating points. The DPB buffering-related parameters max_vps_num_reorder_pics, max_vps_latency_increase_plus1 may be signaled for sublayers for CVS for each layer in the VPS extension. Similarly, the system may specify the operation and flush process for the output queue DPB to use the above signaled DPB buffering parameters from the VPS extension if they are signaled for the operating point being tested or for the selected set of output layers. Otherwise, the corresponding SPS layer parameters from the active SPS (when the currLayerId that corresponds to the nuh_layer_id of the current picture is 0) or from the SPS of the active layer are used depending on the layer_id of the current layer.

[0417] Referring to Fig. 38A, an example of a modified vps_extension is illustrated. The modified vps extension includes a new syntax element, namely, num_op_dpb_info_parameters and operation_point_layer_set_idx[i]. This modified vps extension can be defined in terms of an operation point, which is a bitstream created from another bitstream by an action of a sub-bitstream extraction process using the other bitstream, a target high TemporalId, and a target list of layer identifiers as input.

[0418] The num_output_layer_sets element specifies the number of layer sets for which output layers are specified by output_layer_set_index[i] and output_layer_flag[lsIdx][j]. If not present, the value of num_output_layer_sets is implied to be 0. The layer set that describes the layers to be output is an output layer set.

[0419] The output_layer_set_idx[i] element specifies the index lsIdx of the layer set for which output_layer_flag[lsIdx][j] is present.

[0420] An output_layer_flag[lsIdx][j] of 1 specifies that the layer with nuh_layer_id equal to j is the target output layer of the lsIdxth layer set. An output_layer_flag[lsIdx][j] of 0 specifies that the layer with nuh_layer_id equal to j is not the target output layer of the lsIdxth layer set.

[0421] The num_op_dpb_info_parameters element specifies the number of op_dpb_parameters() syntax structures present in the VPS extension RBSP, specified in terms of an operation point. The value of num_op_dpb_info_parameters for decoders is in the range 0 to vps_num_layer_sets_minus1, inclusive.

[0422] The operation_point_layer_set_idx[i] element specifies an index into the list of layer sets defined according to the operation points to which the i-th op_dpb_info_parameters() syntax structure in the VPS extension applies. The value of operation_point_layer_set_idx[i] can be in the range from 0 to vps_num_layer_sets_minus1, inclusive. For bitstream conformance, operation_point_layer_set_idx[i] is not equal to operation_point_layer_set_idx[j] for any j not equal to i.

[0423] Referring to FIG. 39A, the op_dpb_info_parameters element specifies vps_max_sub_layers_minus1[j], vps_sub_layer_ordering_info_present_flag[j], vps_max_dec_pic_buffering_minus1[j][k], vps_max_num_reorder_pics[j][k] and vps_max_latency_increase_plus1[ j][k].

[0424] The vps_max_sub_layers_minus1[j] element plus 1 specifies how many sublayers are included. The vps_max_sub_layers_minus1[j] element plus 1 specifies the maximum number of temporary sublayers that can be present in CVS for a layer with nuh_layer_id equal to j. The value of vps_max_sub_layers_minus1[j] is in the range 0 to 6, inclusive.

[0425] The vps_sub_layer_ordering_info_present_flag[j] element specifies whether the syntax is for a single set, including all layers, or for each individual layer. A vps_sub_layer_ordering_info_present_flag[j] element of 1 specifies that vps_max_dec_pic_buffering_minus1[j][k], vps_max_num_reorder_pics[j][k], and vps_max_latency_increase_plus1[j][k] are present for the layer with nuh_layer_id equal to j for vps_max_sub_layers_minus1[j]+1 sublayers. A vps_sub_layer_ordering_info_present_flag[j] value of 0 specifies that the values of vps_max_dec_pic_buffering_minus1[j][vps_max_sub_layers_minus1[j]], vps_max_num_reorder_pics[j][vps_max_sub_layers_minus1[j]], and vps_max_latency_increase_plus1[j][vps_max_sub_layers_minus1[j]] apply to all sublayers for the layer with nuh_layer_id equal to j.

[0426] The vps_max_dec_pic_buffering_minus1[j][k] element plus 1 specifies the maximum required decoded picture buffer size for CVS for the layer with nuh_layer_id equal to j, in units of picture storage buffers when HighestTid is k. The value of vps_max_dec_pic_buffering_minus1[j][k] shall be in the range from 0 to MaxDpbSize - 1 (as defined in subclause A.4), inclusive. When k is greater than 0, vps_max_dec_pic_buffering_minus1[j][k] shall be greater than or equal to vps_max_dec_pic_buffering_minus1[j][k-1]. When vps_max_dec_pic_buffering_minus1[j][k] is not present for k in the range 0 to vps_max_sub_layers_minus1[j] - 1, inclusive, due to vps_sub_layer_ordering_info_present_flag[j] being equal to 0, it is implied to be equal to vps_max_dec_pic_buffering_minus1[j][vps_max_sub_layers_minus1[j].

[0427] The vps_max_num_reorder_pics[j][k] element specifies the maximum number of images that may precede any image in the CVS for a layer with nuh_layer_id equal to j in decoding order and follow that image in output order when HighestTid equals k. The value of vps_max_num_reorder_pics[j][k] must be in the range from 0 to vps_max_dec_pic_buffering_minus1[j][k], inclusive. When k is greater than 0, vps_max_num_reorder_pics[j][k] is greater than or equal to vps_max_num_reorder_pics[j][k-1]. When vps_max_num_reorder_pics[j][k] is not present for k in the range 0 to vps_max_sub_layers_minus1[j] - 1, inclusive, due to vps_sub_layer_ordering_info_present_flag[j] being equal to 0, it is implied to be equal to vps_max_num_reorder_pics[j][vps_max_sub_layers_minus1[j]].

[0428] The vps_max_latency_increase_plus1[j][k] element, when not equal to 0, is used to calculate the value of VpsMaxLatencyPictures[j][k], which specifies the maximum number of pictures that can precede any picture in the CVS for a layer with nuh_layer_id equal to j in output order and follow that picture in decoding order when HighestTid equals k.

[0429] When vps_max_latency_increase_plus1[j][k] is not 0, the value of VpsMaxLatencyPictures[j][k] can be set as follows:

[0430] VpsMaxLatencyPictures[j][k]=vps_max_num_reorder_pics[j][k]+vps_max_latency_increase_plus1[j][k] - 1

[0431] When vps_max_latency_increase_plus1[j][k] is 0, the corresponding limit is not expressed.

[0432] The value of vps_max_latency_increase_plus1[j][k] is in the range 0 to 2 ³² - 2, inclusive. When vps_max_latency_increase_plus1[j][k] is not present for k in the range 0 to vps_max_sub_layers_minus1[j] - 1, inclusive, due to vps_sub_layer_ordering_info_present_flag[j] being 0, it is assumed to be vps_max_latency_increase_plus1[j][vps_max_sub_layers_minus1[j]].

[0433] The 'vps_max_sub_layers_minus'[id][j] element plus 1 specifies the maximum number of temporary sublayers that may be present in CVS for the layer with nuh_layer_id equal to j, for the workpoint associated with index id. The value of vps_max_sub_layers_minus1[id][j] must be in the range 0 to 6, inclusive.

[0434] Element 'vps_sub_layer_ordering_info_present_flag'[id][j] equal to 1 specifies that vps_max_dec_pic_buffering_minus1[id][j][k], vps_max_num_reorder_pics[id][j][k], and vps_max_latency_increase_plus1[id][j][k] are present for layer with nuh_layer_id equal to j for the workpoint associated with index id for vps_max_sub_layers_minus1[id][j]+1 sublayers.

[0435] A vps_sub_layer_ordering_info_present_flag[id][j] element of 0 specifies that the values of vps_max_dec_pic_buffering_minus1[id][j][vps_max_sub_layers_minus1[id][j]], vps_max_num_reorder_pics[id][j][vps_max_sub_layers_minus1[id][j]], and vps_max_latency_increase_plus1[id][j][vps_max_sub_layers_minus1[id][j]] apply to all sublayers for the layer with nuh_layer_id equal to j for the workpoint associated with index id.

[0436] The vps_max_dec_pic_buffering_minus1[id][j][k] element plus 1 specifies the maximum required decoded picture buffer size for CVS for the layer with nuh_layer_id equal to j for the work point associated with the index identifier, in units of picture storage buffers when HighestTid is k. The value of vps_max_dec_pic_buffering_minus1[id][j][k] shall be in the range from 0 to MaxDpbSize - 1 (as defined in subclause 4), inclusive. When k is greater than 0, vps_max_dec_pic_buffering_minus1[id][j][k] shall be greater than or equal to vps_max_dec_pic_buffering_minus1[id][j][k-1]. If the vps_max_dec_pic_buffering_minus1[id][j][k] element is not present for k in the range 0 to vps_max_sub_layers_minus1[id][j]-1, inclusive, due to vps_sub_layer_ordering_info_present_flag[id][j] being 0, it is implied to be equal to vps_max_dec_pic_buffering_minus1[id][j][vps_max_sub_layers_minus1[id][j]].

[0437] The 'vps_max_num_reorder_pics'[id][j][k] element specifies the maximum allowed number of pictures that may precede any picture in the CVS for layer with nuh_layer_id equal to j, for the workpoint associated with index id, in decoding order and follow that picture in output order when HighestTid is k. The value of vps_max_num_reorder_pics[id][j][k] must be in the range from 0 to vps_max_dec_pic_buffering_minus1[id][j][k], inclusive. When k is greater than 0, vps_max_num_reorder_pics[id][j][k] will be greater than or equal to vps_max_num_reorder_pics[id][j][k-1]. If the vps_max_num_reorder_pics[id][j][k] element is not present for k in the range 0 to vps_max_sub_layers_minus1[id][j]-1, inclusive, due to vps_sub_layer_ordering_info_present_flag[id][j] being equal to 0, it is implied to be equal to vps_max_num_reorder_pics[id][j][vps_max_sub_layers_minus1[id][j]].

[0438] The 'vps_max_latency_increase_plus1[id][j][k] member, if not equal to 0, is used to calculate the value of VpsMaxLatencyPictures[id][j][k], which specifies the maximum number of pictures that can precede any picture in the CVS for the layer with nuh_layer_id equal to j, for the workpoint associated with index id, in output order and follow that picture in decoding order when HighestTid equals k.

[0439] When vps_max_latency_increase_plus1[id][j][k] is not 0, the value of VpsMaxLatencyPictures[id][j][k] is set as follows:

[0440] VpsMaxLatencyPictures[id][j][k]=vps_max_num_reorder_pics[id][j][k]+vps_max_latency_increase_plus1[id][j][k] - 1

[0441] When vps_max_latency_increase_plus1[id][j][k] is 0, the corresponding limit is not expressed.

[0442] The value of vps_max_latency_increase_plus1[id][j][k] must be in the range 0 to 2 ³² - 2, inclusive. When vps_max_latency_increase_plus1[id][j][k] is not present for k in the range 0 to vps_max_sub_layers_minus1[id][j]-1, inclusive, due to vps_sub_layer_ordering_info_present_flag[id][j] being equal to 0, it is implied to be equal to vps_max_latency_increase_plus1[id][j][vps_max_sub_layers_minus1[id][j]].

[0443] Referring to Fig. 39B, op_dpb_info_parameters can be further modified as shown in relation to op_dpb_info_parameters(id,j). In this case, the syntax of the VPS extension can be as illustrated in Fig. 38B. A hypothetical reference decoder (HRD) is used to check the conformance of the bitstream and decoder conditions. Two types of bitstreams or subsets of the bitstream are subject to the HRD conformance check of the Joint Technicians Group on Video Coding (JCT-VC) standard. The first type, called the Type I bitstream, is a NAL unit stream containing only VCL NAL units and NAL units with nal_unit_type equal to FD_NUT (NAL units with filler data) for all access units in the bitstream. The second type, called a Type II bitstream, contains, in addition to the VCL NAL units and the filler data NAL units for all access units in the bitstream, at least one of (a) additional non-VCL NAL units other than the filler data NAL units, and (b) all leading_zero_8bits, zero_byte, start_code_prefix_one_3bytes, and trailing_zero_8bits syntax elements that form a byte stream from a stream of NAL units.

[0444] The syntax elements of non-VCL NAL units (or their default values for some of the syntax elements) required for HRD are defined in the semantics subclauses in clause 7 of Appendices D and E.

[0445] Two types of HRD parameter sets are used (NAL HRD parameters and VCL HRD parameters). HRD parameter sets are signaled via the hrd_parameters() syntax structure, which may be part of the SPS syntax structure or the VPS syntax structure.

[0446] Multiple tests may be required to verify the conformance of a bitstream, which is referred to as the test bitstream. For each test, the following steps are applied in the order given:

[0447]

(1) The test operating point, designated TargetOp, is selected. The OpLayerIdList of layer identifiers for TargetOp consists of a list of nuh_layer_id values, in ascending order of the nuh_layer_id values present in the subset of the bitstream associated with TargetOp, which is the subset of the nuh_layer_id values present in the bitstream under test. The OpTid for TargetOp is equal to the highest TemporalId present in the subset of the bitstream associated with TargetOp.

[0448] (2) TargetDecLayerIdList is set equal to OpLayerIdList of TargetOp, HighestTid is set equal to OpTid of TargetOp, and the process of extracting a sub-bitstream as specified in item 10 is initiated with the bitstream under test, HighestTid, and TargetDecLayerldList as input, and the output is assigned to BitstreamToDecode.

[0449] (3) Select an hrd_parameters() syntax structure and a sub_layer_hrd_parameters() syntax structure applicable to the TargetOp. If the TargetDecLayerIdList contains all nuh_layer_id values present in the bitstream being tested, select an hrd_parameters() syntax structure in the active SPS (or provided by an external means not described in this Description). Otherwise, select an hrd_parameters() syntax structure in the active VPS (or provided by some external means not described in this Description) that applies to the TargetOp. Within the selected hrd_parameters() syntax structure, if BitstreamToDecode is a Type I bitstream, the sub_layer_hrd_parameters(HighestTid) syntax structure that immediately follows the "if(vcl_hrd_parameters_present_flag)" condition is selected, and the NalHrdModeFlag variable is set to 0; otherwise (BitstreamToDecode is a Type II bitstream), the sub_layer_hrd_parameters(HighestTid) syntax structure that immediately follows either the "if(vcl_hrd_parameters_present_flag)" condition (in which case the NalHrdModeFlag variable is set to 0) or the "if(nal_hrd_parameters_present_flag)" condition is selected (in which case the NalHrdModeFlag variable is set to 1). When BitstreamToDecode is a Type II bitstream and NalHrd ModeFlag is 0, all non-VCL NAL units other than NAL units with filler data, and all leading_zero_8bits, zero_byte, start_code_prefix_one_3bytes, and trailing_zero_8bits syntax elements that form a byte stream from a NAL unit stream (as defined in Appendix B), if present, are discarded from BitstreamToDecode, and the remaining bitstream is assigned to BitstreamToDecode.

[0450] In another case, multiple tests may be required to verify the conformity of the bitstream, which is referred to as the test bitstream. For each test, the following steps are applied in the order given:

[0451] (1) A set of output layers, designated as TargetOpLs, is selected to be tested. The operation point referenced in TargetOpLs by output_layer_set_idx[] identifies the operation point to be tested. The list of output layer identifiers OpLayerIdList in TargetOpLs consists of a list of nuh_layer_id values, in ascending order of the nuh_layer_id values, present in the subset of the bitstream associated with TargetOp and TargetOpLs, which is a subset of the nuh_layer_id values present in the bitstream to be tested. OpTid in TargetOp is equal to the highest TemporalId present in the subset of the bitstream associated with TargetOp.

[0452] (2) TargetDecLayerIdList is set equal to the target list targetDLayerIdList of decoded layer identifiers for the selected set of TargetOpLs of output layers, HighestTid is set equal to the OpTid of the TargetOp, and the process of extracting the sub-bitstream as specified in paragraph 10 is initiated with the test bitstream, HighestTid, and TargetDecLayerIdList as input, and the output is assigned to BitstreamToDecode.

[0453] (3) Select an hrd_parameters() syntax structure and a sub_layer_hrd_parameters() syntax structure applicable to the TargetOp. If the TargetDecLayerIdList contains all nuh_layer_id values present in the bitstream being tested, select an hrd_parameters() syntax structure in the active SPS (or provided by an external means not described in this Description). Otherwise, select an hrd_parameters() syntax structure in the active VPS (or provided by some external means not described in this Description) that applies to the TargetOp. Within the selected hrd_parameters() syntax structure, if BitstreamToDecode is a Type I bitstream, the sub_layer_hrd_parameters(HighestTid) syntax structure that immediately follows the "if (vcl_hrd_parameters_present_flag)" condition is selected, and the NalHrdModeFlag variable is set to 0; otherwise (BitstreamToDecode is a Type II bitstream), the sub_layer_hrd_parameters(HighestTid) syntax structure that immediately follows either the "if (vcl_hrd_parameters_present_flag)" condition (in which case the NalHrdModeFlag variable is set to 0) or the "if (nal_hrd_parameters_present_flag)" condition (in which case the NalHrdModeFlag variable is set to 1) is selected. When BitstreamToDecode is a Type II bitstream and NalHrdModeFlag is 0, all non-VCL NAL units other than NAL units with filler data, and all leading_zero_8bits, zero_byte, start_code_prefix_one_3bytes, and trailing_zero_8bits syntax elements that form the byte stream from a NAL unit stream (as defined in Appendix B), if present, are discarded from BitstreamToDecode, and the remaining bitstream is assigned to BitstreamToDecode.

[0454] A compliant decoder may comply with all of the requirements specified in this subparagraph.

[0455] (1) A decoder claiming conformance to a particular profile, layer, and level shall be able to successfully decode all bitstreams that satisfy the bitstream conformance requirements specified in subclause C.4 in the manner specified in Appendix A, provided that all instances of the VPS, SPS, and PPS referenced in the VCL NAL units and the appropriate SEI messages about the buffering period and picture synchronization are transmitted to the decoder in a timely manner, either in the bitstream (via non-VCL NAL units) or by external means not specified in this Description.

[0456] (2) When a bitstream contains syntax elements that have values that are defined as reserved, and it is specified that decoders will ignore the values of syntax elements or NAL units containing syntax elements that have reserved values, and the bitstream otherwise conforms to this Description, a compliant decoder shall decode the bitstream in the same manner as if decoding a conforming bitstream and will ignore the syntax elements or NAL units containing syntax elements that have reserved values, as defined.

[0457] There are two types of decoder matching: output timing matching and output order matching.

[0458] To verify decoder conformance, test bitstreams conforming to the declared profile, layer, and level, as defined in subclause C.4, are delivered by a hypothetical stream scheduler (HSS) to both the HRD and the decoder under test (DUT). All cropped decoded pictures output by the HRD shall also be output by the DUT, each cropped decoded picture output by the DUT shall be a picture with PicOutputFlag equal to 1, and for each such cropped decoded picture output by the DUT, the values of all samples that are output shall be equal to the values of the samples output by the specified decoding process.

[0459] With respect to decoder compliance with output synchronization conditions, the HSS operates as described above, with delivery schedules selected only from the subset of SchedSelIdx values for which the bit rate and CPB size are constrained as defined in Appendix A for a given profile, layer, and level, or with “interpolated” delivery schedules as defined below for which the bit rate and CPB size are constrained as defined in Appendix A. The same delivery schedule is used for both the HRD and the DUT.

[0460] When the HRD parameters and SEI buffering period messages are present with cpb_cnt_minus1[HighestTid] greater than 0, the decoder shall be capable of decoding the bitstream as delivered from the HSS operating using an "interpolated" delivery schedule specified having a peak bit rate r, a CPB size c(r), and an initial removal delay in the CPB

[0461] [Mathematical expression 15]

as follows:

[0462] for any SchedSelIdx>0 and r such that BitRate[SchedSelIdx-1]<=r<=BitRate[SchedSelIdx], such that r and c(r) are within the constraints as defined in Appendix A for the maximum bit rate and buffer size for a given profile, layer, and level. InitCpbRemovalDelay[SchedSelIdx] may differ from one buffering period to the next and is subject to recalculation.

[0463] To match the output timing decoder, the HRD is used as described above, and the timing (relative to the first bit delivery time) of the image output is the same for both the HRD and the DUT, within a fixed delay.

[0464] To match the decoder to the output order, the following applies:

[0465] (1) The HSS delivers the BitstreamToDecode bitstream to the DUT "on request" from the DUT, meaning that the HSS delivers bits (in decoding order) only when the DUT requests more bits to continue its processing. This means that for this test, the coded picture buffer in the DUT may be as large as the size of the largest decoding block.

[0466] (2) A modified HRD is used as described below, and the HSS delivers the bitstream to the HRD according to one of the schedules specified in the BitstreamToDecode bitstream such that the bit rate and CPB size are limited as defined in Appendix A. The order of image output will be the same for both the HRD and the DUT.

[0467] (3) The size of the HRD CPB is given by CpbSize[SchedSelIdx] as defined in subclause E.2.3, where SchedSelIdx and the HRD parameters are selected as defined in subclause C.1. The size of the DPB is given by sps_max_dec_pic_buffering_minus1[HighestTid]+1 from the active SPS (when nuh_layer_id for the current decoded picture is 0) or from the SPS of the active layer for the value of nuh_layer_id of the current decoded picture. In some cases, if the DPB operation point information parameters, op_dpb_info_parameters(), are present for the selected output layer set, the DPB size is given by vps_max_dec_pic_buffering_minus1[HighestTid] when currLayerId is 0 or set to vps_max_dec_pic_buffering_minus1[CurrLayerId][HighestTid] for the currLayerId for the operation point being tested when currLayerId is greater than 0, where currLayerId is the nuh_layer_id of the current decoded picture. Otherwise, if the DPB operation point information parameters, op_dpb_info_parameters(), are not present for the operation point being tested, the DPB size is given by sps_max_dec_pic_buffering_minus1[HighestTid]+1 from the active SPS (when the nuh_layer_id for the current decoded picture is 0) or from the active layer SPS for the nuh_layer_id value of the current decoded picture.

[0468] In some cases, if the DPB information parameters about output layer sets, oop_dpb_info_parameters(), are present for the selected output layer set, the DPB size is given by vps_max_dec_pic_buffering_minus1[HighestTid] when currLayerId is 0 or set to vps_max_dec_pic_buffering_minus1[CurrLayerId][HighestTid] for the currLayerId for the selected output layer set, where currLayerId is the nuh_layer_id of the current decoded picture. Otherwise, if the DPB output layer set information parameters oop_dpb_info_parameters() are not present for the selected output layer set, the DPB size is given by sps_max_dec_pic_buffering_minus1[HighestTid]+1 from the active SPS (when the nuh_layer_id for the current decoded picture is 0) or from the active layer SPS for the nuh_layer_id value of the current decoded picture.

[0469] The CPB removal time for the HRD is the final bit arrival time and decoding is performed immediately. The operation of the DPB for this HRD is as described in subclauses C.5.2 - C.5.2.3.

[0470] The decoded picture buffer contains picture storage buffers. The number of picture storage buffers for a nuh_layer_id of 0 is obtained from the active SPS. The number of picture storage buffers for each non-zero nuh_layer_id value is obtained from the active layer SPS for that non-zero nuh_layer_id value. Each of the picture storage buffers contains a decoded picture that is marked as "used for reference" or saved for future output. A process is initiated to output and remove pictures from the DPB as specified in subclause F.13.5.2.2, followed by starting a process to decode, mark, optionally flush, and save the picture as specified in subclause F.13.5.2.3. The "drop" process is defined in subclause F.13.5.2.4 and Initiated as defined in subclauses F.13.5.2.2 and F.13.5.2.3.

[0471] The output and removal of pictures from the DPB before decoding the current picture (but after parsing the slice header for the first slice of the current picture) occurs immediately when the first decoding block in the access block containing the current picture is removed from the CPB and continues as described below.

[0472] Invoke the decoding process for the RPS as defined in subclause 8.3.2.

[0473] (1) If the current image is an IRAP image with NoRaslOutputFlag equal to 1 and with nuh_layer_id equal to 0, which is not image 0, the following ordered steps are applied:

(A) Obtain the NoOutputOfPriorPicsFlag variable for the decoder under test as follows:

[0475] (i) If the current image is a CRA image, NoOutputOfPriorPicsFlag is set to 1 (regardless of the value of no_output_of_prior_pics_flag).

[0476] (ii) Otherwise, if the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_pic_buffering_minus1[HighestTid] obtained from the active SPS differs from the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_pic_buffering_minus1[HighestTid], respectively, obtained from the SPS active for the preceding picture, NoOutputOfPrior PicsFlag may (but need not) be set to 1 by the decoder under test regardless of the value of no_output_of_prior_pics_flag. Although setting NoOutputOfPriorPicsFlag equal to no_output_of_prior_pics_flag is preferred under these conditions, it is permissible to set NoOutputOfPriorPicsFlag to 1 for the decoder under test in this case.

[0477] (iii) Otherwise, NoOutputOfPriorPicsFlag is set equal to no_output_of_prior_pics_flag.

[0478] (B) The NoOutputOfPriorPicsFlag value obtained for the decoder under test is applied to HRD as follows:

[0479] (i) If NoOutputOfPriorPicsFlag is 1, all image storage buffers in the DPB are cleared without outputting the images they contain, and the DPB fullness is set to 0.

[0480] (ii) Otherwise (NoOutputOfPriorPicsFlag is 0), all picture storage buffers containing a picture that is marked as "not required for output" and "not used for reference" are cleared (without output), and all non-empty picture storage buffers in the DPB are cleared by repeatedly initiating the "flushing" process defined in subclause F.13.5.2.4, and the DPB fullness is set to 0.

[0481] (iii) Otherwise (the current picture is not an IRAP picture with NoRaslOutputFlag equal to 1 and with nuh_layer_id equal to 0), all picture storage buffers containing a picture that is marked as "not required for output" and "not used for reference" are cleared (no output). For each picture storage buffer that is cleared, the DPB occupancy is decreased by one. The variable currLayerId is set equal to the nuh_layer_id of the current decoded picture.

[0482] The variables MaxNumReorderPics[TargetOp][currLayerId][HighestTid], MaxLatencyIncreasePlus1[TargetOp][currLayerId][HighestTid], MaxLatencyPictures[TargetOp][currLayerId][HighestTid], MaxDecPicBufferingMinus1[TargetOp][currLayerId][HighestTid] are obtained as follows based on the current operating point being tested:

[0483] (1) If the DPB operating point information parameters op_dpb_info_parameters() are present for the operating point TargetOp under test, MaxNumReorderPics[TargetOp][currLayerId][HighestTid] is set to vps_max_num_reorder_pics[HighestTid] when currLayerId is 0, or is set to vps_max_num_reorder_pics[TargetOp][CurrLayerId][HighestTid] for currLayerId for the operating point under test when currLayerId is greater than 0. Otherwise, if the DPB operating point information parameters op_dpb_info_parameters() are not present for the operating point under test, MaxNumReorderPics[TargetOp][currLayerId][HighestTid] is set to sps_max_num_reorder_pics[HighestTid] of active SPS (when currLayerId is 0) or from the active layer SPS for the value of currLayerId.

[0484] (2) If the DPB op_dpb_info_parameters() operating point information parameters are present for the TargetOp operating point under test, MaxLatencyIncreasePlus1[TargetOp][currLayerId][HighestTid] is set to vps_max_latency_increase_plus1[HighestTid] when currLayerId is 0, or set to vps_max_latency_increase_plus1[TargetOp][CurrLayerId][HighestTid] for currLayerId for the operating point under test when currLayerId is greater than 0. If the DPB op_dpb_info_parameters() operating point information parameters are present for the operating point under test, MaxLatencyPictures [TargetOp][currLayerId][HighestTid] is set to VpsMaxLatencyPictures [HighestTid] if currLayerId is 0, or set to VpsMaxLatencyPictures of [TargetOp][CurrLayerId][HighestTid] for currLayerId for the operating point under test if currLayerId is greater than 0. Otherwise, if the operating point DPB info parameters op_dpb_info_parameters() are not present for the operating point under test, MaxLatencyIncreasePlus1[TargetOp][currLayerId][HighestTid] is set to sps_max_latency_increase_plus1[HighestTid] of the active SPS (when currLayerId is 0) or the active layer SPS for the value of currLayerId, and MaxLatencyPictures of [TargetOp][currLayerId][HighestTid] is set to SpsMaxLatencyPictures[HighestTid] obtained from from the active SPS (when currLayerId is 0) or from the active layer SPS for the value of currLayerId.

[0485] (3) If the DPB operating point information parameters op_dpb_info_parameters() are present for the selected TargetOp operating point under test, MaxDecPicBufferingMinus1[TargetOp][currLayerId][HighestTid] is set to vps_max_dec_pic_buffering_minus1[HighestTid] when currLayerId is 0, or is set to vps_max_dec_pic_buffering_minus1[TargetOp][CurrLayerId][HighestTid] for currLayerId for the operating point under test when currLayerId is greater than 0. Otherwise, if the DPB operating point information parameters op_dpb_info_parameters() are not present for the operating point under test, MaxDecPicBufferingMinus1[TargetOp][currLayerId][HighestTid] is set in sps_max_dec_pic_buffering_minus1[HighestTid] from the active SPS (when currLayerId is 0) or from the active layer SPS for the value of currLayerId.

[0486] When one or more of the following conditions are true, the "flushing" process defined in subclause F.13.5.2.4 is called repeatedly, further decreasing the DPB occupancy by one for each additional picture storage buffer that is flushed, until none of the following conditions is true:

[0487] (1) The number of images in DPB with nuh_layer_id equal to currLayerId marked as "required for output" is greater than MaxNumReorderPics[TargetOp][CurrLayerId][HighestTid].

[0488] (2) If MaxLatencyIncreasePlus1[TargetOp][CurrLayerId][HighestTid] is not 0 and there is at least one image in the DPB with nuh_layer_id equal to currLayerId that is marked as "required for output" for which the associated variable PicLatencyCount[currLayerId] is greater than or equal to MaxLatencyPictures[TargetOp][CurrLayerId][HighestTid].

[0489] (3) Number of images with nuh_layer_id equal to currLayerId in DPB is greater than or equal to MaxDecPicBuffering[TargetOp][CurrLayerId][HighestTid].

[0490] The processes defined in this subclause occur immediately when the last decoding block in access block n containing the current picture is removed from the CPB.

[0491] The currLayerId variable is set to the nuh_layer_id of the current decoded image.

[0492] For each image in the DPB that is marked as "required for output" and that has a nuh_layer_id value equal to currLayerId, the associated variable PicLatencyCount[currLayerId] is set to PicLatencyCount[currLayerId]+1.

[0493] The current picture is considered decoded after the last decoding block in the picture is decoded. The current decoded picture is stored in the DPB in an empty picture storage buffer, and the following applies:

[0494] (A) If the PicOutputFlag of the current decoded image is 1, it is marked as "required for output" and its associated PicLatencyCount[currLayerId] variable is set to 0.

[0495] (B) Otherwise (the PicOutputFlag of the current decoded picture is 0), it is marked as "not required for output".

[0496] The current decoded image is marked as "used for short-term reference".

[0497] When one or more of the following conditions are true, the "drop" process defined in subclause F.13.5.2.4 is called repeatedly until none of the following conditions are true.

[0498] (A) The number of images with nuh_layer_id equal to currLayerId in DPB marked as "required for output" is greater than MaxNumReorderPics[TargetOp][CurrLayerId][HighestTid].

[0499] (B) MaxLatencyIncreasePlus1[TargetOp][CurrLayerId][HighestTid] is not equal to 0 and there is at least one image in the DPB with nuh_layer_id equal to currLayerId that is marked as "required for output" for which the associated variable PicLatencyCount[currLayerId] is greater than or equal to MaxLatencyPictures[TargetOp][CurrLayerId][HighestTid].

[0500] Otherwise, the variables MaxNumReorderPics[currLayerId][HighestTid], MaxLatencyIncreasePlus1[currLayerId][HighestTid], MaxLatencyPictures [currLayerId][HighestTid], MaxDecPicBufferingMinus1[currLayerId][HighestTid] can be obtained as follows:

[0501] (1) If the DPB operating point information parameters op_dpb_info_parameters() are present for the operating point under test, MaxNumReorderPics[currLayerId][HighestTid] is set to vps_max_num_reorder_pics[HighestTid] when currLayerId is 0, or set to vps_max_num_reorder_pics[CurrLayerId][HighestTid] for currLayerId for the operating point under test when currLayerId is greater than 0. Otherwise, if the DPB operating point information parameters op_dpb_info_parameters() are not present for the operating point under test, MaxNumReorderPics[currLayerId][HighestTid] is set to sps_max_num_reorder_pics[HighestTid] from the active SPS (when currLayerId is 0) or from the active level SPS for the value currLayerId.

[0502] (2) If the DPB operating point information parameters op_dpb_info_parameters() are present for the operating point under test, MaxLatencyIncreasePlus1[currLayerId][HighestTid] is set to vps_max_latency_increase_plus1[HighestTid] when currLayerId is 0, or set to vps_max_latency_increase_plus1[CurrLayerId][HighestTid] for currLayerId for the operating point under test when currLayerId is greater than 0. If the DPB operating point information parameters op_dpb_info_parameters() are present for the operating point under test, MaxLaten-cyPictures[currLayerId][HighestTid] is set to VpsMaxLatencyPictures[HighestTid] when currLayerId is 0, or set to VpsMaxLatencyPictures[CurrLayerId][HighestTid] is set to currLayerId for the operating point under test when currLayerId is greater than 0. Otherwise, if the operating point DPB info parameters op_dpb_info_parameters() are not present for the operating point under test, MaxLatencyIncreasePlus1[currLayerId][HighestTid] is set to sps_max_latency_increase_plus1[HighestTid] of the active SPS (when currLayerId is 0) or the active layer SPS for the value of currLayerId, and MaxLatencyPictures [currLayerId][HighestTid] is set to SpsMaxLatencyPictures[HighestTid] obtained from the active SPS (when currLayerId is 0) or from the active layer SPS for the value of currLayerId.

[0503] (3) If the DPB operating point information parameters op_dpb_info_parameters() are present for the selected operating point under test, MaxDecPicBufferingMinus1[currLayerId][HighestTid] is set to vps_max_dec_pic_buffering_minus1[HighestTid] when currLayerId is 0, or set to vps_max_dec_pic_buffering_minus1[CurrLayerId][HighestTid] for currLayerId for the operating point under test when currLayerId is greater than 0. Otherwise, if the DPB operating point information parameters op_dpb_info_parameters() are not present for the operating point under test, MaxDecPicBufferingMinus1[currLayerId][HighestTid] is set to sps_max_dec_pic_buffering_minus1[HighestTid] from the active SPS (when currLayerId is 0) or from the active layer SPS for the value of currLayerId.

[0504] When one or more of the following conditions are true, the "flushing" process defined in subclause F.13.5.2.4 is called repeatedly, further decreasing the DPB occupancy by one for each additional picture storage buffer that is cleared, until none of the following conditions is true:

[0505] (1) The number of images in DPB with nuh_layer_id equal to currLayerId that are marked as "required for output" is greater than MaxNumReorderPics[CurrLayerId][HighestTid].

[0506] (2) If MaxLatencyIncreasePlus1[CurrLayerId][HighestTid] is not equal to 0 and there is at least one picture in the DPB with nuh_layer_id equal to currLayerId that is marked as "required for output" for which the associated variable PicLatencyCount[currLayerId] is greater than or equal to MaxLatencyPictures[CurrLayerId][HighestTid].

[0507] (3) The number of images with nuh_layer_id equal to currLayerId in the DPB is greater than or equal to MaxDecPicBuffering[CurrLayerId][HighestTid].

[0508] The processes defined in this subclause occur immediately when the last decoding block in access block n containing the current picture is removed from the CPB.

[0509] The currLayerId variable is set to the nuh_layer_id of the current decoded image.

[0510] For each image in the DPB that is marked as "required for output" and that has a nuh_layer_id value equal to currLayerId, the associated variable PicLatencyCount[currLayerId] is set to PicLatencyCount[currLayerId]+1.

[0511] The current picture is considered decoded after the last decoding block in the picture is decoded. The current decoded picture is stored in the DPB in an empty picture storage buffer, and the following applies:

[0512] (A) If the PicOutputFlag of the current decoded image is 1, it is marked as "required for output" and its associated PicLatencyCount[currLayerId] variable is set to 0.

[0513] (B) Otherwise (PicOutputFlag of the current decoded picture is 0), it is marked as "not required for output".

[0514] The current decoded image is marked as "used for short-term reference".

[0515] When one or more of the following conditions are true, the "drop" process defined in subclause F.13.5.2.4 is called repeatedly until none of the following conditions is true.

[0516] (A) The number of images in DPB with nuh_layer_id equal to currLayerId that are marked as "required for output" is greater than MaxNumReorderPics[CurrLayerId][HighestTid].

[0517] (B) MaxLatencyIncreasePlus1[CurrLayerId][HighestTid] is not equal to 0 and there is at least one picture in the DPB with nuh_layer_id equal to currLayerId that is marked as "required for output" for which its associated PicLatencyCount[currLayerId] variable is greater than or equal to MaxLatencyPictures[CurrLayerId][HighestTid].

[0518] The process of "dropping" consists of the following ordered stages:

[0519] (A) The images that are first to be output are selected as having the lowest PicOrderCntVal value of all images in the DPB marked as "required for output."

[0520] (B) These images are cropped using the cropping condition matching window specified in the active SPS for an image with a nuh_layer_id equal to 0 or in the active layer SPS for a nuh_layer_id value equal to that of the image, the cropped images are output in ascending order of nuh_layer_id, and the images are marked as "not required for output."

[0521] (C) Each image storage buffer that contains an image marked as "not used for reference" and that included one of the images that was cropped and output is cleared.

[0522] The VPS extension can have additional modifications if desired.

[0523] With reference to Fig. 40, a further modification may include DPB parameters sent in the VPS extension for output level sets, instead of the operating points where oops_dpb_info_parameters(j) are illustrated in Fig. 41.

[0524] The num_dpb_info_parameters element specifies the number of oop_dpb_parameters() syntax structures present in the VPS extension RBSP. Decoder parameters num_dpb_info_parameters must be in the range from 0 to num_output_layer_sets, inclusive.

[0525] The output_point_layer_set_idx[i] element specifies the index into the list of target output layer sets to which the i-th oop_dpb_info_parameters() syntax structure in the VPS extension is applied.

[0526] The value of output_point_layer_set_idx[i] must be in the range 0 to num_output_layer_sets, inclusive. A requirement of bitstream conformance is that output_point_layer_set_idx[i] must not be equal to output_point_layer_set_idx[j] for any j not equal to i.

[0527] With reference to Fig. 42, oop_dpb_info_paremters(c) may be further modified, where the syntax in the VPS extension may be as illustrated in Fig. 43.

[0528] With reference to Fig. 44, oop_dpb_info_paremters(c) may be further modified, where the syntax in the VPS extension may be as illustrated in Fig. 45 or Fig. 46.

[0529] An approximate alternative to the syntax in the VPS extension is that

[0530] [Table 18]

[0531] can be changed to

[0532] [Table 19]

[0533] The vps_max_layer_id element specifies the maximum allowed nuh_layer_id value of all NAL units in the CVS. The vps_max_layers_minus1 element specifies the maximum number of layers that may be present in the CVS, where a layer may, for example, be a spatial scalable layer, a quality scalable layer, a texture view, or a depth view.

[0534] Another example alternative for the syntax in the VPS extension is that

[0535] [Table 20]

[0536] can be changed to

[0537] [Table 21]

[0538] where numOutputLayers for the selected set of output layers is the index oplsIdx, obtained as:

[0539] [Table 22]

[0540] Another example alternative for the syntax in the VPS extension is that

[0541] [Table 23]

[0542] can be changed to

[0543] [Table 24]

where numOutputLayers for the selected oplsIdx is obtained as:

[0545] [Table 25]

[0546] Then the target list of decoded layer identifiers targetDLayerIdList and numDecodedLayers for the selected oplsIdx are obtained as:

[0547] [Table 26]

[0548] In one implementation, an additional flag may be signaled to indicate whether oop_dpb_information_parameters is signaled for a particular layer, as follows:

[0549] [Table 27]

[0550] A vps_layer_info_present_flag[j] of 1 specifies that oop_dpb_info_parameters are present for the j-th layer for a particular output layer set. A vps_layer_info_present_flag[j] of 0 specifies that oop_dpb_info_parameters are not present for the j-th layer for a particular output layer set.

[0551] In another implementation, the num_dpb_info_parameters decoders will be in the range 0 to 1024, inclusive. In yet another implementation, another fixed number may be used instead of 1024.

[0552] In the alternative implementation, output_point_layer_set_idx[i] is in the range from 0 to 1023, inclusive.

[0553] Referring to Fig. 47, another modified VPS extension and layer_dpb_info(i) may be used if DPB parameters are sent in the VPS extension for each layer regardless of the output layer sets and operating points.

[0554] With reference to Fig. 48, a modified layer_dpb_info(i) may be used, where the syntax element vps_max_sub_layer_minus1 signaled from the VPS is used for all layers and is not separately signaled in oop_dpb_info_parameters(id)/op_dpb_info_parameters(id).

[0555] Referring to FIGS. 49A and 49B, an exemplary modified vps_extension element is illustrated. The modified vps extension includes new syntax, namely, max_sub_layers_vps_predict_flag[i], max_sub_layers_vps_minus1[i], num_dpb_info_parameters, output_point_layer_set_idx[i], oop_dpb_maxbuffering_parameters(i), and layer_dpb_info_parameters(i). The num_output_layer_sets element specifies the number of layer sets for which output layers are specified by output_layer_set_index[i] and output_layer_flag[lsIdx][j]. If not present, the value of num_output_layer_sets is implied to be 0. A layer set describing the layers to be output is an output layer set.

[0556] A 'max_sub_layers_vps_predict_flag'[i] element of 1 specifies that max_sub_layers_vps_minus1[i] is implied to be equal to max_sub_layers_vps_minus1[i-1].

[0557] max_sub_layers_vps_predict_flag[i] equal to 0 specifies that max_sub_layers_vps_minus1[i] is signaled explicitly. The value of max_sub_layers_vps_predict_flag[0] is implied to be 0.

[0558] The 'max_sub_layers_vps_minus1[i]' element plus 1 specifies the maximum number of temporary sublayers that may be present in the CVS for a layer with nuh_layer_id equal to i. Values of max_vps_sub_layers_vps_minus1[i] will range from 0 to 6, inclusive. In some cases, max_sub_layers_vps_minus1[i] is used to derive the sps_max_sub_layers_minus1 SPS syntax element. When max_sub_layers_vps_predict_flag[i] is 1, max_sub_layers_vps_minus1[i] is implied to be max_sub_layers_vps_minus1[i-1]. The value of max_sub_layers_vps_minus1[0] is implied to be vps_max_sub_layers_minus1.

[0559] The MaxSubLayers[setId] variable for setId in the range 0 to num_dpb_info_parameters-1, inclusive, is obtained as follows:

[0560] [Table 28]

[0561] The 'num_dpb_info_parameters' element specifies the number of oop_dpb_maxbuffering_parameters(i) syntax structures present in the VPS extension RBSP. Decoder parameters num_dpb_info_parameters must be in the range 0 to numOutputLayerSets, inclusive.

[0562] The 'output_point_layer_set_idx'[i] element specifies the index into the list of output layer sets to which the i-th oop_dpb_maxbuffering_parameters(i) syntax structure in the VPS extension applies.

[0563] The value of output_point_layer_set_idx[i] must be in the range 0 to numOut-putLayerSets, inclusive. A requirement of bitstream conformance is that output_point_layer_set_idx[i] will not be equal to output_point_layer_set_idx[j] for any j not equal to i.

[0565] Referring to Fig. 50, the oop_dpb_maxbuffering_parameters element specifies 'sub_layer_vps_buf_info_present_flag'[i], 'max_vps_dec_pic_buffering_minus1[i][j].

[0566] A sub_layer_vps_buf_info_present_flag[i] of 1 specifies that max_vps_dec_pic_buffering_minus1[i][j] values are present for MaxSubLayers[i] sublayers. A sub_layer_vps_buf_info_present_flag[i] of 0 specifies that max_vps_dec_pic_buffering_minus1[i][MaxSubLayers[i]-1] values apply to all sublayers.

[0567] The 'max_vps_dec_pic_buffering_minus1'[i][j] member plus 1 specifies the maximum required decoded picture buffer size for CVS for the layer with nuh_layer_id equal to highestLayerId in the output layer set associated with index i, in units of picture storage buffers when HighestTid is j. The value of max_vps_dec_pic_buffering_minus1[i][j] will be in the range from 0 to MaxDpbSize - 1 (as defined in subclause A.4), inclusive. When j is greater than 0, max_vps_dec_pic_buffering_minus1[i][j] will be greater than or equal to max_vps_dec_pic_buffering_minus1[i][j-1]. In some cases max_vps_dec_pic_buffering_minus1[i][j] is used to infer the values of the SPS syntax elements sps_max_dec_pic_buffering_minus1[j]. When max_vps_dec_pic_buffering_minus1[i][j] is not present for i in the range 0 to MaxSubLayers[i] - 2, inclusive, due to sub_layer_vps_buf_info_present_flag[i] being equal to 0, it is implied to be equal to max_vps_dec_pic_buffering_minus1[i][MaxSubLayers[i] - 1].

[0568] The value of max_vps_dec_pic_buffering_minus1[0][j] for each value of j is assumed to be vps_max_dec_pic_buffering_minus1[j].

[0569] Referring to Fig. 51, the layer_dpb_info_parameters element specifies 'sub_layer_vps_ordering_info_present_flag'[i], max_vps_num_reorder_pics[i][j], max_vps_latency_increase_plus1'[i][j].

[0570] Element 'sub_layer_vps_ordering_info_present_flag'[i] equal to 1 specifies that max_vps_num_reorder_pics[i][j] and max_vps_latency_increase_plus1[i][j] are present for max_sub_layers_vps_minus1+1 sublayers.

[0571] A sub_layer_vps_ordering_info_present_flag[i] element of 0 specifies that the values of max_vps_num_reorder_pics[i][vps_max_sub_layers_minus1] and max_vps_latency_increase_plus1[i][max_sub_layers_vps_minus1] apply to all sublayers.

[0572] The 'max_vps_num_reorder_pics'[i][j] element specifies the maximum number of images that may precede any image in the CVS for a layer with nuh_layer_id equal to i in decoding order and follow that image in output order when HighestTid equals j. The value of max_vps_num_reorder_pics[i][j] will be in the range from 0 to max_vps_dec_pic_buffering_minus1[i][j], inclusive. When j is greater than 0, max_vps_num_reorder_pics[i][j] will be greater than or equal to max_vps_num_reorder_pics[i][j-1]. In some cases, max_vps_num_reorder_pics[i][j] is used to derive values of the SPS syntax element sps_max_num_reorder_pics[j]. When max_vps_num_reorder_pics[i][j] is not present for i in the range 0 to max_sub_layers_vps_minus1[i]-1, inclusive, due to sub_layer_vps_ordering_info_present_flag[i] being equal to 0, it is implied to be equal to max_vps_num_reorder_pics[i][max_sub_layers_vps_minus1[i]].

[0573] The max_vps_latency_increase_plus1[i][j] member, when not equal to 0, is used to calculate the value of VpsMaxLatencyPictures[i][j], which specifies the maximum number of pictures that can precede any picture in the CVS for a layer with nuh_layer_id equal to i in output order and follow that picture in decoding order when HighestTid equals j.

[0574] When max_vps_latency_increase_plus1[i][j] is not 0, the MaxLatencyPictures Vps-[i][j] value is set as follows:

[0575] VpsMaxLatencyPictures[i][j]=max_vps_num_reorder_pics[i][j]+max_vps_latency_increase_plus1[i][j] - 1

[0576] When max_vps_latency_increase_plus1[i][j] is 0, the corresponding limit value is not expressed.

[0577] The value of max_vps_latency_increase_plus1[j][k] must be in the range 0 to 2 ³² - 2, inclusive. In some cases, max_vps_latency_increase_plus1[i][j] is used to derive the values of the SPS syntax elements sps_max_latency_increase_plus1[j]. When max_vps_latency_increase_plus1[i][j] is not present for i in the range 0 to max_sub_layers_vps_minus1[i]-1, inclusive, due to sub_layer_vps_ordering_info_present_flag[i] being 0, it is implied to be equal to max_vps_latency_increase_plus1[i][max_sub_layers_vps_minus1[i]].

[0578] Referring to Fig. 52, an exemplary modified vps_extension is illustrated. The modified vps extension includes a further modification of the syntax of Fig. 49 with new syntax, namely, sub_layer_vps_buf_info_predict_flag[i], sub_layer_vps_ordering_info_predict_flag[i], which signal conditionally. The element sub_layer_vps_buf_info_predict_flag[i] equal to 1 specifies that max_vps_dec_pic_buffering_minus1[i][j] is implied to be equal to max_vps_dec_pic_buffering_minus1[i-1][j] for each value of j.

[0579] A sub_layer_vps_buf_info_predict_flag[i] element of 0 specifies that max_vps_dec_pic_buffering_minus1[i][j] for at least one value of j is signaled explicitly. The value of sub_layer_vps_buf_info_predict_flag[0] is implied to be 0. If not present, sub_layer_vps_buf_info_predict_flag[i] is implied to be 0.

[0580] A sub_layer_vps_ordering_info_predict_flag[i] element equal to 1 specifies that the syntax elements sub_layer_vps_ordering_info_present_flag[i], max_vps_num_reorder_pics[i][j], and max_vps_latency_increase_plus1[i][j] are implied to be sub_layer_vps_ordering_info_present_flag[i-1], max_vps_num_reorder_pics[i-1][j], and max_vps_latency_increase_plus1[i 1][j], respectively. The sub_layer_vps_ordering_info_predict_flag[i] flag, equal to 0, specifies that the sub_layer_vps_ordering_info_present_flag[i], max_vps_num_reorder_pics[i][j], and max_vps_latency_increase_plus1[i][j] syntax elements are signaled explicitly. If not present, the sub_layer_vps_ordering_info_predict_flag[i] value is set to 0.

[0581] The other syntactic elements and their semantic meanings for Fig. 52 are the same as those of Fig. 49.

[0582] Referring to Fig. 53, an exemplary modified vps_extension element is illustrated. The modified vps extension includes a further modification of the syntax of Fig. 49. In Fig. 53, oop_dpb_maxbuffering_parameters(i,j) is signaled for each level j for a particular set i of output levels, compared to Fig. 49, in which a single set of parameters oop_dpb_maxbuffering_parameters(i) is signaled for the identifier (id) of the highest level in the set i of output levels.

[0583] Referring to Fig. 54, the oop_dpb_maxbuffering_parameters element specifies sub_layer_vps_buf_info_present_flag[i][k], max_vps_dec_pic_buffering_minus1[i][k][j].

[0584] A sub_layer_vps_buf_info_present_flag[i][k] element of 1 specifies that max_vps_dec_pic_buffering_minus1[k][j] values are present for max_sub_layers_vps_minus1[k] sublayers. A sub_layer_vps_buf_info_present_flag[i] element of 0 specifies that max_vps_dec_pic_buffering_minus1[i][k][max_sub_layers_vps_minus1[k]] values apply to all sublayers.

[0585] The max_vps_dec_pic_buffering_minus1[i][k][j] element plus 1 specifies the maximum required decoded picture buffer size for CVS for the layer with nuh_layer_id equal to k, for the output layer set associated with index i, in units of picture storage buffers when HighestTid is j. The values of max_vps_dec_pic_buffering_minus1[i][k][j] will be in the range from 0 to MaxDpbSize - 1 (as defined in subclause 4), inclusive.

[0586] When j is greater than 0, max_vps_dec_pic_buffering_minus1[i][k][j] will be greater than or equal to max_vps_dec_pic_buffering_minus l[i][k][j-l] is used to derive the values of SPS syntax elements sps_max_dec_pic_buffering_minus1[j].

[0587] When max_vps_dec_pic_buffering_minus1[i][j] is not present for i in the range 0 to max_sub_layers_vps_minus1[k]-1, inclusive, due to sub_layer_vps_buf_info_present_flag[i][k] being equal to 0, it is inferred to be max_vps_dec_pic_buffering_minus1[i][k] max_sub_layers_vps_minus1[k]].

[0588] The value of max_vps_dec_pic_buffering_minus1[i][0][j] for each value of i and j is assumed to be vps_max_dec_pic_buffering_minus1[j].

[0589] With reference to Fig. 55, the element oop_dpb_maxbuffering_parameters specifies sub_layer_vps_buf_info_present_flag[i][k], max_vps_dec_pic_buffering_minus1[i][k][j]. The figure of Fig. 55 is a variant of the syntax for oop_dpb_maxbuffering_parameters compared to the syntax in Fig. 54 for oop_dpb_maxbuffering_parameters.

[0590] The MaxSubLayers[setId][k] variable for setId in the range from 0 to num_dpb_info_parameters - 1, inclusive, is obtained as follows:

[0591] [Table 29]

[0592] In this case, the parameters oop_dpb_maxbuffering_parameters(i,k) will be defined as in Fig.55.

[0593] Referring to Fig. 56, an exemplary modified vps_extension element is illustrated. The modified vps extension includes a further modification of the syntax of Fig. 52 with a new syntax. In Fig. 53, in this embodiment, oop_dpb_maxbuffering_parameters(i, j) are signaled for each level j for a particular set i of output levels, compared to Fig. 52, in which a single set of parameters oop_dpb_maxbuffering_parameters(i) are signaled for the identifier of the highest level in the set i of output levels.

[0594] The values of oop_dpb_maxbuffring_parameters(i,k) are as shown in Fig.57.

[0595] With reference to Fig. 58, the element oop_dpb_maxbuffering_parameters specifies sub_layer_vps_buf_info_present_flag[i][k], max_vps_dec_pic_buffering_minus1[i][k][j]. The figure of Fig. 58 is a variant of the syntax for oop_dpb_maxbuffering_parameters compared to the syntax in Fig. 57 for oop_dpb_maxbuffering_parameters.

[0596] The MaxSubLayers[setId][k] variable for setId in the range from 0 to num_dpb_info_parameters - 1, inclusive, is obtained as follows:

[0597] [Table 30]

[0598] In this case, the parameters oop_dpb_maxbuffering_parameters(i,k) will be defined as in Fig.58.

[0599] Referring to Fig. 59, an exemplary modified vps_extension element is illustrated. The modified vps extension includes an additional modification of the syntax of Fig. 56 with a new syntax, namely, an additional layer_dpb_info_parameters_presence_flag flag, which is signaled conditionally. The layer_dpb_info_parameters_presence_flag makes the signaling of layer_dpb_info_parameters(i) in the VPS extension optional.

[0600] A layer_dpb_info_parameters_presence_flag[i] element of 1 specifies that the sub_layer_vps_ordering_info_predict_flag[i] and layer_dpb_info_parameters(i) syntax elements are present for vps_max_num_layers_minus1 levels. A layer_dpb_info_parameters_presence_flag[i] element of 0 specifies that the sub_layer_vps_ordering_info_predict_flag[i] and layer_dpb_info_parameters(i) syntax elements are not present for vps_max_num_layers_minus1 levels.

[0601] The values of oop_dpb_maxbuffring_parameters(i,k) are as shown in Fig.60.

[0602] With reference to Fig. 61, the element oop_dpb_maxbuffering_parameters specifies sub_layer_vps_buf_info_present_flag[i][k], max_vps_dec_pic_buffering_minus1[i][k][j]. The figure of Fig. 61 is a variant of the syntax for oop_dpb_maxbuffering_parameters compared to the syntax in Fig. 60 for oop_dpb_maxbuffering_parameters.

[0603] The MaxSubLayers[setId][k] variable for setId in the range from 0 to num_dpb_info_parameters - 1, inclusive, is obtained as follows:

[0604] [Table 31]

[0605] In this case, the parameters oop_dpb_maxbuffering_parameters(i,k) will be defined as in Fig.61.

[0606] An approximate alternative to the syntax in the VPS extension is that

[0607] [Table 32]

[0608] can be changed to

[0609] [Table 33]

[0610] Thus, the index k can start at 0 rather than 1.

[0611] An approximate alternative to the syntax in the VPS extension is that

[0612] [Table 34]

can be changed to

[0614] [Table 35]

[0615] The vps_max_layer_id element specifies the maximum allowed nuh_layer_id value of all NAL units in the CVS. The vps_max_layers_minus1 element specifies the maximum number of layers that may be present in the CVS, where a layer may, for example, be a spatial scalable layer, a quality scalable layer, a texture view, or a depth view.

[0616] Another example alternative for the syntax in the VPS extension is that

[0617] [Table 36]

[0618] can be changed to

[0619] [Table 37]

[0620] where numOutputLayers is obtained as

[0621] [Table 38]

[0622] or this can be changed to

[0623] [Table 39]

[0624] In one implementation, an additional flag may be signaled to indicate if oop_dpb_information_parameters is signaled for a particular layer, as follows:

[0625] [Table 40]

[0626] A vps_layer_info_present_flag[k] of 1 specifies that oop_dpb_maxbuffering_parameters are present for the k-th layer for a particular output layer set. A vps_layer_info_present_flag[k] of 0 specifies that oop_dpb_maxbuffering_parameters are not present for the k-th layer for a particular output layer set.

[0627] The syntax elements of non-VCL NAL units (or their default values for some of the syntax elements) required for HRD are defined in the semantic subclauses of clause 7, Appendices D and E.

[0628] Two types of HRD parameter sets are used (HRD parameters for NAL and HRD parameters for VCL). HRD parameter sets are signaled via the hrd_parameters() syntax structure, which can be part of the SPS syntax structure or the VPS syntax structure.

[0629] Multiple tests may be required to verify the conformance of a bitstream, which is referred to as the test bitstream. For each test, the following steps are applied in the order given:

[0630] (1) A set of output layers, designated as TargetOpLs, is selected to be tested. The operation point referenced by output_layer_set_idx[] in TargetOpLs identifies the operation point to be tested. The list of output layer identifiers, OpLayerIdList, in TargetOpLs consists of a list of nuh_layer_id values, in ascending order, of the nuh_layer_id values present in the subset of the bitstream associated with TargetOp and TargetOpLs, which is a subset of the nuh_layer_id values present in the bitstream to be tested. OpTid in TargetOp is equal to the highest TemporalId present in the subset of the bitstream associated with TargetOp.

[0631] (2) TargetDecLayerIdList is set equal to the target list targetDLayerIdList of decoded layer identifiers for the selected set of TargetOpLs of output layers, HighestTid is set equal to the OpTid of the TargetOp, and the process of extracting the sub-bitstream as specified in paragraph 10 is initiated with the bitstream under test, HighestTid, and TargetDecLayerIdList as input, and the output is assigned to BitstreamToDecode.

[0632] (3) Select an hrd_parameters() syntax structure and a sub_layer_hrd_parameters() syntax structure applicable to the TargetOp. If the TargetDecLayerIdList contains all nuh_layer_id values present in the bitstream being tested, select an hrd_parameters() syntax structure in the active SPS (or provided by an external means not described in this Description). Otherwise, select an hrd_parameters() syntax structure in the active VPS (or provided by some external means not described in this Description) that applies to the TargetOp. Within the selected hrd_parameters() syntax structure, if BitstreamToDecode is a Type I bitstream, the sub_layer_hrd_parameters(HighestTid) syntax structure that immediately follows the "if(vcl_hrd_parameters_present_flag)" condition is selected, and the NalHrdModeFlag variable is set to 0; otherwise (BitstreamToDecode is a Type II bitstream), the sub_layer_hrd_parameters(HighestTid) syntax structure that immediately follows either the "if(vcl_hrd_parameters_present_flag)" condition (in which case the NalHrdModeFlag variable is set to 0) or the "if(nal_hrd_parameters_present_flag)" condition is selected (in which case the NalHrdModeFlag variable is set to 1). When BitstreamToDecode is a Type II bitstream and NalHrdModeFlag is 0, all non-VCL NAL units other than NAL units with filler data, and all leading_zero_8bits, zero_byte, start_code_prefix_one_3bytes, and trailing_zero_8bits syntax elements that form a byte stream from a NAL unit stream (as defined in Appendix B), if present, are discarded from BitstreamToDecode and the remaining bitstream is assigned to BitstreamToDecode.

[0633] A compliant decoder may comply with all of the requirements specified in this subclause.

[0634] (1) A decoder claiming conformance to a particular profile, layer, and level shall be capable of successfully decoding all bitstreams that satisfy the bitstream conformance requirements specified in subclause C.4 in the manner specified in Appendix A, provided that all instances of the VPS, SPS, and PPS referenced in the VCL NAL units and the corresponding SEI messages for buffering period and picture synchronization are transmitted to the decoder in a timely manner, either in the bitstream (non-VCL NAL units) or by an external means not specified in this Description.

[0635] (2) When a bitstream contains syntax elements that have values that are defined as reserved, and it is specified that decoders will ignore the values of syntax elements or NAL units containing syntax elements that have reserved values, and the bitstream otherwise conforms to this Description, a compliant decoder will decode the bitstream in the same manner as it would decode a compliant bitstream and will ignore the syntax elements or NAL units containing syntax elements that have reserved values, as defined.

[0636] There are two types of decoder matching: output timing matching and output order matching.

[0637] To verify decoder conformance, test bitstreams conforming to the declared profile, layer, and level, as defined in subclause C.4, are delivered by a hypothetical stream scheduler (HSS) to both the HRD and the decoder under test (DUT). All cropped decoded pictures output by the HRD shall also be output by the DUT, each cropped decoded picture output by the DUT shall be a picture with PicOutputFlag equal to 1, and for each such cropped decoded picture output by the DUT, the values of all samples being output shall be equal to the values of the samples output by the specified decoding process.

[0638] For the HSS output synchronization decoder compliance, it operates as described above with delivery schedules selected only from the subset of SchedSelIdx values for which the bit rate and CPB size are constrained as defined in Appendix A for a given profile, layer, and level, or with "interpolated" delivery schedules as defined below for which the bit rate and CPB size are constrained as defined in Appendix A. The same delivery schedule is used for both the HRD and the DUT.

[0639] When the HRD parameters and SEI buffering period messages are present with cpb_cnt_minus1[HighestTid] greater than 0, the decoder will be able to decode the bitstream as delivered from the HSS using an "interpolated" delivery schedule specified having a peak bit rate r, a CPB size c(r), and an initial CPB removal delay

[0640] [Mathematical expression 16]

as follows:

[0641] for any SchedSelIdx>0 and r such that BitRate[SchedSelIdx-1] <=r<= BitRate[SchedSelIdx], such that r and c(r) are within the bounds as defined in Appendix A for the maximum bit rate and buffer size for a given profile, layer, and level. InitCpbRemovalDelay[SchedSelIdx] may differ from one buffering period to the next and is subject to recalculation.

[0642] To match the output timing decoder, the HRD is used as described above, and the timing (relative to the first bit delivery time) of the image output is the same for both the HRD and the DUT, within a fixed delay.

[0643] To match the decoder to the output order, the following is used:

[0644] (1) The HSS delivers the BitstreamToDecode bitstream to the DUT "on request" from the DUT, meaning that the HSS delivers bits (in decoding order) only if the DUT requests more bits to continue its processing. This means that for this test, the DUT's coded picture buffer may be within the size of the largest decoding block.

[0645] (2) A modified HRD is used as described below, and the HSS delivers the bitstream to the HRD according to one of the schedules specified in the BitstreamToDecode bitstream such that the bit rate and CPB size are limited as defined in Appendix A. The order of image output will be the same for both the HRD and the DUT.

[0646] (3) The size of the HRD CPB is given by CpbSize[SchedSelIdx] as defined in subclause E.2.3, where SchedSelIdx and the HRD parameters are selected as defined in subclause C.l. The size of the DPB is given by sps_max_dec_pic_buffering_minus1[HighestTid]+1 from the active SPS (when nuh_layer_id for the current decoded picture is 0), or from the SPS of the active layer for the value of nuh_layer_id of the current decoded picture.

[0647] In some cases, if the output layer set DPB information parameters, oop_dpb_maxbuffering_parameters(), are present for the selected output layer set, the DPB size is given by max_vps_dec_pic_buffering_minus1[CurrLayerId][HighestTid], where currLayerId is the nuh_layer_id of the current decoded picture. Otherwise, if the output layer set DPB information parameters, oop_dpb_maxbuffering_parameters(), are not present for the selected output layer set, the DPB size is given by sps_max_dec_pic_buffering_minus1[HighestTid]+1 from the active SPS (when the nuh_layer_id for the current decoded picture is 0), or from the active layer SPS for the nuh_layer_id value of the current decoded picture).

[0648] The CPB removal time for the HRD is the final bit arrival time and decoding occurs immediately. The operation of the DPB for this HRD is as described in subclauses C.5.2 - C.5.2.3.

[0649] The decoded picture buffer contains picture storage buffers. The number of picture storage buffers for a nuh_layer_id of 0 is obtained from the active SPS. The number of picture storage buffers for each non-zero nuh_layer_id value is obtained from the active layer SPS for that non-zero nuh_layer_id value. Each of the picture storage buffers contains a decoded picture that is marked as "used for reference" or saved for future output. A process is initiated to output and remove pictures from the DPB as specified in subclause F.13.5.2.2, followed by a call to the process to decode the picture, mark, optionally flush, and save as specified in subclause F.13.5.2.3. The "flush" process is defined in subclause F.13.5.2.4 and is called as specified in subclauses F.13.5.2.2 and F.13.5.2.3.

[0650] The output and removal of pictures from the DPB before decoding the current picture (but after parsing the slice header for the first slice of the current picture) occurs immediately when the first decoding block in the access block containing the current picture is removed from the CPB, and continues as described below.

[0651] Invoke the decoding process for the RPS as defined in subclause 8.3.2.

[0652] (1) If the current image is an IRAP image with NoRaslOutputFlag equal to 1 and with nuh_layer_id equal to 0 that is not image 0, the following ordered steps are applied:

[0653] (A) Obtain the NoOutputOfPriorPicsFlag variable for the decoder under test as follows:

[0654] (i) If the current image is a CRA image, NoOutputOfPriorPicsFlag is set to 1 (regardless of the value of no_output_of_prior_pics_flag).

[0655] (ii) Otherwise, if the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_pic_buffering_minus1[HighestTid], obtained from the active SPS, differs from the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_pic_buffering_minus1[HighestTid], respectively, obtained from the SPS active for the preceding picture, NoOutputOfPriorPicsFlag may (but need not) be set to 1 by the decoder under test regardless of the value of no_output_of_prior_pics_flag. Although setting NoOutputOfPriorPicsFlag equal to no_output_of_prior_pics_flag is preferred under these conditions, the decoder under test is allowed to set NoOutputOfPriorPicsFlag to 1 in this case.

[0656] (iii) Otherwise, NoOutputOfPriorPicsFlag is set equal to no_output_of_prior_pics_flag.

[0657] (B) The NoOutputOfPriorPicsFlag value obtained for the decoder under test is applied to HRD as follows:

[0658] (i) If NoOutputOfPriorPicsFlag is 1, all image storage buffers in the DPB are cleared without outputting the images they contain, and the DPB fullness is set to 0.

[0659] (ii) Otherwise (NoOutputOfPriorPicsFlag is 0), all picture storage buffers containing a picture that is marked as "not required for output" and "not used for reference" are cleared (without output), and all non-empty picture storage buffers in the DPB are cleared by repeatedly initiating the "flushing" process defined in subclause F.13.5.2.4, and the DPB fullness is set to 0.

[0660] (iii) Otherwise (the current picture is not an IRAP picture with NoRaslOutputFlag equal to 1 and with nuh_layer_id equal to 0), all picture storage buffers containing pictures that are marked as "not required for output" and "not used for reference" are flushed (no output). For each picture storage buffer that is flushed, the DPB occupancy is decreased by one. The variable currLayerId is set equal to the nuh_layer_id of the current decoded picture.

[0661] The variables MaxNumReorderPics[currLayerId][HighestTid], MaxLatencyln-creasePlus1[currLayerId][HighestTid], MaxLatencyPictures [currLayerId][HighestTid], MaxDecPicBufferingMinus1[currLayerId][HighestTid] are obtained as follows:

[0662] If the DPB layer info parameters, layer_dpb_info_parameters(), are present in the VPS, MaxNumReorderPics[currLayerId][HighestTid] is set to vps_max_num_reorder_pics[HighestTid] when currLayerId is 0, or set to max_vps_num_reorder_pics[CurrLayerId][HighestTid] for currLayerId when currLayerId is greater than 0. Otherwise, if the DPB layer info parameters, layer_dpb_info_parameters(), are not present, MaxNumReorderPics[currLayerId][HighestTid] is set to sps_max_num_reorder_pics[HighestTid] obtained from the active SPS (when currLayerId is 0) or from the active layer SPS for the value of currLayerId.

[0663] If the DPB layer info parameters, layer_dpb_info_parameters(), are present in the VPS, MaxLatencyIncreasePlus1[currLayerId][HighestTid] is set to vps_max_latency_increase_plus1[HighestTid] when currLayerId is 0, or set to max_vps_latency_increase_plus1[CurrLayerId][HighestTid] for currLayerId when currLayerId is greater than 0. If the DPB layer info parameters, layer_dpb_info_parameters(), are present in the VPS, MaxLatencyPictures[currLayerId][HighestTid] is set to SpsMaxLatencyPictures[HighestTid] when currLayerId is 0, or set to VpsMaxLatencyPictures[CurrLayerId][HighestTid] for currLayerId when currLayerId is greater than 0. Otherwise, if the DPB layer info parameters, layer_dpb_info_parameters(), are not present for the operating point being tested, MaxLatencyIncreasePlus1[currLayerId][HighestTid] is set to sps_max_latency_increase_plus1[HighestTid] of the active SPS (when currLayerId is 0) or the active layer SPS for the value of currLayerId, and MaxLatencyPictures [currLayerId][HighestTid] is set to SpsMaxLatencyPictures[currLayerId][HighestTid] obtained from the active SPS (when currLayerId is 0) or from the active layer SPS for the value of currLayerId.

[0664] If the DPB operating point information parameters, oop_dpb_maxbuffering_parameters(), are present for the selected output level set, MaxDecPicBufferingMinus1[currLayerId][HighestTid] is set to vps_max_dec_pic_buffering_minus1[HighestTid] when currLayerId is 0, or set to max_vps_dec_pic_buffering_minus1[CurrLayerId][HighestTid] for currLayerId for the operating point being tested when currLayerId is greater than 0. Otherwise, if the DPB operating point information parameters, oop_dpb_maxbuffering_parameters(), are not present for the operating point being tested, MaxDecPicBufferingMinus1[currLayerId][HighestTid] is set to sps_max_dec_pic_buffering_minus1[HighestTid] from the active SPS (when currLayerId is 0), or from the active layer SPS for the value of currLayerId.

[0665] When one or more of the following conditions are true, the "flushing" process defined in subclause F.13.5.2.4 is called repeatedly, further decreasing the DPB occupancy by one for each additional image storage buffer that is flushed, until none of the following conditions is true:

[0666] (1) The number of images with nuh_layer_id equal to currLayerId in DPB that are marked as "required for output" is greater than MaxNumReorderPics[CurrLayerId][HighestTid].

[0667] (2) If MaxLatencyIncreasePlus1[CurrLayerId][HighestTid] is not 0 and there is at least one image in the DPB with nuh_layer_id equal to currLayerId that is marked as "required for output" for which the associated variable PicLatencyCount[currLayerId] is greater than or equal to MaxLatencyPictures[CurrLayerId][HighestTid].

[0668] (3) The number of images in DPB with nuh_layer_id equal to currLayerId is greater than or equal to MaxDecPicBuffering[CurrLayerId][HighestTid].

[0669] The processes defined in this subclause occur immediately when the last decoding block of access block n containing the current picture is removed from the CPB.

[0670] The currLayerId variable is set to the nuh_layer_id of the current decoded image.

[0671] For each image in the DPB that is marked as "required for output" and that has a nuh_layer_id value equal to currLayerId, its associated PicLatencyCount[currLayerId] variable is set to PicLatencyCount[currLayerId]+1.

[0672] The current picture is considered decoded after the last decoding block in the picture is decoded. The current decoded picture is stored in the DPB in an empty picture storage buffer and the following applies:

[0673] (A) If the PicOutputFlag of the current decoded image is 1, it is marked as "required for output" and its associated PicLatencyCount[currLayerId] variable is set to 0.

[0674] (B) Otherwise (the PicOutputFlag of the current decoded picture is 0), it is marked as "not required for output".

[0675] The current decoded image is marked as "used for short-term reference".

[0676] When one or more of the following conditions are true, the "drop" process defined in subclause F.13.5.2.4 is called repeatedly until none of the following conditions are true.

[0677] (A) The number of images with nuh_layer_id equal to currLayerId in the DPB that are marked as "required for output" is greater than MaxNumReorderPics[CurrLayerId][HighestTid].

[0678] (B) MaxLatencyIncreasePlus1[CurrLayerId][HighestTid] is not equal to 0 and there is at least one picture in the DPB with nuh_layer_id equal to currLayerId that is marked as "required for output" for which the associated variable PicLatencyCount[currLayerId] is greater than or equal to MaxLatencyPictures[CurrLayerId][HighestTid].

[0679] The "dropping" process consists of the following ordered steps:

[0680] (A) The images that are first to be output are selected as those with the lowest PicOrderCntVal value from among all images in the DPB marked as "required for output."

[0681] (B) These images are cropped using the cropping condition matching window specified in the active SPS for an image with a nuh_layer_id equal to 0 or in the active layer SPS for a nuh_layer_id value equal to that of the image, the cropped images are output in ascending order of nuh_layer_id, and the images are marked as "not required for output."

[0682] (C) Each image storage buffer that contains an image marked as "not used for reference" and that included one of the images that was cropped and output is cleared.

[0683] Referring to Fig. 62, an exemplary modified sequence parameter set (sps) syntax, seq_parameter_set_rbsp, is illustrated. The modified sps includes sps_dpb_params_present_flag. In one design, based on the value of this flag and based on the value of nuh_layer_id, some of the syntax elements (e.g., sps_sub_layer_ordering_info_present_flag, sps_max_dec_pic_buffering_minus1[i], sps_max_num_reorder_pics[i], sps_max_latency_incease_plus1[i]) may not be signaled.

[0684] An sps_dpb_params_present_flag member equal to 0 specifies that for an SPS with nuh_layer_id>0, the sps_sub_layer_ordering_info_present_flag, sps_max_dec_pic_buffering_minus1[i], sps_max_num_reorder_pics[i], sps_max_latency_increase_plus1[i] syntax elements are not present in this SPS. The value of these parameters is set equal to the value of the sub_layer_vps_buf_info_present_flag[i], max_vps_num_reorder_pics[i][j], and max_vps_latency_increase_plus1[i][j] parameters signaled in the active VPS extension. An element sps_dpb_params_present_flag equal to 1 specifies that for an SPS with nuh_layer_id>0, the syntax elements sps_sub_layer_ordering_info_present_flag, sps_max_dec_pic_buffering_minus1[i], sps_max_num_reorder_pics[i], sps_max_latency_increase_plus1[i] are present in this SPS.

[0685] In another implementation, one or more of the syntactic elements may be signaled using a known fixed number of bits instead of u(v), instead of ue(v). For example, they may be signaled using u(8), or u(16), or u(32), or u(64), etc.

[0686] In another implementation, one or more of these syntactic elements may be signaled using ue(v) or some other encoding scheme instead of a fixed number of bits, such as u(v) encoding.

[0687] In another implementation, the names of various syntactic elements and their semantics may be changed by adding plus1 or plus2 or subtracting minus1 or minus2 compared to the described syntax and semantics.

[0688] In another implementation, various syntax elements may be signaled per picture somewhere in the bitstream. For example, they may be signaled in a slice segment header, pps/sps/vps/or any other parameter set or other normative part of the bitstream.

[0689] In still other embodiments, all concepts defined in this invention relating to output level sets may be applied to output operating points [2,3] and/or to operating points [1].

Example 3

[0690] A method for encoding video is disclosed. The method includes initiating a process of parsing a first slice header of a current image. Determining which steps performed by a decoded picture buffer (DPB) will be based on the image and which steps will be based on an access unit (AU). Removing from the DPB. Outputting the image from the DPB. Decoding and storing the current decoded image in the DPB. Marking the current decoded image in the DPB. Also performing additional output of the image from the DPB.

[0691] In some configurations, the removal and output from the DPB may be based on at least one AU output flag, such as an AU output flag, an AU output flag without RASL, and/or an AU output flag without prior pictures. The AU output flag may be obtained based on syntax elements signaled in the bitstream and other conditions. The AU flags represent flags that are received and applied at the access unit (AU) level. In some cases, they will be flags that are different from similar flags that are signaled or received at the picture level, such as a picture output flag (e.g., pic_output_flag), a picture output flag without rasl (e.g., NoRaslOutputFlag), and/or a picture output flag without prior pictures (e.g., no_output_of_prior_pics flag).

[0692] In one configuration, the first AU output flag may be obtained and used by the DPB, and the second AU output flag is obtained and used by the DPB. In another configuration, the values of the whole picture output flag syntax element may be constrained to the same value for all coded pictures in the AU when the picture output flag is present in the first slice header. In some configurations, the values of the whole picture output flag syntax element may be constrained to the same value for all coded pictures in the AU when any picture output flag is not present in the first slice header.

[0693] In one configuration, deletion may be based on a picture, output of a picture may be based on an AU (access unit), storage and decoding may be based on a picture, marking may be based on a picture, and additional output of a picture may be based on an AU. Deletion from a DPB may delete one or more pictures from the DPB before the current picture is decoded.

[0694] In another configuration, the deletion may be based on an image, and the image output, decoding and storing, marking and additionally outputting the image may be based on an AU. In another configuration, the deletion, image output, decoding and storing, marking and additionally outputting the image may be based on an AU. In another configuration, the deletion, decoding and storing, and marking may be based on an image, and the image output and additionally outputting the image may be based on an AU. In another configuration, the deletion, image output, decoding and storing, marking and additionally outputting the image may be based on an image.

[0695] Marking the current decoded picture in the DPB may include a reference marking step and an output marking step. The reference marking step may be based on the picture, and the output marking step may be based on the AU. The DPB occupancy may be increased by one when the decoded picture is stored in the DPB in an empty storage buffer. The DPB occupancy may be decreased by one when the picture is output from the DPB. The DPB occupancy may be tracked "per level". The DPB occupancy may also be tracked for a set of output levels.

[0696] The DPB may include separately identifiable and managed image buffers for decoded images having one or more of different resolutions, different color depths, and different color characteristics. The DPB may include a common pool of image storage buffers. The decoded image may be stored in the image storage buffers based on at least one of size, resolution, and color depth. In one configuration, the decoded image may be stored in one slot of the image buffer in the image storage buffers.

[0697] The method may be performed by a decoder within an electronic device that complies with the scalable high-efficiency video coding (SHVC) standard. The method may also be performed by a decoder in an electronic device that complies with the multi-view high-efficiency video coding (MV-HEVC) standard.

[0698] An electronic device configured for encoding video is also disclosed. The electronic device includes a processor and a memory in electronic communication with the processor. Instructions in the memory are executable to begin parsing a first slice header of a current picture. The instructions in the memory are also executable to determine which steps performed by the decoded picture buffer (DPB) will be based on the picture, which steps will be based on the access unit (AU). The instructions in the memory are further executable to perform deletion from the DPB. The instructions in the memory are also executable to perform output of the picture from the DPB. The instructions in the memory are further executable to perform decoding and storage of the current decoded picture in the DPB. The instructions in the memory are also executable to mark the current decoded picture in the DPB. The instructions in the memory are further executable to perform additional output of the picture from the DPB.

[0699] Various configurations are now described with reference to the figures of the drawings, where like reference numerals may indicate functionally similar elements. The systems and methods, as generally described and illustrated in the figures of the drawings in the description, can be structured and designed in a wide variety of different configurations. Thus, the following more detailed description of several configurations, as shown in the figures of the drawings, is not intended to limit the scope as claimed in the claims, but is merely representative of the systems and methods.

[0700] Figure 63 is a block diagram illustrating video encoding between multiple electronic devices 2102a-b. A first electronic device 2102a and a second electronic device 2102b are illustrated. However, it should be noted that one or more of the features and functionality described with respect to the first electronic device 2102a and the second electronic device 2102b can be combined into a single electronic device 102 in some configurations. Each electronic device 102 can be configured to encode video and/or decode video. The electronic devices 102 can be configured to use a hybrid mode of operation of the decoded picture buffer (DPB). The hybrid mode of operation of the decoded picture buffer (DPB) refers to scenarios where the various steps of deletion, output (flush), storage, marking, and additional output (flush) performed on the decoded picture buffer (DPB) 116 occur on either a picture basis or an AU basis. Although specific combinations of these steps are referred to as being performed on a picture basis or an AU basis, all possible combinations of performing each of these steps separately on either a picture basis or an AU basis are supported.

[0701] As used in the description, an access unit (AU) refers to a set of network abstraction layer (NAL) units that are associated with each other according to a given classification rule, that are consecutive in decoding order, and that includes video coding layer (VCL) NAL units of all coded pictures associated with the same output time and their associated non-VCL NAL units. A base layer is a layer in which all VCL NAL units have a nuh_layer_id equal to 0. A coded picture is a coded representation of a picture that includes VCL NAL units with a specific nuh_layer_id value and that includes all tree coding units of the picture. In some cases, a coded picture may be referred to as a layer component. Additional details about the steps based on a picture or an access unit (AU) are given in relation to FIGS. 69 and 70 below.

[0702] In one configuration, each of the electronic devices 102 may comply with the high efficiency video coding (HEVC) standard, the scalable high efficiency video coding (SHVC) standard, or the high efficiency multi-view video coding (MV-HEVC) standard. The HEVC standard is a video compression standard that acts as a successor to H.264/MPEG-4 AVC (Advanced Video Coding), and provides improved video quality and increased data compression ratios. As used in the description, an image is an array of luminance samples in a monochrome format, or an array of luminance samples and two corresponding arrays of chroma samples in a color format of 4:2:0, 4:2:2 and 4:4:4 or some other color format. The operation of the hypothetical reference decoder (HRD) and the operation of the decoded picture buffer (DPB) 116 of the output queue are described for SHVC and MV-HEVC in JCTVC-M1008, JCTVC-L1008, JCTVC-D1004, JCT3V-C1004, JCTVC-L0453, and JCTVC-L0452.

[0703] The first electronic device 2102a may include an encoder 2108 and a service signaling module 2112. The first electronic device 2102a may receive an input image 2106. In some configurations, the input image 2106 may be obtained by recording on the first electronic device 2102a using an image input sensor, retrieved from memory and/or received from another electronic device 102. The encoder 2108 may encode the input image 2106 to create encoded data 2110. For example, the encoder 2108 may encode a series of input images 2106 (e.g., video). The encoded data 2110 may be digital data (e.g., a bitstream).

[0704] The overhead signaling module 2112 may generate overhead signaling based on the encoded data 2110. For example, the overhead signaling module 2112 may add overhead data to the encoded data 2110, such as slice header information, video parameter set (VPS) information, sequence parameter set (SPS) information, picture parameter set (PPS) information, picture order counter (POC) information, a reference picture designation, etc. In some configurations, the overhead signaling module 2112 may output a wrap indicator that indicates a transition between two sets of pictures.

[0705] The encoder 2108 (and the overhead signaling module 2112, for example) can output a bitstream 2114. The bitstream 2114 can include coded picture data based on the input picture 2106. In some configurations, the bitstream 2114 can also include overhead data, such as slice header information, VPS information, SPS information, PPS information, etc. Since the additional input pictures 2106 are coded, the bitstream 2114 can include one or more coded pictures. For example, the bitstream 2114 can include one or more coded reference pictures and/or other pictures.

[0706] The bitstream 2114 may be provided to the decoder 2104. In one example, the bitstream 2114 may be transmitted to the second electronic device 2102b using a wired or wireless communication link. In some cases, this may be done over a network, such as the Internet or a local area network (LAN). As illustrated in Fig. 63, the decoder 2104 may be implemented on the second electronic device 2102b separately from the encoder 2108 on the first electronic device 2102a. However, it should be noted that the encoder 2108 and the decoder 2104 may be implemented on the same electronic device 102 in some configurations. When the encoder 2108 and decoder 2104 are implemented on the same electronic device, for example, the bitstream 2114 may be provided over a bus to the decoder 2104 or stored in memory for retrieval by the decoder 2104.

[0707] The decoder 2104 may receive (e.g., obtain) the bitstream 2114. The decoder 2104 may generate a decoded image 2118 (e.g., one or more decoded images 2118) based on the bitstream 2114. The decoded image 2118 may be displayed, reproduced, stored in memory, and/or transmitted to another device, etc.

[0708] The decoder 2104 may include a decoded picture buffer (DPB) 116. The decoded picture buffer (DPB) 116 may be a buffer that stores decoded pictures for reference, output reordering, or an output delay specified for a hypothetical reference decoder (HRD). On the electronic device 102, the decoded picture buffer (DPB) 116 may be used to store reconstructed (e.g., decoded) pictures in the decoder 2104. These stored pictures may then be used, for example, in an inter-picture prediction engine. When pictures are decoded out of order, the pictures may be stored in the decoded picture buffer (DPB) 116 so that they can be displayed in order later.

[0709] In the H.264 or advanced video coding (AVC) standard, management of the decoded picture buffer (DPB) 116 (e.g., deletion, addition of pictures, reordering of pictures, etc.) is performed using memory management control operations (MMCO). Many different approaches to managing the decoded picture buffer (DPB) 116 are considered.

[0710] The decoder 2104 may include a hybrid DPB operation module 2120. The hybrid DPB operation module 2120 may provide for the DPB 116 control approaches with picture-based steps 2122 and/or for the DPB 116 control approaches with access unit (AU)-based steps 2124. For example, one advantage of using the AU-based steps 2124 for deletion marking, storage, and reference is that the various layers will use optimal memory of the DPB 116. Thus, the total memory required when using picture-based steps may be less. One advantage of using the AU-based steps 2124 for output (including output, output marking, and additional output) is that the output process can be simplified.

[0711] Figure 64 is a flow chart of a method 2200 for hybrid operation of the decoded picture buffer (DPB) 116. The method 2200 may be performed by the decoded picture buffer (DPB) 116 as part of the decoder 2104 in the electronic device 102. In one configuration, the method 2200 may be performed by a hybrid operation module 2120 of the decoded picture buffer (DPB). The decoded picture buffer (DPB) 116 may begin parsing 2202 a first slice header of the current picture. The decoded picture buffer (DPB) 116 may determine 2204 which steps of the hybrid operation of the decoded picture buffer (DPB) will be based on a picture and which steps will be based on an access unit (AU).

[0712] The decoded picture buffer (DPB) 116 may perform 2206 deletion (without output) from the decoded picture buffer (DPB). Deletion may delete pictures from the decoded picture buffer (DPB) 116 before the current picture is decoded. The decoded picture buffer (DPB) 116 may perform 2208 output (flushing) of a picture from the decoded picture buffer (DPB). Output (flushing) of a picture may refer to outputting pictures from the decoded picture buffer (DPB) 116 during erasure of the coded picture buffer (CPB). In some configurations, the term flushing may be used to indicate outputting one or more pictures from the decoded picture buffer (DPB) 116. Thus, the terms "flushing" and "output" may be used interchangeably.

[0713] The decoded picture buffer (DPB) 116 may decode 2210 and store the current picture in the decoded picture buffer (DPB) 116. The decoded picture buffer (DPB) 116 may mark 2212 the current decoded picture stored in the decoded picture buffer (DPB). For example, the decoded picture buffer (DPB) 116 may mark 2212 the current decoded picture as "not used for reference," "used for reference," "required for output," or "not required for output." The decoded picture buffer (DPB) 116 may also perform 2214 outputs of another picture (additional flushing) from the decoded picture buffer (DPB). In some configurations, repeated removal/flushing of pictures from the decoded picture buffer (DPB) 116 may occur until certain conditions are met.

[0714] Figure 65 is a flow chart of another method 2300 for hybrid operation of the decoded picture buffer (DPB) 116. For example, the method 2300 of Figure 65 may be a preferred method for hybrid operation of the decoded picture buffer (DPB) 116. The method 2300 may be performed by the decoded picture buffer (DPB) 116 as part of the decoder 2104 on the electronic device 102. In one configuration, the method 2300 may be performed by the hybrid operation module 2120 of the decoded picture buffer (DPB) 2120. The decoded picture buffer (DPB) 116 may begin parsing 2302 of the first slice header of the current picture. The term "hybrid" refers to the fact that some of the steps of the DPB 116 operation are performed on a picture basis, and some of the steps of the DPB 116 operation are performed on an AU basis. The DPB 116 may perform a picture-based deletion (without output) from the DPB 116 in step 2304. The DPB 116 may perform an AU-based output (flush) of a picture from the DPB 116 in step 2306. The DPB 116 may perform a picture-based decoding and storing of the current picture in the DPB in step 2308.

[0715] The decoded picture buffer (DPB) 116 may perform a picture-based marking 310 of the current decoded picture in the decoded picture buffer (DPB). The marking step performed by the decoded picture buffer (DPB) 116 may be further subdivided to include both a link marking step and an output marking step. As used in the description, marking a picture as "not used for reference" or "used for reference" refers to the link marking step. A decoded picture in the decoded picture buffer (DPB) 116 may only be marked as one of "not used for reference," "used for short-term reference," or "used for long-term reference" at any given time during the operation of the decoding process. Assigning one of these marks to a picture implicitly removes the other of the marks that is assigned to the picture. When an image is referred to as being marked as "used for reference", this collectively refers to the image being marked as either "used for short-term reference" or "used for long-term reference", but never both. As used in the description, marking an image as "required for output" or "not required for output" refers to the output marking step.

[0716] The decoded picture buffer (DPB) 116 can operate in such a way that the link marking step and the output marking step can occur either on a picture basis or on an access unit (AU) basis. In general, all possible combinations (usually four combinations) of performing these two marking steps separately on a picture basis or on an access unit (AU) basis are supported. However, it may be preferable that the link marking step is based on a picture and the output marking step is based on an access unit (AU).

[0717] The decoded picture buffer (DPB) 116 may also perform 2312 marking of output based on the access unit (AU) for the current decoded picture. The decoded picture buffer (DPB) 116 may perform 2314 another output (additional flushing) based on the access unit (AU) of the picture from the decoded picture buffer (DPB) 116. In some configurations, repeated removal/flushing of pictures from the decoded picture buffer (DPB) 116 may occur until some conditions are met.

[0718] In some approaches, bitstream conformance constraints may be required for flags in the slice segment header. In some cases, the constraints may apply to flags across all pictures belonging to the same access unit (AU). For example, the pic_output_flag and/or no_output_of_prior_pics_flag flags may be required to follow bitstream conformance constraints. For example, JCTVC-L1003, JCTVC-M1008, and JCT3V-D1004 each describe signaling in the slice segment header using pic_output_flag and no_output_of_prior_pics_flag. Additionally, flags such as PicOutputFlag, NoRaslOutputFlag, and NoOutputOfPriorPicsFlag may be derived based on the types of syntax elements and the NAL unit.

[0719] JCTVC-L1003, JCTVC-M1008, and JCT3V-D1004 also include DPB descriptions for HEVC, SПVC, and MV-HEVC. JCTVC-M1008 SHVC Draft Text 1 provides a draft text for a "scalable" extension of HEVC. JCT3V-D1004 MV-HEVC Draft Text 4 describes a draft text for a "multi-view" extension of HEVC.

[0720] A coded video bitstream according to JCTVC-L1003, JCTVC-M1008 and/or JCT3V-D1004 may include a syntax structure that is placed in logical data packets, commonly referred to as network abstraction layer (NAL) units. Each NAL unit includes a NAL unit header, such as a two-byte header (e.g., 16 bits) of a NAL unit, to identify the purpose of the associated data payload. For example, each coded slice (and/or picture) may be coded in one or more slice (and/or picture) NAL units. Other NAL units may be included for other categories of data, such as, for example, additional extension information, a coded slice of a picture with temporary sublayer access (TSA), a coded slice of a picture with step-by-step temporary sublayer access (STSA), a coded slice of a non-TSA, a non-STSA trailer picture, a coded slice of a picture with broken link access, a coded slice of a picture with immediate decoding refresh, a coded slice of a picture with random access, a coded slice of a decodable head picture, a coded slice of a picture marked for discarding, a video parameter set, a sequence parameter set, a picture parameter set, an access unit delimiter, an end of a sequence, an end of a bitstream, filler data and/or sequence information extension messages. Table (7) below illustrates one example of NAL unit codes and classes of NAL unit types. Other NAL unit types may also be included as needed.

[0721] It should also be understood that the NAL unit type values for the NAL units shown in Table (7) may be rearranged and reassigned. In addition, additional NAL unit types may be added or removed.

[0722] An intra-frame random access point (IRAP) picture is a coded picture for which each NAL unit of the video coding layer has a nal_unit_type in the range from BLA_W_LP to RSV_IRAP_VCL23, inclusive, as shown in Table (7) below. An IRAP picture includes only intra-frame (I) slices.

[0723] An instantaneous decoding refresh (IDR) picture is an IRAP picture for which each NAL unit of the video coding layer has a nal_unit_type equal to IDR_W_RADL or IDR_N_LP, as shown in Table (7). An instantaneous decoding refresh (IDR) picture includes only I-slices and may be the first picture in the bitstream in decoding order, or may appear later in the bitstream.

[0724] Each IDR picture is the first picture of a coded video sequence (CVS) in decoding order. A broken link access (BLA) picture is an IRAP picture for which each video coding layer NAL unit has a nal_unit_type equal to BLA_W_LP, BLA_W_RADL, or BLA_N_LP, as shown in Table (7).

[0725] A BLA picture includes only I-slices, and may be the first picture in the bitstream in decoding order, or may appear later in the bitstream. Each BLA picture begins a new coded video sequence and has the same effect on the decoding process as an IDR picture. However, a BLA picture includes syntax elements that specify a non-empty set of reference pictures.

[0726]

Table 42 nal_unit_type Block type name nal NAL unit content and raw byte sequence payload (RBSP) syntax structure NAL Unit Type Class 0
1 TRAIL_N
TRAIL_R Coded segment of non-TSA slice, non-STSA tail image
slice_segment_layer_rbsp() Video Coding Layer (VCL) 2
3 TSA_N
TSA_R Encoded slice segment in a TSA image slice_segment_layer_rbsp() VCL 4
5 STSA_N
STSA_R Coded image slice segment with step-wise temporal sublayer access (STSA) slice_segment_layer rbsp() VCL 6
7 RADL_N
RADL_R Coded Random Access Decodable Head (RADL) Slice Segment slice_segment_layer_rbsp() VCL 8
9 RASL_N
RASL_R Coded random access skipped leading (RASL) slice_segment_layer_rbsp() VCL 10
12
14 RSV_VCL_N10 RSV_VCL_ N12 RSV_VCL_ N14 Reserved non-IRAP VCL NAL unit types without sublayer reference VCL 11
13
15 RSV_VCL_R11 RSV_VCL_R13 RSV_VCL_R15 Reserved non-IRAP VCL sublayer referenced NAL unit types VCL 16
17
18 BLA_W_LP BLA_W_RADL BLA_N_LP Coded image slice segment with broken link access (BLA)
slice_segment_layer rbsp() VCL 19
20 IDR_W_RADL IDR_N_LP Coded image slice segment with instantaneous decoding refresh (IDR)
slice_segment_layer_rbsp() VCL 21 CRA_NUT Coded Random Access (CRA) Image Slice Segment slice_segment_layer_rbsp() VCL 22
23 RSV_IRAP_VCL22 RSV_IRAP_VCL23 Reserved VCL NAL Unit Types for IRAP VCL 24..31 RSV_VCL24..
RSV_VCL31 Reserved VCL NAL Unit Types for Non-IRAP VCL 32 VPS_NUT Video Parameter Set video_parameter_set_rbsp() Non-Video Coding Layer (non-VCL) 33 SPS_NUT Sequence parameter set seq_parameter_set_rbsp() non-VCL 34 PPS_NUT Image parameter set pic_parameter_set rbsp() non-VCL 35 AUD_NUT Access Unit Delimiter access_unit_delimiter_rbsp() non-VCL 36 EOS_NUT End of sequence end_of_ seq_rbsp() non-VCL 37 EOB_NUT End of bitstream end_of_bitstream_rbsp() non-VCL 38 FD_NUT Filler data filler_data_rbsp() non-VCL 39
40 PREFIX_SEI_NUT
SUFFIX_SEI_NUT Additional extended information sei_rbsp() non-VCL 41..47 RSV_NVCL41..RSV_NVCL47 Reserved non-VCL 48..63 UNSPEC48..
UNSPEC63 Not defined non-VCL

Table (7)

[0727] With reference to Table (8) below, the NAL unit header syntax may include two bytes of data, namely, 16 bits. The first bit may be a "forbidden_zero_bit", which is always set to zero at the beginning of a NAL unit. The next six bits may be a "nal_unit_type" field, which specifies the type of raw byte sequence payload ("RBSP") data structure included in the NAL unit, as shown in Table (7) above. The next 6 bits may be a "nuh_layer_id", which specifies the layer identifier. In some cases, these six bits may be specified as "nuh_reserved_zero_6bits" instead. nuh_reserved_zero_6bits may be equal to 0 in the base specification of the standard. In scalable video coding and/or syntax extensions, nuh_layer_id may specify that this particular NAL unit belongs to the layer identified by the value of these 6 bits.

[0728] The next syntax element may be "nuh_temporal_id_plus1". The element nuh_temporal_id_plus1 minus 1 may specify a temporal identifier for the NAL unit. The "temporal identifier" variable TemporalId may be specified as TemporalId=nuh_temporal_id_plus1-1. The TemporalId is used to identify the temporal sublayer. The variable HighestTid identifies the highest temporal sublayer to be decoded.

[0729] [Table 43]

[0730] Table (9) below shows the syntax structure of an exemplary sequence parameter set (SPS). The pic_width_in_luma_samples element specifies the width of each decoded picture in units of luma samples. The value of pic_width_in_luma_samples must not be equal to 0. The pic_height_in_luma_samples element specifies the height of each decoded picture in units of luma samples. The value of pic_height_in_luma_samples must not be equal to 0.

[0731] sps_max_sub_layers_minus1 plus 1 specifies the maximum number of temporary sublayers that can be present in each CVS that references an SPS. sps_max_sub_layers_minus1 has a value between 0 and 6, inclusive.

[0732] A sps_sub_layer_ordering_info_present_flag of 1 specifies that the sps_max_dec_pic_buffering_minus1[i], sps_max_num_reorder_pics[i], and sps_max_latency_increase_plus1[i] syntax elements are present for sps_max_sub_layers_minus1+1 sublayers. A sps_sub_layer_ordering_info_present_flag of 0 specifies that the sps_max_dec_pic_buffering_minus1[sps_max_sub_layers_minus1], sps_max_num_reorder_pics [sps_max_sub_layers_minus1], and sps_max_latency_increase_plus1 [sps_max_sub_layers_minus1] values apply to all sublayers.

[0733] sps_max_dec_pic_buffering_minus1[i] plus 1 specifies the maximum required decoded picture buffer size for CVS in units of picture storage buffers when HighestTid is i. sps_max_dec_pic_buffering_minus1[i] has a value from 0 to MaxDpbSize - 1, inclusive, where MaxDpbSize specifies the maximum decoded picture buffer size in units of picture storage buffers. When i is greater than 0, sps_max_dec_pic_buffering_minus1[i] will be greater than or equal to sps_max_dec_pic_buffering_minus1[i-1]. When sps_max_dec_pic_buffering_minus1[i] is not present for i in the range 0 to sps_max_sub_layers_minus1-1, inclusive, due to sps_sub_layer_ordering_info_present_flag being 0, it is assumed to be sps_max_dec_pic_buffering_minus1[sps_max_sub_layers_minus1].

[0734] The sps_max_num_reorder_pics[i] member specifies the maximum number of images that may precede any image in CVS in decoding order and follow that image in output order when HighestTid is i. sps_max_num_reorder_pics[i] has a value from 0 to sps_max_dec_pic_buffering_minus1[i], inclusive. When i is greater than 0, sps_max_num_reorder_pics[i] may be greater than or equal to sps_max_num_reorder_pics[i-1]. When sps_max_num_reorder_pics[i] is not present for i in the range 0 to sps_max_sub_layers_minus1-1, inclusive, due to sps_sub_layer_ordering_info_present_flag being 0, it is assumed to be sps_max_num_reorder_pics[sps_max_sub_layers_minus1].

[0735] sps_max_latency_increase_plus1[i] when not equal to 0 can be used to calculate the value of SpsMaxLatencyPictures[i], which specifies the maximum number of pictures that can precede any picture in CVS in output order and follow that picture in decoding order when HighestTid is i. When sps_max_latency_increase_plus1[i] is not equal to 0, the value of SpsMaxLatencyPictures[i] is given by SpsMaxLatencyPictures[i]=sps_max_num_reorder_pics[i]+sps_max_latency_increase_plus1[i] - l. When sps_max_latency_increase_plus1[i] is equal to 0, the corresponding bound is not expressed.

[0736] The value of sps_max_latency_increase_plus1[i] is between 0 and 2 ^{32 -} 2, inclusive. When sps_max_latency_increase_plus1[i] is not present for i in the range 0 to sps_max_sub_layers_minus1-1, inclusive, due to sps_sub_layer_ordering_info_present_flag being equal to 0, it is inferred to be sps_max_latency_increase_plus1[sps_max_sub_layers_minus1].

[0737] [Table 44]

[0738] In addition, JCTVC-L1003 describes the HEVC standard. For example, the detail regarding pic_out_flag and no_output_of_prior_pics_flag is given in Table (10) below:

[0739] [Table 45]

[0740] In Table (10), no_output_of_prior_pics_flag affects the output of pre-decoded pictures in the decoded picture buffer (DPB) after decoding an IDR or BLA picture that is not the first picture in the bitstream as defined in Appendix C of JCTVC-L1003.

[0741] output_flag_present_flag equal to 1 specifies that the pic_output_flag syntax element is present in the associated slice headers. output_flag_present_flag equal to 0 specifies that the pic_output_flag syntax element is not present in the associated slice headers. pic_output_flag affects the decoded picture output and discard processes as defined in JCTVC-L1003 Appendix C. When pic_output_flag is not present, it is implied to be 1.

[0742] For the general decoding process (as defined in 8.1 of JCTVC-L1003), PicOutputFlag is set as follows:

[0743] - If the current image is a RASL image, and NoRaslOutputFlag for the associated IRAP image is 1, PicOutputFlag may be set to 0.

[0744] - Otherwise, PicOutputFlag may be set to pic_output_flag.

In addition, during the general decoding process for generating unavailable (for use) reference pictures (as defined in section 8.3.3.1 of JCTVC-L1003), the PicOutputFlag value for the generated picture may be set to 0 under some conditions.

[0746] When the current image is an IRAP image, the following applies:

[0747] - If the current picture is an IDR picture, a BLA picture, the first picture in the bitstream in decoding order, or the first picture following the end of a sequence, NAL unit in decoding order, the NoRaslOutputFlag variable may be set to 1.

[0748] - Otherwise, if some external facility not defined in JCTVC-L1003 is available to set the HandleCraAsBlaFlag variable to the value for the current image, the HandleCraAsBlaFlag variable may be set equal to the value provided by the external facility, and the NoRaslOutputFlag variable may be set equal to HandleCraAsBlaFlag.

[0749] - Otherwise, the HandleCraAsBlaFlag variable may be set to 0, and the NoRaslOutputFlag variable may be set to 0.

[0750] As described above, an access unit (AU) refers to a set of network abstraction layer (NAL) units that are associated with each other according to a given classification rule, that are consecutive in decoding order, and that include video coding layer (VCL) NAL units of all coded pictures associated with the same output time and their associated non-VCL NAL units. A base layer is a layer in which all VCL NAL units have a nuh_layer_id equal to 0. A coded picture is a coded representation of a picture that includes VCL NAL units with a specific nuh_layer_id value and that includes all tree coding units of the picture. In some cases, a coded picture may be referred to as a layer component. Further details on the steps of being picture-based or AU-based are given with respect to FIGS. 69 and 70 below.

[0751] In some configurations, the bitstream matching constraints for pic_output_flag and/or no_output_of_prior_pics_flag may be used for coded pictures in an access unit (AU). In addition, three new access unit output flags, an AU output flag (e.g., AuOutputFlag), an AU output flag without RASL (e.g., AuNoRaslOutputFlag), and an AU output flag without prior pictures (e.g., AuNoOutputOfPriorPicsFlag), may be derived for an AU based on the value of various syntax elements and NAL unit types for coded pictures in the AU. In some configurations, picture output and removal may be based on these three flags (e.g., AuOutputFlag, AuNoRaslOutputFlag, and AuNoOutputOfPriorPicsFlag) for SHVC and multi-view HEVC.

[0752] For example, bitstream conformance constraints for HEVC extensions may be followed as described in the systems and methods herein. In particular, bitstream conformance constraints may be applied to the SHVC bitstream. In addition, bitstream conformance constraints may be applied to the MV-HEVC bitstream.

[0753] In one configuration, when present, the value of the pic_output_flag slice segment header syntax elements may be required to be the same in all slice segment headers for coded pictures in an access unit (AU). In another configuration, the value for the pic_output_flag slice segment header syntax elements, when present, may be the same in all slice segment headers of coded pictures in an access unit (AU) when the coded pictures have the same NAL unit type.

[0754] In one configuration, when present, the value of the pic_output_flag slice segment header syntax elements for slice segments with nuh_layer_id equal to the nuh_layer_id value for the target layer may be the same in all slice segment headers of such coded pictures in an access unit (AU). In another configuration, when present, the value of the pic_output_flag slice segment header syntax elements for slice segments with nuh_layer_id not equal to the nuh_layer_id value for the target layer may be 0 in all slice segment headers of such coded pictures in an access unit (AU).

[0755] In one configuration, the target layer may be a layer that belongs to a layer set or a target layer set or DpbOutputTime[n] as defined in JCTVC-L1003, JCTVC-M1008, or JCT3V-D1004. In another configuration, the target layer may be a layer that is intended to be decoded. In yet another configuration, the target layer may be a layer that is intended to be decoded and output (displayed or otherwise sent for output).

[0756] In some configurations, when present, the value of the no_output_of_prior_pics_flag slice segment header syntax elements may be the same in all slice segment headers of coded pictures in an AU. In other configurations, when present, the value of the no_output_of_prior_pics_flag slice segment header syntax elements may be the same in all slice segment headers of coded pictures in an AU when the coded pictures have the same NAL unit type. In one configuration, when present, the value of the no_output_of_prior_pics_flag slice segment header syntax elements for slice segments with a nuh_layer_id equal to the nuh_layer_id value of the target layer may be the same in all slice segment headers of such coded pictures in the AU.

[0757] In some configurations, the pic_output_flag and/or no_output_of_prior_pics_flag syntax elements may not be signaled when nuh_layer_id>0. In this case, the values for layers with nuh_layer_id>0 may be implied to be equal to their signaled values for nuh_layer_id equal to 0.

[0758] In some configurations, additional flags such as AuOutputFlag and AuNoRaslOutputFlag may be used. The AuOutputFlag and AuNoRaslOutputFlag flags may be obtained according to a number of approaches. In one approach or configuration, two flags, AuOutputFlag and AuNoRaslOutputFlag, may be obtained and used for DPB operation. AuOutputFlag may be set to 1 if PicOutputFlag is equal to 1 for all images in the AU. Otherwise, AuOutputFlag may be set to 0.

[0759] In another approach, AuOutputFlag may be set to 1 if PicOutputFlag is equal to 1 for at least one picture in the AU. Otherwise, AuOutputFlag may be set to 0. Thus, in this case, AuOutputFlag may be set to 0 if PicOutputFlag is equal to 0 for all pictures in the AU.

[0760] In another approach, AuOutputFlag may be set to 1 if PicOutputFlag is 1 for images that belong to all target output layers in the AU. Otherwise, AuOutputFlag may be set to 0.

[0761] In yet another approach, AuOutputFlag may be set to 1 if PicOutputFlag is 1 for a picture that belongs to at least one output target layer in the AU. Otherwise, AuOutputFlag may be set to 0.

[0762] In one approach, AuNoRaslOutputFlag may be set to 1 if NoRaslOutputFlag is 1 for all images in the AU. Otherwise, AuNoRaslOutputFlag may be set to 0.

[0763] In another approach, AuNoRaslOutputFlag may be set to 1 if NoRaslOutputFlag is 1 for at least one image in the AU. Otherwise, AuNoRaslOutputFlag may be set to 0. Thus, in this case, AuNoRaslOutputFlag may be set to 0 if NoRaslOutputFlag is 0 for all images in the AU.

[0764] In another approach, AuNoRaslOutputFlag may be set to 1 if NoRaslOutputFlag is 1 for images that belong to all target output layers in the AU. Otherwise, AuNoRaslOutputFlag may be set to 0.

[0765] In yet another approach, AuNoRaslOutputFlag may be set to 1 if NoRaslOutputFlag is 1 for an image that belongs to at least one output target layer in the AU. Otherwise, AuNoRaslOutputFlag may be set to 0.

[0766] In some of the above approaches and configurations, the DPB operation may use AuOutputFlag instead of PicOutputFlag. Additionally, the DPB operation may use AuNoRaslOutputFlag instead of NoRaslOutputFlag.

[0767] Examples showing the use of AU output flags, such as AuOutputFlag, and AuNoRaslOutputFlag, according to the present systems and methods are provided below in Listing (1A) and Listing (2A) below. In addition, as described below in Listing (1), Listing (1A), Listing (2), and Listing (2A), the AU output flag AuNoOutputOfPriorPicsFlag may be received and used for DPB operation.

[0768] The present systems and methods may be implemented by modifications to standards documents. List (1) below provides the sections of JCTVC-L1003 that are intended to be modified to provide the present systems and methods.

List 1

[0769] C.3 Operation of the Decoded Picture Buffer (DPB)

C.3.1 General information

The descriptions in this subclause apply independently to each set of selected DPB parameters as defined in subclause C.1. The DPB operates separately or independently for each layer. Thus, the subsequent steps take place separately for each decoded image with a specific nuh_layer_id value.

The decoded image buffer contains image storage buffers. Each layer consists of its own set of image storage buffers. Thus, the image storage buffers for each layer are associated with the layer's nuh_layer_id value. Each of the image storage buffers may contain a decoded image that is marked as "used for reference" or saved for future output. The processes defined in subclauses C.3.2, C.3.3, and C.3.4 are applied sequentially as follows.

C.3.2 Removing Images from DPB

The removal of pictures from the DPB before the decoding of the current picture (but after parsing the slice header for the first slice of the current picture) occurs immediately during the removal in the CPB of the first decoding block of the current picture belonging to access block n (containing the current picture), and continues as follows:

- Invoke the decoding process for RPS as defined in subclause 8.3.2.

- When the current image is an IRAP image with NoRaslOutputFlag equal to 1, which is not image 0, the following ordered steps are applied:

1. Obtain the NoOutputOfPriorPicsFlag variable for the decoder under test as follows:

- If the current image is a CRA image, NoOutputOfPriorPicsFlag is set to 1 (regardless of the value of no_output_of_prior_pics_flag).

- Otherwise, if the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_pic_buffering_minus1[HighestTid] obtained from the active SPS corresponding to the nuh_layer_id value of the current picture differs from the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_pic_buffering_minus1[HighestTid], respectively, obtained from the SPS active for the preceding picture with a nuh_layer_id value equal to the nuh_layer_id value of the current picture, NoOutputOfPriorPicsFlag may (but need not) be set to 1 by the decoder under test, regardless of the value of no_output_of_prior_pics_flag.

Note - Although setting NoOutputOfPriorPicsFlag equal to no_output_of_prior_pics_flag is preferred under these conditions, the decoder under test is allowed to set NoOutputOfPriorPicsFlag to 1 in this case.

- Otherwise, NoOutputOfPriorPicsFlag is set to no_output_ofjprior_pics_flag.

2. The NoOutputOfPriorPicsFlag value obtained for the decoder under test is applied to the HRD such that when the NoOutputOfPriorPicsFlag value is 1, all picture storage buffers corresponding to the nuh_layer_id value of the current picture in the DPB are cleared without outputting the pictures they contain, and the DPB fullness for the nuh_layer_id value of the current picture is set to 0.

In one implementation, the NoOutputOfPriorPicsFlag value obtained for the decoder under test is applied to the HRD, such that when the NoOutputOfPriorPicsFlag value is 1, all picture storage buffers corresponding to all nuh_layer_id values in the DPB are cleared without outputting the pictures they contain, and the DPB fullness for all nuh_layer_id values is set to 0.

In one implementation, the NoOutputOfPriorPicsFlag value obtained for the decoder under test is applied to the HRD such that when the NoOutputOfPriorPicsFlag value is 1, all PSB[currLayerId] picture storage buffers with a currLayerId equal to the nuh_layer_id value of the current picture in the DPB are cleared without outputting the pictures they contain, and the DPBFullness[currLayerId] for the nuh_layer_id value of the currLayerId value of the current picture is set to 0.

In one implementation, the NoOutputOfPriorPicsFlag value obtained for the decoder under test is applied to the HRD, such that when the NoOutputOfPriorPicsFlag value is 1, all PSB[nuh_layer_id] image storage buffers for all nuh_layer_id values in the DPB are cleared without outputting the images they contain, and the DPBFullness[nuh_layer_id] for all nuh_layer_id values is set to 0.

- When both of the following conditions are true for any k images in the image storage buffer corresponding to the nuh_layer_id value of the current image in the DPB, all such k images in the DPB are removed from the DPB:

- image k is marked as "not used for reference"

- image k has PicOutputFlag equal to 0 or its output time DPB is less than or equal to the removal time CPB of the first decoding block (denoted as decoding block m ) of the current image n; that is, DpbOutputTime[k] is less than or equal to CpbRemovalTime(m)

- For each image that is removed from the DPB, the DPB occupancy is decreased by one.

- In one execution, for each image k with nuh_layer_id value nuhLayerIdk that is removed from the DPB, the DPB fullness DPBfuliness[nuhLayerIdk] is decreased by one.

C.3.3 Image output

The processes defined in this subclause occur instantaneously at the time of removal in CPB of access block n, AuCpbRemovalTime[n].

When image n has PicOutputFlag equal to 1, its DPB output time, DpbOutputTime[n], is obtained as follows, where the variable firstPicInBufferingPeriodFlag is 1 if access block n is the first access block of the buffering period and 0 otherwise:

if(!SubPicHrdFlag) {

DpbOutputTime[n]=AuCpbRemovalTime[n]+ClockTick * picDpbOutputDelay (C-16)

if (firstPicInBufferingPeriodFlag)

DpbOutputTime[n] -= ClockTick * DpbDelayOffset

} else

DpbOutputTime[n]=AuCpbRemovalTime[n]+ClockSubTick * picSptDpbOutputDuDelay

where picDpbOutputDelay is the value of pic_dpb_output_delay in the picture synchronization SEI message associated with access block n, and picSptDpbOutputDuDelay is the value of pic_spt_dpb_output_du_delay, when present, in the decoding block information SEI messages associated with access block n, or the value of pic_dpb_output_du_delay in the picture synchronization SEI message associated with access block n when there is no decoding block information SEI message associated with access block n or there is no decoding block information SEI message associated with access block n with pic_spt_dpb_output_du_delay present.

Note - When the pic_spt_dpb_output_du_delay syntax element is not present in any decoding block information SEI message associated with access block n, the value is implied to be pic_dpb_output_du_delay from the picture synchronization SEI message associated with access block n.

The output of the current image, if its nuh_layer_id belongs to a layer in the TargetDecLayerIdList, is specified as follows:

- If PicOutputFlag is 1 and DpbOutputTime[n] is AuCpbRemovalTime[n], the current image is being output.

- Otherwise, if PicOutputFlag is 0, the current image is not output, but will be stored in the image storage buffer corresponding to the nuh_layer_id value of the current image in the DPB, as defined in subclause C.3.4.

In one embodiment: Otherwise, if PicOutputFlag is 0, the current image is not output, but will be stored in the image storage buffer PSB[currLayerId] corresponding to the nuh_layer_id currLayerId value of the current image in the DPB, as specified in subclause C.3.4.

- Otherwise (PicOutputFlag is 1 and DpbOutputTime[n] is greater than AuCpbRemovalTime[n], the current picture is subsequently output and will be stored in the picture storage buffer corresponding to the nuh_layer_id value of the current picture in the DPB (as defined in subclause C.3.4) and output during DpbOutputTime[n], unless specified to be output by decoding or inferring no_output_of_prior_pics_flag equal to 1 at a time that precedes DpbOutputTime[n].

In another implementation, the above steps are defined for:

Output the current image if its nuh_layer_id belongs to a layer that belongs to the output layer set corresponding to the (current) working point.

In another implementation, the above steps are defined for:

Output the current image (without checking whether it belongs to the TargetDecLayerIdList or the output layer set for the current working point).

If output, the image is cropped using the cropping window specified in the active SPS for the image.

When image n is an image being output and is not the last image in the bitstream being output, the value of the DpbOutputInterval[n] variable is obtained as follows:

DpbOutputInterval[n]=DpbOutputTime[nextPicInOutputOrder] - DpbOutputTime[n] (C-17)

where nextPicInOutputOrder is the image that follows image n in the output order and has PicOutputFlag equal to 1.

C.3.4 Marking and saving the current decoded image

The process defined in this subclause occurs immediately at the time of removal in CPB of access block n, CpbRemovalTime[n].

The current decoded image is stored in the DPB in an empty image storage buffer corresponding to the nuh_layer_id value of the current image, the DPB fullness for the nuh_layer_id value of the current image is incremented by one, and the current image is marked as "used for short-term reference".

In one embodiment:

The current decoded image with nuh_layer_id equal to currLayerId is stored in the DPB in an empty image storage buffer PSB[currLayerId] with currLayerId equal to the nuh_layer_id value of the current image in the DPB, the DPB fullness for the nuh_layer_id value of the current image DPBFullness[currLayerId] is incremented by one, and the current image is marked as "used for short-term reference".

C.4 Bitstream Conformance

The technical conditions from subparagraph C.4 apply.

C.5 Decoder Matching

F.8.1.1

C.5.1 General information

The technical conditions from subparagraph C.5.1 apply.

List (1)

[0770] As used in Listing (1) above, PSB refers to the picture storage buffer. DPBFullness refers to a variable used to describe the fullness of the decoded picture buffer (DPB) 116.

[0771] Listing (1A) below provides an alternative approach to Section C.3.2 of Listing (1) as provided in this document. In some configurations, Listing (1A) may represent only the changes to Section C.3.2 of JCTVC-L1003. Listing (1A) may use the AuNoOutputOfPriorPicsFlag and AuNoRaslOutputFlag flags defined above.

List 1A

[0772] C.3.2 Removing Images from DPB

The removal of pictures from the DPB before decoding the current picture (but after parsing the slice header for the first slice of the current picture) occurs immediately during the removal of the first decoding block of the current picture belonging to access block n (containing the current picture) into the CPB, and continues as follows:

- Invoke the decoding process for RPS as defined in subclause 8.3.2.

- When the current image is an IRAP image with AuNoRaslOutputFlag equal to 1, which is not the 0th image, the following ordered steps are applied:

- In another configuration, when the current image is an IRAP image with NoRaslOutputFlag equal to 1, which is not image 0, the following ordered steps are applied:

1. Obtain the AuNoOutputOfPriorPicsFlag variable for the decoder under test as follows:

- If the current image is a CRA image, AuNoOutputOfPriorPicsFlag is set to 1 (regardless of the value of no_output_of_prior_pics_flag for the current image or other images in the AU).

- Otherwise, if the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_pic_buffering_minus1[HighestTid] obtained from the active SPS differs from the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_buffering_minus1[HighestTid], respectively, obtained from the SPS active for the preceding picture, AuNoOutputOfPriorPicsFlag may (but need not) be set to 1 by the decoder under test regardless of the value of no_output_of_prior_pics_flag.

Note - Although setting AuNoOutputOfPriorPicsFlag equal to no_output_of_prior_pics_flag is preferred under these conditions, the decoder under test is allowed to set AuNoOutputOfPriorPicsFlag to 1 in this case.

- Otherwise, AuNoOutputOfPriorPicsFlag is set based on the value of no_output_pf_prior_pics_flag for the current image and other images in the AU, as follows:

- AuNoOutputOfPriorPicsFlag is set to 1 if no_output_of_prior_pics_flag is 1 for at least one image in the AU. Otherwise, AuNoOutputOfPriorPicsFlag is set to 0. So in this case, AuNoOutputOfPriorPicsFlag is set to 0 if no_output_of_prior_pics_flag is 0 for all images in the AU.

- In another implementation, AuNoOutputOfPriorPicsFlag is set to 1 if no_output_ofjprior_pics_Flag is 1 for the current image. Otherwise, AuNoOutputOfPriorPicsFlag is left unchanged.

- In another implementation, AuNoOutputOfPriorPicsFlag is set to 1 if no_output_of_prior_pics_flag is 1 for all images in the AU. Otherwise, AuNoOutputOfPriorPicsFlag is set to 0.

- In another implementation, AuNoOutputOfPriorPicsFlag is set to 1 if no_output_of_prior_pics_flag is 1 for all images belonging to the output target layers in the AU. Otherwise, AuNoOutputOfPriorPicsFlag is set to 0.

- In another implementation, AuNoOutputOfPriorPicsFlag is set to 1 if no_output_of_prior_pics_flag is 1 for at least one picture belonging to the output target layers in the AU. Otherwise, AuNoOutputOfPriorPicsFlag is set to 0.

2. The value of AuNoOutputOfPriorPicsFlag obtained for the decoder under test is applied to HRD, such that when the value of AuNoOutputOfPriorPicsFlag is 1, all picture storage buffers corresponding to all nuh_layer_id values in the DPB are cleared without outputting the pictures they contain, and the DPB fullness for all nuh_layer_id values is set to 0.

In another implementation, the AuNoOutputOfPriorPicsFlag value obtained for the decoder under test is applied to the HRD, such that when the AuNoOutputOfPriorPicsFlag value is 1, all picture storage buffers corresponding to the nuh_layer_id value of the current picture in the DPB are cleared without outputting the pictures they contain, and the DPB fullness for the nuh_layer_id value of the current picture is set to 0.

List (1A)

[0773] List (2) below provides the sections of JCTVC-L1008 that are proposed to be modified to provide the present systems and methods.

List 2

[0774] F.13 Hypothetical Reference Decoder

F.13.1 General

The technical conditions of subparagraph C. 1 apply.

F.13.2 Coded Picture Buffer (CPB) Operation

The technical conditions of subparagraph C.2 apply.

F.13.3 Operation of the Decoded Picture Buffer (DPB)

The technical conditions of subclause C.3 apply separately for each set of decoded images with a specific nuh_layer_id value.

PicOutputFlag for images that are not included in the output target level is set to 0.

Decoded images with the same DPB output time and with PicOutputFlag equal to 1 are output in the ascending order of the nuh_layer_id values of these decoded images.

F.13.5 Decoder Matching

F.13.5.1 General information

The technical conditions according to subparagraph C.5. 1 apply.

F.13.5.2 Operation of the DPB Output Queue

F.13.5.2.1 General information

The decoded picture buffer contains picture storage buffers. Each layer consists of its own set of picture storage buffers. The picture storage buffers of each layer are thus associated with the nuh_layer_id value of the layer. The number of picture storage buffers for nuh_layer_id equal to 0 is obtained from the active SPS of the layer with nuh_layer_id equal to 0. The number of picture storage buffers for each non-zero value of nuh_layer_id is obtained from the SPS of the active layer for that non-zero value of nuh_layer_id. Each of the picture storage buffers contains a decoded picture that is marked as "used for reference" or saved for future output. The process for outputting and removing pictures from the DPB as defined in subclause F.13.5.2.2 is initiated, followed by a call to the process for decoding, marking, optionally flushing, and storing the picture as defined in subclause F.13.5.2.3. The "dropping" process is defined in subclause F.13.5.2.4 and is initiated as defined in subclauses F.13.5.2.2 and F.13.5.2.3.

F.13.5.2.2 Displaying and Deleting Images from DPB

The output and removal of pictures from the DPB before decoding the current picture (but after parsing the slice header for the first slice of the current picture) occurs immediately when the first decoding block of the current picture belonging to the access block containing the current picture is removed from the CPB and continues as described below. The decoding process for the RPS is called as defined in subclause 8.3.2.

- If the current image is an IRAP image with NoRaslOutputFlag equal to 1 and with nuh_layer_id equal to 0, which is not image 0, the following ordered steps are applied:

1. Obtain the NoOutputOfPriorPicsFlag variable for the decoder under test as follows:

- If the current image is a CRA image, NoOutputOfPriorPicsFlag is set to 1 (regardless of the value of no_output_of_prior_pics_flag).

- Otherwise, if the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_pic_buffering_minus1[HighestTid] obtained from the active SPS corresponding to the nuh_layer_id value of the current picture identifier differs from the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_pic_buffering_minus1[HighestTid], respectively, obtained from the SPS active for the preceding picture with a nuh_layer_id value equal to the nuh_layer_id value of the current picture, NoOutputOfPriorPicsFlag may (but need not) be set to 1 by the decoder under test regardless of the value of no_output_of_prior_pics_flag.

Note - Although setting NoOutputOfPriorPicsFlag equal to no_output_of_prior_pics_flag is preferred under these conditions, the decoder under test is allowed to set NoOutputOfPriorPicsFlag to 1 in this case.

- Otherwise, NoOutputOfPriorPicsFlag is set to no_output_of_prior_pics_flag.

2. The NoOutputOfPriorPicsFlag value obtained for the decoder under test is applied to HRD as follows:

- If NoOutputOfPriorPicsFlag is 1, all image storage buffers corresponding to the current image's nuh_layer_id value in the DPB are cleared without outputting the images they contain, and the DPB fullness for the current image's nuh_layer_id value is set to 0.

In one implementation, if NoOutputOfPriorPicsFlag is 1, all image storage buffers corresponding to all nuh_layer_id values in the DPB are cleared without outputting the images they contain, and the DPB fullness for all nuh_layer_id values is set to 0.

In one implementation, if NoOutputOfPriorPicsFlag is 1, all image storage buffers PSB[currLayerId] corresponding to the nuh_layer_id currLayerId value of the current image in the DPB are cleared without outputting the images they contain, and the DPB fullness, DPBFullness[currLayerId] is set to 0.

In one implementation, if NoOutputOfPriorPicsFlag is 1, all PSB[nuh_layer_id] image storage buffers for all nuh_layer_id values in the DPB are cleared without outputting the images they contain, and the DPBFullness[nuh_layer_id] for all nuh_layer_id values is set to 0.

- Otherwise (NoOutputOfPriorPicsFlag is 0), all image storage buffers containing an image that is marked as "not required for output" and "not used for reference" are cleared (without output), and all non-empty image storage buffers in the DPB corresponding to the nuh_layer_id value of the current image are cleared by repeatedly initiating the "flushing" process defined in subclause F.13.5.2.4, and the DPB fullness for the nuh_layer_id value of the current image is set to 0.

In another implementation, when NoOutputOfPriorPicsFlag is 0, all image storage buffers corresponding to all nuh_layer_id values containing an image that is marked as "not required for output" and "not used for reference" are cleared (without output), and all non-empty image storage buffers corresponding to all nuh_layer_id values in the DPB are cleared by repeatedly initiating the "flushing" process defined in subclause F.13.5.2.4, and the DPB fullness for all nuh_layer_id values is set to 0.

- In another implementation Otherwise (NoOutputOfPriorPicsFlag is 0), all image storage buffers containing an image that is marked as "not required for output" and "not used for reference" are cleared (without output), and all non-empty image storage buffers PSB[currLayerId] in the DPB corresponding to the nuh_layer_id value currLayerId of the current image are cleared by repeatedly initiating the "flushing" process defined in subclause F.13.5.2.4, and the DPB Fullness[currLayerId] for the nuh_layer_id value of the current image is set to 0.

- In another implementation, when NoOutputOfPriorPicsFlag is 0, all PSB[nuh_layer_id] image storage buffers for all nuh_layer_id values in the DPB containing the image that is marked as "not required for output" and "not used for reference" are cleared (without output), and all non-empty PSB[nuh_layer_id] image storage buffers corresponding to all nuh_layer_id values in the DPB are cleared by repeatedly initiating the "flushing" process defined in subclause F.13.5.2.4, and the DPBFullness[nuh_layer_id] for all nuh_layer_id values is set to 0.

- Otherwise (the current image is not an IRAP image with NoRaslOutputFlag equal to 1 and with nuh_layer_id equal to 0), all image storage buffers corresponding to the nuh_layer_id value of the current image that contain an image that is marked as "not required for output" and "not used for reference" are flushed (without output). For each image storage buffer that is flushed, the occupancy of the DPB corresponding to the nuh_layer_id value of the current decoded image is decreased by one. The variable currLayerId is set equal to the nuh_layer_id of the current decoded picture and when one or more of the following conditions are true, the "flushing" process defined in subclause F.13.5.2.4 is called repeatedly, further decreasing the DPB occupancy by one for each additional picture storage buffer corresponding to the nuh_layer_id of the current decoded picture that is flushed, until none of the following conditions is true:

- The number of images with nuh_layer_id equal to currLayerId in the DPB that are marked as "required for output" is greater than sps_max_num_reorderjpics[HighestTid] from the active SPS (when currLayerId is 0), or from the active layer SPS for the value of currLayerId.

- sps_max_latency_increase_plus1[HighestTid] of the active SPS (when currLayerId is 0) or the active layer SPS for a currLayerId value that is not equal to 0, and there is at least one picture with nuh_layer_id equal to currLayerId in the DPB that is marked as "required for output" for which the associated variable PicLatencyCount[currLayerId] is greater than or equal to SpsMaxLatencyPictures[HighestTid] obtained from the active SPS (when currLayerId is 0) or from the active layer SPS for a currLayerId value.

- The number of images with nuh_layer_id equal to currLayerId in the DPB is greater than or equal to sps_max_dec_pic_buffering_minus1[HighestTid]+1 from the active SPS (when currLayerId is 0), or from the active layer SPS for the value of currLayerId.

In another implementation: Otherwise (the current image is not an IRAP image with NoRaslOutputFlag equal to 1 and with nuh_layer_id equal to 0), all image storage buffers corresponding to all nuh_layer_id values containing an image that are marked as "not required for output" and "not used for reference" are cleared (no output). For each image storage buffer that is cleared, the DPB fullness corresponding to the image's nuh_layer_id value is decreased by one.

In another implementation: Otherwise (the current image is not an IRAP image with NoRaslOutputFlag equal to 1 and with nuh_layer_id equal to 0), all image storage buffers PSB[currLayerId] in the DPB corresponding to the nuh_layer_id value currLayerId of the current image containing images that are marked as "not required for output" and "not used for reference" are cleared (no output). For each image storage buffer that is cleared, the fullness of the DPB DPBfullness[currLayerId] corresponding to the nuh_layer_id value of the current decoded image is decreased by one.

In another implementation: Otherwise (the current image is not an IRAP image with NoRaslOutputFlag equal to 1 and with nuh_layer_id equal to 0), all image storage buffers PSB[nuh_layer_id] for all nuh_layer_id values in the DPB containing the image that are marked as "not required for output" and "not used for reference" are cleared (no output). For each image storage buffer that is cleared, the DPBFullness[nuh_layer_id] corresponding to the nuh_layer_id value of the cleared image is decreased by one.

F.13.5.2.3 Decoding, marking, additional resetting and storing of the image

The processes defined in this subclause occur immediately when the last decoding block of access block n containing the current picture is removed from the CPB.

The variable currLayerId is set to the nuh_layer_id of the currently decoded image. For each image in the DPB that is marked as "required for output" and that has a nuh_layer_id equal to currLayerId, the associated variable PicLatencyCount[currLayerId] is set to PicLatencyCount[currLayerId]+1.

The current image is considered decoded after the last decoding block in the image is decoded. The current decoded image is stored in an empty image storage buffer corresponding to currLayerId (the nuh_layer_id value of the current image) in the DPB, and the following applies:

- If the PicOutputFlag of the current decoded image is 1, it is marked as "required for output" and its associated PicLatencyCount[currLayerId] variable is set to 0.

- Otherwise (PicOutputFlag of the current decoded image is 0), it is marked as "not required for output."

In one implementation, the current decoded image is stored in an empty image storage buffer corresponding to currLayerId (the nuh_layer_id value of the current image) in the DPB, and the following is applied:

- If the PicOutputFlag of the current decoded image is 1, it is marked as "required for output" and its associated PicLatencyCount[currLayerId] variable is set to 0. All image storage buffers containing an image that has the same picture order counter value (PicOrderCntVal) as the current decoded image are marked as "required for output."

- Otherwise (PicOutputFlag of the current decoded image is 0), it is marked as "not required for output".

In one implementation, the current decoded image is stored in an empty image storage buffer PSB[currLayerId] corresponding to currLayerId (the nuh_layer_id value of the current image) in the DPB, and the DPB fullness for the nuh_layer_id value of the current image, DPBFullness[currLayerId] is incremented by one, and the following is applied

The currently decoded image is marked as "used for short-term reference."

When one or more of the following conditions are true, the "drop" process defined in subclause F.13.5.2.4 is called repeatedly until none of the following conditions is true.

- The number of images with nuh_layer_id equal to currLayerId in the DPB that are marked as "required for output" is greater than sps_max_num_reorder_pics[HighestTid] from the active SPS (when currLayerId is 0), or from the active layer SPS for the value of currLayerId.

- sps_max_latency_increase_plus1[HighestTid] is not 0 and there is at least one image in the DPB with nuh_layer_id equal to currLayerId that is marked as "required for output" for which the associated variable PicLatencyCount[currLayerId] is greater than or equal to SpsMaxLatencyPictures[HighestTid] obtained from the active SPS (when currLayerId is 0) or from the active layer SPS for the value of currLayerId.

F.13.5.2.4 The "dropping" process

The "dropping" process consists of the following ordered steps:

1. The images that are first to be output are selected as those with the lowest PicOrderCntVal value from all images in the DPB marked as "required for output".

2. These images are cropped using the cropping window specified in the active SPS for an image with nuh_layer_id equal to 0 or in the active layer SPS for a nuh_layer_id equal to that of the image, the cropped images are output in ascending order of nuh_layer_id, and the images are marked as "not required for output"

3. Each image storage buffer that contains an image marked as "not used for reference" and that included one of the images that was cropped and output is cleared.

List (2)

[0775] Listing (2A) below provides an alternative approach to section F.13.5.2.2 of Listing (2) as provided by these systems and methods. In some configurations, Listing (2A) may only represent changes to section F.13.5.2.2 in JCTVC-L1008. Listing (2A) may use the AuNoOutputOfPriorPicsFlag and AuNoRaslOutputFlag flags defined above.

List 2A

[0776] F.13.5.2.2 Outputting and Deleting Images from DPB

The deletion and removal of pictures from the DPB before the current picture is decoded (but after the slice header for the first slice of the current picture is parsed) occurs immediately when the first decoding block of the access block containing the current picture is removed from the CPB, and continues as follows:

Invoke the decoding process for RPS as defined in subclause 8.3.2.

- If the current image is an IRAP image with AuNoRaslOutputFlag equal to 1 and nuh_layer_id equal to 0, which is not image 0, the following ordered steps are applied:

- In another configuration, if the current image is an IRAP image with NoRaslOutputFlag equal to 1 and with nuh_layer_id equal to 0, which is not image 0, the following ordered steps are applied:

1. The AuNoOutputOfPriorPicsFlag variable is obtained for the decoder under test as described below.

- If the current image has nuh_layer_id equal to 0, AuNoOutputOfPriorPicsFlag is initialized to 0. Then:

- If the current image is a CRA image, AuNoOutputOfPriorPicsFlag is set to 1 (regardless of the value of no_output_of_prior_pics_flag for the current image or other images in the AU).

- Otherwise, if the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_dec_pic_buffering_minus1[HighestTid] obtained from the active SPS differs from the value of pic_width_in_luma_samples, pic_height_in_luma_samples, or sps_max_decjpic_buffering_minus1[HighestTid], respectively, obtained from the SPS active for the preceding picture, AuNoOutputOfPriorPicsFlag may (but need not) be set to 1 by the decoder under test regardless of the value of no_output_of_prior_pics_flag.

Note - Although setting AuNoOutputOfPriorPicsFlag to no_output_of_prior_pics_flag is preferred under these conditions, the decoder under test is allowed to set AuNoOutputOfPriorPicsFlag to 1 in this case.

- Otherwise, AuNoOutputOfPriorPicsFlag is set based on the value of no_output_of_prior_pics_flag for the current image and other images in the AU, as follows:

- AuNoOutputOfPriorPicsFlag is set to 1 if no_output_of_prior_pics_flag is 1 for at least one image in the AU. Otherwise, AuNoOutputOfPriorPicsFlag is set to 0. So in this case, AuNoOutputOfPriorPicsFlag is set to 0 if no_output_of_prior_pics_flag is 0 for all images in the AU.

- In another implementation, AuNoOutputOfPriorPicsFlag is set to 1 if no_output_of_prior_pics_flag is 1 for the current image. Otherwise, AuNoOutputOfPriorPicsFlag is left unchanged.

- In another implementation, AuNoOutputOfPriorPicsFlag is set to 1 if no_output_of_prior_pics_flag is 1 for all images in the AU. Otherwise, AuNoOutputOfPriorPicsFlag is set to 0.

- In another implementation, AuNoOutputOfPriorPicsFlag is set to 1 if no_output_of_prior_pics_flag is 1 for all images belonging to the output target layers in the AU. Otherwise, AuNoOutputOfPriorPicsFlag is set to 0.

- In another implementation, AuNoOutputOfPriorPicsFlag is set to 1 if no_output_of_prior_pics_flag is 1 for at least one image belonging to the output target layer in the AU. Otherwise, AuNoOutputOfPriorPicsFlag is set to 0.

2. The AuNoOutputOfPriorPicsFlag value obtained for the decoder under test is applied to HRD as follows:

- If AuNoOutputOfPriorPicsFlag is 1, all image storage buffers in the DPB corresponding to all nuh_layer_id values are cleared without outputting the images they contain, and the DPB fullness is set to 0 for all nuh_layer_id values.

- In another implementation, if AuNoOutputOfPriorPicsFlag is 1, all image storage buffers in the DPB corresponding to all nuh_layer_id values of the current picture are cleared without outputting the pictures they contain, and the DPB fullness is set to 0 for the nuh_layer_id value of the current picture. Otherwise (AuNoOutputOfPriorPicsFlag is 0), all image storage buffers corresponding to all nuh_layer_id values containing a picture that is marked as "not required for output" and "not used for reference" are cleared (without output), and all non-empty image storage buffers in the DPB are cleared by repeatedly initiating the "flushing" process defined in subclause F.13.5.2.4, and the DPB fullness for all nuh_layer_id values is set to 0. In another implementation:

Otherwise (AuNoOutputOfPriorPicsFlag is 0), all image storage buffers corresponding to the current picture's nuh_layer_id value that contain an image that is marked as "not required for output" and "not used for reference" are cleared (without output), and all non-empty image storage buffers in the DPB are cleared by repeatedly initiating the "flushing" process defined in subclause F.13.5.2.4, and the DPB fullness for the current picture's nuh_layer_id value is set to 0.

List (2A)

[0777] In one configuration, for the proposed text in List (1), List (1A), List (2), and List (2A) above, all occurrences of "current image" may be replaced with "current decoded image."

[0778] Listing (3) below provides an additional section regarding decoding to provide the present systems and methods. Listing (3) provides sections of JCTVC-L1003 (i.e., version 34 of the HEVC specification) that are intended to be modified to provide the present systems and methods.

List 3

[0779] 8.3.2 Decoding Process for a Set of Reference Pictures

When the current image is an IRAP image with NoRaslOutputFlag equal to 1, all reference images currently in the DPB corresponding to the current image's nuh_layer_id value (if any) are marked as "not used for reference."

In another implementation: When the current image is an IRAP image with NoRaslOutputFlag equal to 1, all reference images currently in the DPB corresponding to all nuh_layer_id values (if any) are marked as "unused for reference."

List (3)

[0780] Many different options can be used to specify hybrid decoded picture buffer (DPB) 116 operations with different stages being on a picture basis or on an access unit basis. Tables (11) - (15) below list multiple varieties of stages and corresponding picture basis/access unit (AU) basis settings.

[0781] [Table 46]

Table 46 Stage Image Based/AU Based Removal Based on the image Conclusion Based on AU Preservation Based on the image Note Based on AU for output labeling,
based on image for link tagging Additional conclusion Based on AU Table (11)

[0782] [Table 47]

Table 47 Stage Image Based/AU Based Removal Based on the image Conclusion Based on AU Preservation Based on AU Note Based on AU for output labeling,
based on AU for link tagging Additional conclusion Based on AU Table (12)

[0783] [Table 48]

Table 48 Stage Image Based/AU Based Removal Based on AU Conclusion Based on AU Preservation Based on AU Note Based on AU for output labeling,
based on AU for link tagging Additional conclusion Based on AU Table (13)

[0784][Table 49]

Table 49 Stage Image Based/AU Based Removal Based on the image Conclusion Based on AU Preservation Based on the image Note Based on the image to mark the output,
Based on image to mark link Additional conclusion Based on AU Table (14)

[0785] [Table 50]

Table 50 Stage Image Based/AU Based Removal Based on the image Conclusion Based on the image Preservation Based on the image Note Based on the image to mark the output,
based on image for link tagging Additional conclusion Based on the image Table (15)

[0786] Figure 66 is a block diagram illustrating one configuration of a decoder 2404. The decoder 2404 may be included in the electronic device 2402. For example, the decoder 2404 may be a high efficiency video coding (HEVC) decoder. The decoder 2404 and/or one or more of the elements illustrated as included in the decoder 2404 may be implemented in hardware, software, or a combination of both. The decoder 2404 may receive a bitstream 2414 (e.g., one or more coded pictures included in the bitstream 2414) for decoding. In some configurations, the received bitstream 2414 may include received overhead information, such as received slice header information, a received PPS, a received buffer description, etc. The coded pictures included in the bitstream 2414 may include one or more coded reference pictures and/or one or more other coded pictures.

[0787] The received symbols (in one or more coded images included in the bitstream 2414) may be entropy decoded by an entropy decoding module 454, thereby producing a motion data signal 456 and quantized, scaled, and/or transformed coefficients 458.

[0788] The motion data signal 456 may be combined with a portion of the reference frame signal 484 from the frame memory 464 in the motion compensation module 460, which may output an inter-prediction signal 468. The quantized, descaled, and/or transformed coefficients 458 may be inversely quantized, scaled, and inversely transformed by the inversion (inverse transform/scaling/quantization) module 462, thereby creating a decoded residual signal 470. The decoded residual signal 470 may be summed with the prediction signal 478 to create a combined signal 472. The prediction signal 478 may be a signal selected from either the inter-prediction signal 468 or the intra-prediction signal 476 output by the intra-prediction module 474. In some configurations, this signal selection may be based on (e.g. controlled by) the 2414 bitstream.

[0789] The intra-frame prediction signal 476 may be predicted based on previously decoded information from the combined signal 472 (in the current frame, for example). The combined signal 472 may also be filtered by a deblocking filter 480. The resulting filtered signal 482 may be written to the frame memory 464. The resulting filtered signal 482 may include a decoded image.

[0790] Frame memory 464 may include a decoded picture buffer (DPB) 2416, as described herein. The decoded picture buffer (DPB) 2416 may be capable of hybrid operations of the decoded picture buffer (DPB) 116. The decoded picture buffer (DPB) 2416 may include one or more decoded pictures, which may be stored as short-term or long-term reference frames. Frame memory 464 may also include overhead information corresponding to the decoded pictures. For example, frame memory 464 may include slice headers, video parameter set (VPS) information, sequence parameter set (SPS) information, picture parameter set (PPS) information, loop parameters, buffer description information, etc. One or more of these pieces of information may be signaled from an encoder (e.g., encoder 2108, overhead signaling module 2112).

[0791] Figure 67A is a block diagram illustrating the use of both an enhancement layer and a base layer for coding video with separate decoded picture buffers (DPBs) 516a-b and separate hybrid DPB operation units 520a-b for the base layer and the enhancement layer. A first electronic device 502a and a second electronic device 502b are illustrated. The first electronic device 502a may include a video encoder 508 that includes an enhancement layer encoder 526 and a base layer encoder 528. Each of the elements included in the first electronic device 502a (that is, the enhancement layer encoder 526 and the base layer encoder 528) may be implemented in hardware, software, or a combination of both. The first electronic device 502a may receive an input image 2506. In some configurations, the input image 2506 may be obtained by recording on the first electronic device 502a using an image input sensor, retrieved from memory, or retrieved from another electronic device 502.

[0792] The enhancement layer encoder 526 may encode the input image 2506 to generate encoded data. For example, the enhancement layer encoder 526 may encode a series of input images 2506 (e.g., video). The encoded data may be included in the encoded bitstream 530 of the enhancement layer video. The enhancement layer encoder 526 may generate service signaling based on the input image 2506.

[0793] The enhancement layer video decoder 534 may include a decoded picture buffer (DPB) 516a and a hybrid DPB operation module 520a. Similarly, the base layer decoder 536 may include a decoded picture buffer (DPB) 516b and a hybrid DPB operation module 520b.

[0794] The base layer encoder 528 may also encode the input image 2506. In one configuration, the same input image 2506 used by the enhancement layer encoder 526 may also be used by the base layer encoder 528. In another configuration, a different (but similar) input image than the input image 2506 used by the enhancement layer encoder 526 may be used by the base layer encoder 528. For example, for signal-to-noise ratio (SNR) scalability (also referred to as quality scalability), the same input image 2506 may be used by both the enhancement layer encoder 526 and the base layer encoder 528. As another example, for spatial scalability, a subsampled image may be used by the base layer encoder 528. In yet another example, for multi-view scalability, an image with a different view may be used by the base layer encoder 528. The base layer encoder 528 may create encoded data included in the encoded base layer video bitstream 532.

[0795] The coded enhancement layer video bitstream 530 and the coded base layer video bitstream 532 may each include coded data based on the input image 2506. In one example, the coded enhancement layer video bitstream 530 and the coded base layer video bitstream 532 may include coded image data. In some configurations, the coded enhancement layer video bitstream 530 and/or the coded base layer video bitstream 532 may also include service data such as sequence parameter set (SPS) information, picture parameter set (PPS) information, video parameter set (VPS) information, slice header information, etc.

[0796] The coded bitstream 530 of the enhancement layer video may be provided to the second electronic device 502b. Similarly, the coded bitstream 532 of the base layer video may be provided to the second electronic device 502b. The second electronic device 502b may include a video decoder 2504. The video decoder 2504 may include an enhancement layer decoder 534 and a base layer decoder 536. In one configuration, the coded bitstream 530 of the base layer video is decoded by the base layer decoder 536, while the coded bitstream 530 of the enhancement layer video is decoded by the enhancement layer decoder 534.

[0797] In one example, the coded bitstream 530 of the enhancement layer video and the coded bitstream 532 of the base layer video may be transmitted to the second electronic device 502b using a wired or wireless link. In some cases, this may be done over a network, such as the Internet, a Local Area Network (LAN) or another type of network for communicating between devices. It should be noted that in some configurations, the encoders (that is, the enhancement layer encoder 526 and the base layer encoder 528) and the decoder 2504 (for example, the base layer decoder 536 and the enhancement layer decoder 534) may be implemented on the same electronic device 502 (that is, the first electronic device 502a and the second electronic device 502b may be part of the same electronic device 502). In an implementation where the encoders and decoders are implemented on the same electronic device 502, for example, the coded bitstream 530 of the enhancement layer video and the coded bitstream 532 of the base layer video may be made available to the video decoder 2504 in different ways. For example, the coded bitstream 530 of the enhancement layer video and the coded bitstream 532 of the base layer video may be provided over a bus to the video decoder 2504 or stored in memory for retrieval by the video decoder 2504.

[0798] Video decoder 2504 may generate one or more decoded pictures based on coded enhancement layer video bitstream 530 and coded base layer video bitstream 532. Decoded picture 2118 (which may include decoded enhancement layer picture 538 and decoded base layer picture 540) may be displayed, reproduced, stored in memory and/or transmitted to another device, etc.

[0799] In one example, the decoded image 2118 may be transmitted to another device or back to the first electronic device 502a. The decoded image 2118 may also be stored or otherwise maintained on the second electronic device 502b. In another example, the second electronic device 502b may display the decoded image 2118. In other configurations, the decoded image 2118 includes elements of the input image 2506 with different characteristics based on the encoding and other operations performed on the bitstream 2114. In some configurations, the decoded image 2118 may be included in an image stream with a different resolution, format, specifications, or other attribute from the input image 2506.

[0800] Figure 67B is a block diagram illustrating the use of a shared DPB 516c and a shared DPB hybrid operation module 520c for a base layer and an enhancement layer. Figure 67B includes the same components as those of Figure 67A, except that the enhancement layer video decoder 534 and the base layer decoder 536 share both the DPB 516c and the DPB hybrid operation module 520c.

[0801] Figure 68 is a timing diagram illustrating a hybrid operation of a decoded picture buffer (DPB) 116. The hybrid operation of the decoded picture buffer (DPB) 116 of Figure 68 shows the steps for a preferred embodiment where the removal, storage, and marking of references are based on a picture, and the output, output marking, and additional output are based on an access unit (AU). An ideal DPB 116 and a hypothetical reference decoder (HRD) may operate such that the various individual steps shown (e.g., removal 621a-b, output 623, storage 625a-c, marking 627a-c, 629, and additional output 631) are all performed instantaneously. The sequence of these steps and the timing offset between the individual steps is illustrated for illustrative purposes. For the output synchronization decoder to match, the timing (relative to the first bit delivery time) of the image output is the same for both the hypothetical reference decoder (HRD) and the decoder under test (DUT) within a fixed delay. As such, the illustrated timing offset can occur for a DUT that introduces a fixed delay compared to the HRD.

[0802] Steps for the decoded picture buffer (DPB) 116 for the first enhancement layer (EL1) 2642a, the decoded picture buffer (DPB) 116 for the second enhancement layer (EL2) 2642b, and the decoded picture buffer (DPB) 116 for the base layer (BL) 2644 are illustrated. The timing of the coded picture buffer (CPB) removal 2646 is illustrated. After the coded picture buffer (CPB) removal time 2646, the picture-based removal 621a (without output) can be performed by the decoded picture buffer (DPB) 116 for the second enhancement layer (EL2) 2642b and the picture-based removal (without output) 621b can be performed by the decoded picture buffer (DPB) 116 for the base layer (BL) 2644. After the time shift, the output (flush) 623 of the picture based on the access unit (AU) can be performed by the decoded picture buffer (DPB) 116 for the base layer (BL) 2644, the decoded picture buffer (DPB) 116 for the first enhancement layer (EL1) 2642a and the decoded picture buffer (DPB) 116 for the second enhancement layer (EL2) 2642b.

[0803] A process 613 associated with a current decoded picture after another time offset is illustrated. The step 625a-c of storing based on the picture may be performed by the decoded picture buffer (DPB) 116 of the base layer (BL) 2644, the decoded picture buffer (DPB) 116 for the first enhancement layer (EL1) 2642a, and the decoded picture buffer (DPB) 116 of the second enhancement layer (EL2) 2642b. The storing step may be performed after each of the base layer (BL) 2644, the first enhancement layer (EL1) 2642a, and the second enhancement layer (EL2) 2642b are decoded. The storing step 625 may be further subdivided. In the storing step 625, the decoded picture is stored in the decoded picture buffer (DPB) 116 in an empty storage buffer, and the occupancy of the decoded picture buffer (DPB) is increased by one. In addition, when the picture is removed (without being output) from the decoded picture buffer (DPB) 116, the occupancy of the decoded picture buffer (DPB) is decreased by one. In addition, when the picture is removed from the decoded picture buffer (DPB) 116 (either during flushing or during additional flushing), the occupancy of the decoded picture buffer (DPB) is decreased by one.

[0804] The decoded picture buffer (DPB) 116 may include separately identifiable and managed picture buffers for decoded pictures with different characteristics. For example, the decoded picture buffer (DPB) 116 may include separately identifiable and managed picture buffers for decoded pictures with different resolutions, different bit depths - and/or different color chromaticities.

[0805] The decoded picture may instead be stored in a common pool of picture storage buffers in the decoded picture buffer (DPB) 116. For example, two additional special cases may be used to define limitations on the size of the decoded picture buffer (DPB) 116 that affect the process of discarding/discarding and determining the level. In a limitation on a byte-oriented decoded picture buffer (DPB) 116, the decoded picture may be stored taking into account the size based on the resolution and/or the bit depth. Limitations on the size of the decoded picture buffer (DPB) 116 may be specified as a boundary value in bytes that considers the resolution and bit depth of each decoded picture. In a limitation on the decoded picture buffer (DPB) 116 based on a picture block, the decoded picture may be stored (and is considered to occupy one slot of the picture buffer). The limits on the size of the decoded picture buffer (DPB) 116 can then be defined as some bound on the number of picture slots without considering the resolution and color depth of each decoded picture.

[0806] In one configuration, the DPB fullness may be tracked on a per-layer basis. For example, DPB size limits 116 may be signaled, and flushing may be applied, per layer. Where each layer includes its own picture storage buffers, the DPBFullness[nuh_layer_id] variable may be used to track the DPB fullness for each layer. When a picture is removed from a layer with a layer identifier value equal to nuh_layer_id, the DPBFullness[nuh_layer_id] variable may be set to DPBFullness[nuh_layer_id] - 1 (that is, DPBFullness[nuh_layer_id] may be decremented by one). In this case, the picture has been removed from the PSB[nuh_layer_id] picture storage buffer.

[0807] Similarly, when the current decoded picture with a layer ID value equal to nuh_layer_id is stored in the decoded picture buffer (DPB) 116, the variable DPBFullness[nuh_layer_id] is set to DPBFullness[nuh_layer_id]+l (that is, DPBFullness[nuh_layer_id] is incremented by one). In this case, the picture has been stored in the picture storage buffer PSB[nuh_layer_id].

[0808] The DPB fullness may also be monitored for a set of output levels. Limitations on the size of the DPB 116 may be signaled, and a reset may be applied based on the limits defined for the set of output levels. The DPBFullness value may be monitored for the set of output levels that is associated with the operating point being tested. Thus, when a picture is removed from a level belonging to the set of output levels, the DPB fullness value may be decreased by one as DPBFullness=DPBFullness-1. Similarly, when the current decoded picture is stored in the DPB 116, the DPB fullness may be increased (decreased?) by one as DPBFullness=DPBFullness+1.

[0809] Within the process 613 related to the current decoded picture, a marking process 611 is illustrated. The marking process 611 may include a link marking 627a-c, and an output marking 629. After a temporary offset from the storage steps 625a-c, the step 627a-c of marking the link based on the picture may be performed by means of the decoded picture buffer (DPB) 116 of the base layer (BL) 2644, the decoded picture buffer (DPB) 116 of the first enhancement layer (EL1) 2642a, and the decoded picture buffer (DPB) 116 of the second enhancement layer (EL2) 2642b. After another time shift, the step 629 of marking the output based on the access unit (AU) can be performed by the decoded picture buffer (DPB) 116 of the base layer (BL) 2644, the decoded picture buffer (DPB) 116 of the first enhancement layer (EL1) 2642a and the decoded picture buffer (DPB) 116 of the second enhancement layer (EL2) 2642b. Once the process 613 related to the current decoded picture is completed, the step 631 of outputting the picture (additional flushing) based on the access unit (AU) can be performed by the decoded picture buffer (DPB) 116 of the base layer (BL) 2644, the decoded picture buffer (DPB) 116 of the first enhancement layer (EL1) 2642a and the decoded picture buffer (DPB) 116 of the second enhancement layer 2642b.

[0810] Figure 69 is a block diagram illustrating the structure and timing for network abstraction layer (NAL) units for coded picture layers and access units (AUs) when the second enhancement layer (EL2) 942b has a picture rate lower than the base layer (BL) 944 and the first enhancement layer (EL1) 942a. The NAL units of the coded picture 953a EL1 are illustrated along the line of the first enhancement layer (EL1) 942a. The NAL units of the coded picture 953b EL2 are illustrated along the line of the second enhancement layer (EL2) 942b. The NAL units of the coded picture 953c of the base layer are illustrated along the line of the base layer (BL) 944.

[0811] At time t1, the NAL units of the coded picture 953a EL1, the NAL units of the coded picture 953b EL2, and the NAL units of the coded picture 953c of the base layer are part of the access unit (AU) 955a. At time t2, the NAL units of the coded picture 953a EL1 and the NAL units of the coded picture 953c of the base layer are part of the access unit (AU) 955b. At time t3, the NAL units of the coded picture 953a EL1, the NAL units of the coded picture 953b EL2, and the NAL units of the coded picture 953c of the base layer are part of the access unit (AU) 955a. At time t4, the NAL units of the coded picture 953a EL1 and the NAL units of the coded picture 953c of the base layer are part of the access unit (AU) 955d.

[0812] Figure 70 is a block diagram illustrating the structure and timing for network abstraction layer (NAL) units for coded picture layers and access units (AUs) when the base layer (BL) 1044 has a picture rate lower than the first enhancement layer (EL1) 1042a and the second enhancement layer (EL2) 1042b. The NAL units of the coded picture 1053a EL1 are illustrated along the line of the first enhancement layer (EL1) 1042a. The NAL units of the coded picture 1053b EL2 are illustrated along the line of the second enhancement layer (EL2) 1042b. The NAL units of the coded picture 1053c of the base layer are illustrated along the line of the base layer (BL) 1044.

[0813] At time t1, the NAL units of the coded picture 1053a EL1, the NAL units of the coded picture 1053b EL2, and the NAL units of the coded picture 1053c of the base layer are part of the access unit (AU) 1055a. At time t2, the NAL units of the coded picture 1053a EL1 and the NAL units of the coded picture 1053b EL2 are part of the access unit (AU) 1055b. At time t3, the NAL units of the coded picture 1053a EL1, the NAL units of the coded picture 1053b EL2, and the NAL units of the coded picture 1053c of the base layer are part of the access unit (AU) 1055c. At time t4, the NAL units of the coded EL1 picture 1053a and the NAL units of the coded EL1 picture 1053b are part of the access unit (AU) 1055d.

[0814] The term "computer-readable medium" refers to any accessible medium that can be accessed by a computer or a processor. The term "computer-readable medium" as used herein may mean a computer- and/or processor-readable medium that is non-transitory and tangible. By way of example, and not limitation, computer-readable or processor-readable medium may comprise random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), compact disc ROM (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer or a processor. Magnetic disk and non-magnetic disk as used in the description include compact disc (CD), laser disc, optical disc, digital multi-function disc (DVD), floppy disk and Blu-ray disc (registered trademark), where magnetic disks generally reproduce data magnetically while non-magnetic disks reproduce data optically using lasers.

[0815] It should be noted that one or more of the methods described herein may be implemented in and/or implemented using hardware. For example, one or more of the methods or approaches described herein may be implemented in and/or implemented using a microprocessor set, an application-specific integrated circuit (ASIC), a large-scale integrated circuit (LSI), or an integrated circuit, etc.

[0816] Each of the methods disclosed herein comprises one or more steps or actions for carrying out the described method. The method steps and/or actions may be interchangeable with one another and/or combined into a single step, without departing from the scope of the claims. In other words, if a particular order of steps or actions is not required for the proper operation of the described method, the order and/or use of particular steps and/or actions may be modified, without departing from the scope of the claims.

[0817] It should be understood that the claims are not limited to the precise configuration and components illustrated above. Various modifications, changes, and variations may be made in the arrangement, operation, and details of the systems, methods, and apparatus described herein without departing from the scope of the claims.

Claims (18)

1. A method for decoding a video bitstream, comprising: obtaining a set of image synchronization parameters including a coded image buffer deletion delay parameter for deleting an access block from the coded image buffer at the access block level, wherein each access block comprises a plurality of decoding blocks; obtaining data representing the encoded video image; storing data in the encoded image buffer; receiving a first flag indicating whether it is necessary to remove an access block from the coded picture buffer for decoding at the access block level or whether it is necessary to remove a decoding block from the coded picture buffer for decoding at the image fragment level, wherein the decoding block is a subset of the access block; determining, based on the first flag, that the access block or the decoding block is to be removed from the coded picture buffer; and in response to determining that a decoding block is to be removed from the coded picture buffer for decoding at the image fragment level, (i) receiving a second flag indicating whether the video bitstream contains a parameter representing a common ablation delay for all decoding blocks in the access block, wherein the common ablation delay is the delay between the current decoding block and the immediately preceding decoding block; (ii) in response to determining that the flag indicates that a parameter representing a common erasure delay for all decoding blocks is contained in the bitstream, obtaining a parameter representing the common erasure delay; (iii) in response to determining that the flag indicates that a parameter representing a common ablation delay for all decoding units is not contained in the bitstream, obtaining individual ablation delay parameters for the decoding units, each of which represents a delay between successive decoding units in the access unit, wherein at least two of the individual ablation delay parameters for the decoding units are different, (iv) removing a decoding block from the coded picture buffer based on either a parameter representing an overall removal delay or individual removal delay parameters for the decoding blocks; (v) decoding of decoding blocks; (vi) extracting an image based on the decoded decoding blocks; and in response to determining that the access unit is to be removed from the coded picture buffer for decoding at the access unit level, (i) determining the deletion delay for an access block for an access block in the data, (ii) removing the access block from the coded picture buffer based on the removal delay for the access block, and (iii) entropy decoding of the remote access block to generate a set of quantized transformed coefficients. 2. The method of claim 1, further comprising receiving a set of image synchronization parameters in a supplemental extended information (SEI) message, wherein the set of image synchronization parameters includes a parameter representing a total deletion delay.