KR102688478B1 - Layered coding and data structure for compressed higher-order ambisonics sound or sound field representations - Google Patents

Abstract

This document relates to a method for layered encoding of frames of compressed higher-order ambisonics (HOA) representations of sounds or sound fields. The compressed HOA representation includes a plurality of transmitted signals. The method includes assigning a plurality of transmission signals to a plurality of hierarchical layers, the plurality of layers including a base layer and one or more hierarchical enhancement layers, for each layer, a respective layer and a respective layer. Generating respective HOA extension payloads containing auxiliary information for parametrically improving the reconstructed HOA representation obtainable from transmission signals assigned to lower arbitrary layers, the generated HOA extension payloads It includes assigning to the respective layers, and signaling the generated HOA extension payloads in the output bitstream. This document further describes a method for decoding a frame of a compressed HOA representation of a sound or sound field, an encoder and decoder for layered coding of a compressed HOA representation, and a data structure representing a frame of a compressed HOA representation of a sound or sound field. It's about.

Description

LAYERED CODING AND DATA STRUCTURE FOR COMPRESSED HIGHER-ORDER AMBISONICS SOUND OR SOUND FIELD REPRESENTATIONS}

Cross-reference to related applications

This application claims priority to European Patent Application No. 15306653.5, filed October 15, 2015, which European Patent Application is hereby incorporated by reference in its entirety.

Technology field

This document relates to methods and devices for layered audio coding. Specifically, this document relates to methods and apparatus for layered audio coding of frames of compressed Higher-Order Ambisonics (HOA) sound (or sound field) representations. This document also relates to data structures (e.g., bitstreams) for representing frames of compressed HOA sound (or sound field) representations.

In the current definition of HOA layered coding, auxiliary information about HOA decoding tools such as spatial signal prediction, subband directional signal synthesis, and Parametric Ambience Replication (PAR) decoder is provided to improve specific HOA representation. is created. That is, in the current definition of layered HOA coding, the provided data only appropriately extends the HOA representation of the top layer (e.g., top enhancement layer). For lower layers, including the base layer, these tools do not adequately enhance the partially reconstructed HOA representation.

Tools such as subband directional signal synthesis and parametric ambience replica decoder are specifically designed for low data rates, where only a few transmitted signals are available. However, in HOA layered coding, adequate enhancement of (partially) reconstructed HOA representations is not possible, especially for low bitrate layers, such as the base layer. This is clearly undesirable from the point of view of sound quality at low bitrates.

In addition, the conventional way of processing encoded V-vector elements for vector-based signals is to ensure proper decoding if CodedVVecLength equal to 1 is signaled in HOADecoderConfig() (i.e. vector coding mode is active). It turns out that it doesn't bring . In this vector coding mode, V-vector elements for HOA coefficient indices included in the ContAddHoaCoeff set are not transmitted. This set contains all HOA coefficient indices AmbCoeffIdx[i] with AmbCoeffTransitionState equal to 0. Conventionally, there is no need to add a weighted V-vector signal because the original HOA coefficient sequence for these indices is explicitly transmitted (signaled). Therefore, for these indices the V-vector elements are set to 0.

However, in layered coding mode, the set of consecutive HOA coefficient indices depends on the transport channels that are currently part of the active layer. Additional HOA coefficient indices transmitted in upper layers may be missing in lower layers. Then, for HOA coefficient indices belonging to HOA coefficient sequences included in upper layers, the assumption that the vector signal should not contribute to the HOA coefficient sequence is incorrect.

As a result, the V-vector in layered HOA coding may not be suitable for decoding any layers below the top layer.

Accordingly, there is a need for coding schemes and bitstreams adapted for layered coding of compressed HOA representations of sound or sound fields.

This document addresses the above issues. In detail, methods and encoders/decoders for layered coding of frames of compressed HOA sound or sound field representations as well as data structures for representing frames of compressed HOA sound or sound field representations are described.

According to one aspect, a method for layered encoding of frames of a compressed higher-order ambisonics (HOA) representation of a sound or sound field is described. The compressed HOA representation complies with the draft MPEG-H 3D audio standard and any other future adopted or draft standards. The compressed HOA representation may include a plurality of transmitted signals. The transmission signals may relate to monaural signals, for example, representing either the dominant sound signals or the coefficient sequences of the HOA representation. The method may include assigning a plurality of transmission signals to a plurality of hierarchical layers. For example, transmission signals may be distributed to multiple layers. The plurality of layers may include a base layer and one or more hierarchical enhancement layers. The plurality of hierarchical layers may be ordered from the base layer, through the first enhancement layer, the second enhancement layer, etc., to the overall highest enhancement layer (overall highest layer). The method provides, for each layer, auxiliary information (e.g., enhancement) to parametrically improve the reconstructed HOA representation obtainable from transmission signals assigned to each layer and any layers lower than each layer. A step of generating each HOA extension payload containing auxiliary information may be additionally included. Reconstructed HOA representations for lower layers may be referred to as partially reconstructed HOA representations. The method may additionally include the step of assigning the generated HOA extension payloads to their respective layers. The method may also further include signaling the generated HOA extension payloads in the output bitstream. HOA extension payloads can be signaled in the HOAEnhFrame() payload. Accordingly, auxiliary information can be moved from HOAFrame() to HOAEnhFrame().

When configured as above, the proposed method applies layered coding to (frames of) compressed HOA representations to enable high-quality decoding even at low bitrates. In detail, the proposed method allows each layer to load a suitable HOA extension payload (e.g., enhancement assistance) to enhance the (partially) reconstructed sound representation obtained from the transmitted signals in any layers up to the current layer. information) is included. There, the layers up to the current layer are understood to include, for example, a base layer, a first enhancement layer, a second enhancement layer, etc., up to the current layer. There, the layers up to the current layer are understood to include, for example, a base layer, a first enhancement layer, a second enhancement layer, etc., up to the current layer. For example, the decoder may refer to the HOA extension payload assigned to the base layer to enhance the (partially) reconstructed sound representation obtained from the base layer. In conventional approaches, only the reconstructed HOA representation of the top enhancement layer would be enhanced by the HOA extension payload. Therefore, regardless of the actual highest available layer (e.g., the layer below the lowest available layer that has not been validly received, such that all of the layers below the highest available layer and the highest available layer itself are received validly), the decoder must: Although the (partially) reconstructed sound representation may differ from the complete (e.g., full) sound representation, it may be possible to improve or enhance the reconstructed sound representation. Specifically, regardless of the actual highest available layer, the decoder improves or enhances the (partially) reconstructed sound representation obtainable based on all of the transmitted signals contained in the layers up to the actual highest available layer. To achieve this, it is sufficient to decode the HOA extension payload for a single layer only (i.e. for the highest available layer). It is not required to decode HOA extension payloads of upper or lower layers. On the other hand, the proposed method makes it possible to fully exploit the reduction in required bandwidth that can be achieved when applying layered coding.

In embodiments, the method may further include transmitting data payloads for multiple layers with respective error protection levels. Data payloads may include respective HOA extension payloads. The base layer may have the highest error tolerance and one or more enhancement layers may have sequentially decreasing error tolerance. Thereby, it can be ensured that at least a number of lower layers are transmitted reliably, while on the other hand reducing the overall required bandwidth by not applying excessive error prevention to the upper layers.

In embodiments, HOA extension payloads may include bitstream elements for an HOA spatial signal prediction decoding tool. Additionally or alternatively, HOA extension payloads may include bitstream elements for the HOA subband directional signal synthesis decoding tool. Additionally or alternatively, HOA extension payloads may include bitstream elements for the HOA parametric ambience replica decoding tool.

In embodiments, HOA extension payloads may have a usacExtElementType of ID_EXT_ELE_HOA_ENH_LAYER.

In embodiments, the method includes an HOA configuration extension payload comprising bitstream elements for configuring an HOA spatial signal prediction decoding tool, an HOA subband directional signal synthesis decoding tool, and/or an HOA parametric ambience replica decoding tool. An additional step of generating may be included. HOA configuration extension payload can be included in HOADecoderEnhConfig(). The method may further include signaling the HOA configuration extension payload in the output bitstream.

In embodiments, the method may further include generating an HOA decoder configuration payload that includes information indicating assignment of HOA extension payloads to a plurality of layers. The method may further include signaling the HOA decoder configuration payload in the output bitstream.

In embodiments, the method may further include determining whether the vector coding mode is active. The method may further include, when the vector coding mode is active, determining, for each layer, a set of consecutive HOA coefficient indices based on the transmission signals assigned to the respective layer. HOA coefficient indexes in the set of consecutive HOA coefficient indexes may be HOA coefficient indexes included in the ContAddHOACoeff set. This method is such that for each transmission signal, each transmission signal includes elements for arbitrary transmission signals assigned to layers higher than the layer to which each transmission signal is assigned. It may further include generating a V-vector based on the determined set of consecutive HOA coefficient indices for the assigned layer. The method may further include signaling the generated V-vector in the output bitstream.

According to another aspect, a method for layered encoding of frames of a compressed higher-order ambisonics (HOA) representation of a sound or sound field is described. The compressed HOA representation may include a plurality of transmitted signals. The transmission signals may relate to monaural signals, for example, representing either the dominant sound signals or the coefficient sequences of the HOA representation. The method may include assigning a plurality of transmission signals to a plurality of hierarchical layers. For example, transmission signals may be distributed to multiple layers. The plurality of layers may include a base layer and one or more hierarchical enhancement layers. The method may further include determining whether the vector coding mode is active. The method may further include, when the vector coding mode is active, determining, for each layer, a set of consecutive HOA coefficient indices based on the transmission signals assigned to the respective layer. HOA coefficient indexes in the set of consecutive HOA coefficient indexes may be HOA coefficient indexes included in the ContAddHOACoeff set. This method is such that for each transmission signal, each transmission signal includes elements for arbitrary transmission signals assigned to layers higher than the layer to which each transmission signal is assigned. It may further include generating a V-vector based on the determined set of consecutive HOA coefficient indices for the assigned layer. The method may further include signaling the generated V-vector in the output bitstream.

When configured as such, the proposed method ensures that an appropriate V-vector is available for all transmitted signals belonging to layers up to the highest available layer in vector coding mode. In detail, the proposed method excludes cases where elements of the V-vector corresponding to transmission signals in higher layers are not explicitly signaled. Accordingly, the information contained in the layers up to the highest available layer is sufficient to decode any transmission signals belonging to the layers up to the highest available layer. Thereby, there is proper decompression of the respective reconstructed HOA representations for lower layers (low bitrate layers) even if the upper layers may not have been validly received by the decoder. On the other hand, the proposed method makes it possible to fully exploit the reduction in required bandwidth that can be achieved when applying layered coding.

According to another aspect, a method for decoding a frame of a compressed higher-order ambisonics (HOA) representation of a sound or sound field is described. The compressed HOA representation may be encoded in multiple hierarchical layers. The plurality of hierarchical layers may include a base layer and one or more hierarchical enhancement layers. The method may include receiving a bitstream associated with a frame of the compressed HOA representation. The method may further include extracting payloads for a plurality of layers. Each payload may include transmission signals assigned to each layer. The method may further include determining the highest available layer among the plurality of layers for decoding. The method may further include extracting the HOA extension payload assigned to the highest available layer. This HOA extension payload may contain auxiliary information to parametrically enhance the (partially) reconstructed HOA representation corresponding to the highest available layer. A (partially) reconstructed HOA representation corresponding to the highest available layer may be obtainable based on transmission signals assigned to the highest available layer and any layers lower than the highest available layer. The method further includes generating a (partially) reconstructed HOA representation corresponding to the highest available layer based on the transmission signals assigned to the highest available layer and any layers lower than the highest available layer. It can be included. The method may also further include enhancing (e.g., parametrically enhancing) the (partially) reconstructed HOA representation using auxiliary information contained in the HOA extension payload assigned to the highest available layer. You can. As a result, an improved reconstructed HOA representation can be obtained.

When configured as such, the proposed method uses the available (e.g., validly received) information to the greatest extent possible to ensure that the final (e.g., improved) reconstructed HOA representation is of optimal quality.

In embodiments, HOA extension payloads may include bitstream elements for an HOA spatial signal prediction decoding tool. Additionally or alternatively, HOA extension payloads may include bitstream elements for the HOA subband directional signal synthesis decoding tool. Additionally or alternatively, HOA extension payloads may include bitstream elements for the HOA parametric ambience replica decoding tool.

In embodiments, HOA extension payloads may have a usacExtElementType of ID_EXT_ELE_HOA_ENH_LAYER.

In embodiments, the method may further include extracting the HOA configuration extension payload by parsing the bitstream. The HOA configuration extension payload may include bitstream elements for configuring the HOA spatial signal prediction decoding tool, the HOA subband directional signal synthesis decoding tool, and/or the HOA parametric ambience replica decoding tool.

In embodiments, the method may further include extracting HOA extension payloads respectively assigned to a plurality of layers. Each HOA extension payload may contain auxiliary information to parametrically enhance the (partially) reconstructed HOA representation corresponding to its respective assigned layer. A (partially) reconstructed HOA representation corresponding to its respective assigned layer may be obtainable from transmission signals assigned to that layer and any layers lower than that layer. Assignment of HOA extension payloads to respective layers can be known from configuration information included in the bitstream.

In embodiments, determining the highest available layer may include determining a set of invalid layer indices that indicate layers that were not validly received. This may further include determining the highest available layer as a layer that is one layer below the layer indicated by the smallest (lowest) index in the set of invalid layer indices. The base layer may have the lowest layer index (eg, a layer index of 1), and hierarchical enhancement layers may have sequentially higher layer indices. Thereby, the proposed method allows the highest available layer to be configured in such a way that all of the information required to decode the (partially) reconstructed HOA representation from the top available layers and any layers below the top available layer is available. Ensure that you are selected.

In embodiments, determining the highest available layer may include determining a set of invalid layer indices that indicate layers that were not validly received. This may further include determining the highest available layer of the previous frame preceding the current frame. This further includes determining the highest available layer as the highest available layer of the previous frame and the lower layer among the layers that is one layer below the layer indicated by the lowest index in the set of invalid layer indices. can do. Thereby, even if the current frame was differentially encoded with respect to the preceding frame, the highest available layer for the current frame is a (partially) reconstructed HOA representation from the highest available layer and any layers below the highest available layer. is chosen in such a way that all of the information required to decode is available.

In embodiments, the method may determine the HOA assigned to the highest available layer if the highest available layer of the current frame is lower than the highest available layer of the previous frame and if the current frame was differentially coded with respect to the previous frame. It may further include deciding not to perform parametric enhancement of the (partially) reconstructed HOA representation using auxiliary information included in the extension payload. Thereby, if the current frame (including auxiliary information contained in the HOA extension payload assigned to the highest available layer) has been differentially encoded with respect to the preceding frame, the reconstructed HOA representation can be decoded without error. .

In embodiments, the set of invalid layer indices may be determined by evaluating validity flags of corresponding HOA extension payloads. If the validity flag for the HOA extension payload assigned to the respective layer is not set, the layer index of the given layer may be added to the set of invalid layer indexes. Thereby, the set of invalid layer indices can be determined in an efficient manner.

According to another aspect, a data structure (e.g., a bitstream) representing a frame of a compressed higher order ambisonics (HOA) representation of a sound or sound field is described. The compressed HOA representation may include a plurality of transmitted signals. The data structure may include a plurality of HOA frame payloads corresponding to respective layers of the plurality of hierarchical layers. HOA frame payloads may include respective transmission signals. A plurality of transmission signals may be assigned (eg, distributed) to a plurality of layers. The plurality of layers may include a base layer and one or more hierarchical enhancement layers. The data structure contains, for each layer, auxiliary information to parametrically improve the (partially) reconstructed HOA representation obtainable from the transmission signals assigned to the respective layer and any layers lower than the respective layer. Each HOA extension payload containing may be additionally included.

In embodiments, HOA frame payloads and HOA extension payloads for multiple layers may be provided with respective error prevention levels. The base layer may have the highest error tolerance and one or more enhancement layers may have sequentially decreasing error tolerance.

In embodiments, HOA extension payloads may include bitstream elements for an HOA spatial signal prediction decoding tool. Additionally or alternatively, HOA extension payloads may include bitstream elements for the HOA subband directional signal synthesis decoding tool. Additionally or alternatively, HOA extension payloads may include bitstream elements for the HOA parametric ambience replica decoding tool.

In embodiments, HOA extension payloads may have a usacExtElementType of ID_EXT_ELE_HOA_ENH_LAYER.

In embodiments, the data structure includes an HOA configuration extension payload containing bitstream elements for constructing an HOA spatial signal prediction decoding tool, an HOA subband directional signal synthesis decoding tool, and/or an HOA parametric ambience replica decoding tool. Additional information may be included.

In embodiments, the data structure may further include an HOA decoder configuration payload that includes information indicating assignment of HOA extension payloads to a plurality of layers.

In embodiments, methods and apparatuses relate to decoding a compressed higher order ambisonics (HOA) representation of a sound or sound field. The device receives a bitstream comprising a compressed HOA representation corresponding to a plurality of hierarchical layers including a base layer and one or more hierarchical enhancement layers, wherein the plurality of layers are a basic compressed sound representation of the sound or sound field. components are assigned, and the components are assigned to respective layers within the respective component groups - determine the highest available layer among the plurality of layers for decoding; Extracts an HOA extension payload assigned to the highest available layer, wherein the HOA extension payload contains auxiliary information to parametrically enhance the reconstructed HOA representation corresponding to the highest available layer, and A reconstructed HOA representation is obtainable based on transmission signals assigned to the highest available layer and any layers lower than the highest available layer; Decode the compressed HOA representation corresponding to the highest available layer based on layer information, where transmission signals are assigned to the highest available layer and any layers lower than the highest available layer; and parametrically enhance the decoded HOA representation using auxiliary information contained in the HOA extension payload assigned to the highest available layer.

The HOA extension payload may include bitstream elements for the HOA spatial signal prediction decoding tool. The layer information may indicate the number of active directional signals within the current frame of the enhancement layer.

The layer information may indicate the total number of additional surrounding HOA coefficients for the enhancement layer. The layer information may include HOA coefficient indices for each additional surrounding HOA coefficient for the enhancement layer. The layer information may include enhancement information including at least one of spatial signal prediction, subband directional signal synthesis, and parametric ambience replication decoder. The compressed HOA representation is adapted to the layered coding mode for HOA-based content if CodedVVecLength equal to 1 is signaled in HOADecoderConfig(). Additionally, V-vector elements may not be transmitted for indices that are the same as the indices of the additional HOA coefficients included in the ContAddHoaCoeff set. A ContAddHoaCoeff set may be defined individually for each of the plurality of hierarchical layers. The layer information includes NumLayers elements, where each element indicates the number of transmission signals included in all layers up to the i-th layer. The layer information may include an indicator of all layers actually used for the k-th frame. The layer information may also indicate that all coefficients for the dominant vectors are specified. The layer information may indicate that coefficients of dominance vectors corresponding to numbers greater than MinNumOfCoeffsForAmbHOA are specified. The layer information may indicate that all of the elements defined in ContAddHoaCoeff[lay] and MinNumOfCoeffsForAmbHOA are not transmitted, where lay is the index of the layer containing the vector-based signal corresponding to the vector.

According to another aspect, an encoder for layered encoding of frames of compressed higher-order ambisonics (HOA) representations of sounds or sound fields is described. The compressed HOA representation may include a plurality of transmitted signals. The encoder may comprise a processor configured to perform some or all of the method steps of the methods according to the above-mentioned first aspect and the above-mentioned second aspect.

According to another aspect, a decoder is described for decoding frames of a compressed higher-order ambisonics (HOA) representation of a sound or sound field. The compressed HOA representation may be encoded in a plurality of hierarchical layers, including a base layer and one or more hierarchical enhancement layers. The decoder may comprise a processor configured to perform some or all of the method steps of the methods according to the third aspect mentioned above.

According to another aspect, a software program is described. A software program may be adapted to run on a processor and to perform some or all of the method steps outlined herein when executed on a computing device.

According to another aspect, a storage medium is described. The storage medium may include a software program adapted to run on a processor and to perform some or all of the method steps outlined herein when executed on a computing device.

It will be appreciated that statements made regarding any of the above aspects or embodiments thereof also apply to the respective other aspects or embodiments thereof, as those skilled in the art will recognize. Repeating these statements for every aspect or embodiment has been omitted for the sake of brevity.

It should be noted that the methods and devices may be used alone or in combination with other methods and systems disclosed herein, including their preferred embodiments outlined herein. Moreover, all aspects of the methods and devices outlined in this document may be arbitrarily combined. In detail, features of the claims may be combined with each other in an arbitrary manner.

It should be further noted that method steps and device features can be interchanged in many ways. In particular, as will be appreciated by those skilled in the art, details of the disclosed method may be implemented as an apparatus adapted to perform some or all of the steps of the method and vice versa.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is described below by way of example with reference to the accompanying drawings.
1 is a block diagram schematically illustrating the assignment of payloads to the base layer and M-1 enhancement layers at the encoder side;
Figure 2 is a block diagram schematically illustrating an example of a receiver and decompression stage;
3 is a flowchart illustrating an example of a layered encoding method of a frame of a compressed HOA representation according to embodiments of the present disclosure;
4 is a flowchart illustrating another example of a layered encoding method of a frame of a compressed HOA representation according to embodiments of the present disclosure;
5 is a flowchart illustrating an example of a method for decoding a frame of a compressed HOA representation according to embodiments of the present disclosure;
6 is a block diagram schematically illustrating an example of a hardware implementation of an encoder according to embodiments of the present disclosure;
7 is a block diagram schematically illustrating an example hardware implementation of a decoder according to embodiments of the present disclosure.

First, a compressed sound (or sound field) representation to which methods and encoders/decoders according to the present disclosure may be applicable will be described.

To stream a compressed sound (or sound field) representation over a transmission channel with time-varying conditions, layered coding is used to adapt the quality of the received sound representation to the transmission conditions, and specifically to eliminate unwanted signal dropout ( It is a means to avoid signal dropouts.

In the case of layered coding, the compressed sound (or sound field) representation usually consists of a high-priority base layer of relatively small size and an additional enhancement layer with decreasing priorities and arbitrary sizes. It is subdivided into layers. Each enhancement layer is typically assumed to contain incremental information that complements the information of all lower layers to improve the quality of the compressed sound (or sound field) representation. The idea is to then control the amount of error prevention to transmit individual layers according to their priorities. Specifically, the base layer is provided with high error tolerance, which is reasonable and acceptable due to the small size of the base layer.

In the following, it is assumed that a complete compressed sound (or sound field) representation generally consists of the following three components:

1. A basic compressed sound (or sound field) representation consisting of a number of complementary components, which make up significantly the largest proportion of the complete compressed sound (or sound field) representation.

2. Basic auxiliary information required to decode the underlying compressed sound representation, which is assumed to be of much smaller size compared to the underlying compressed sound (or sound field) representation. It is further assumed that it mostly consists of the following two components, both of which specify the decompression of only one specific component of the basic compressed sound representation.

a) The first component contains auxiliary information that describes individual complementary components of the basic compressed sound (or sound field) representation independently of other complementary components .

b) The second (optional) component contains auxiliary information that describes individual complementary components of the basic compressed sound (or sound field) representation in dependence on other complementary components . Specifically, a dependency has the following properties:

Dependent auxiliary information for each individual complementary component of the basic compressed sound (or sound field) representation is its largest size, if the basic compressed sound (or sound field) representation does not contain any other specific complementary components. (extent) can be achieved.

When additional specific complementary components are added to the basic compressed sound (or sound field) representation, the dependent auxiliary information for the individual complementary components considered becomes a subset of the original dependent auxiliary information, thereby reducing its size. reduce.

3. Arbitrary enhancement auxiliary information to improve the basic compressed sound (or sound field) representation. Its size is also assumed to be much smaller than that of the underlying compressed sound (or sound field) representation.

One prominent example of this type of fully compressed sound (or sound field) representation is given by the compressed HOA sound field representation as specified by the preliminary version of the MPEG-H 3D audio standard.

1. Its basic compressed sound field representation can be identified with a number of quantized monaural signals, representing either the so-called dominant sound signals or coefficient sequences of the so-called ambient HOA sound field components.

2. Basic auxiliary information describes, inter alia, for each of these monaural signals how the monaural signal contributes spatially to the sound field. This information can be further separated into two different components:

(a) Auxiliary information related to specific individual monaural signals, independent of the presence of other monaural signals. This auxiliary information may specify, for example, a monaural signal representing a directional signal (meaning a normal plane wave) with a specific direction of incidence. Alternatively, the monaural signal can be specified as a sequence of coefficients of the original HOA representation with specific indices.

(b) Auxiliary information related to specific individual monaural signals, dependent on the presence of other monaural signals. This auxiliary information is generated, for example, if the monaural signals are specified as so-called vector-based signals, meaning that they are directionally distributed in the sound field, where the directional distribution is specified by a vector. In certain modes (i.e. CodedVVecLength = 1), certain components of this vector are implicitly set to 0 and are not part of the compressed vector representation. These components are components with indices identical to those of the coefficient sequences of the original HOA representation and are part of the basic compressed sound field representation. This means that when the individual components of the vector are coded, their total depends on the basic compressed sound field representation, and in particular on which coefficient sequences of the original HOA representation the basic compressed sound field representation contains.

If the coefficient sequences of the original HOA representation are not included in the basic compressed sound field representation, the dependent basic auxiliary information for each vector-based signal consists of all of the vector components and has their largest magnitude. When coefficient sequences of the original HOA representation with certain indices are added to the basic compressed sound field representation, the vector components with those indices are removed from the side information for each vector-based signal, thereby adding Reduce the size of dependent basic auxiliary information.

3. Enhancement auxiliary information consists of the following components:

Parameters involved in the so-called (wideband) spatial prediction, which predicts (linearly) the missing parts of the sound field from directional signals.

So far, compression tools allow the frequency dependent, parametric prediction of additional monaural signals to be spatially distributed to compensate for spatially incomplete or defective compressed HOA representations, the so-called subband directional signals. Parameters involved in synthesis and parametric ambience replication. The prediction is based on coefficient sequences of the basic compressed sound field representation. An important aspect is that the mentioned complementary contribution to the sound field is represented within the compressed HOA representation, not by additional quantized signals, but rather by additional auxiliary information of comparably much smaller size. Therefore, the two mentioned coding tools are particularly suitable for compression of HOA representations at low data rates.

A second example of a compressed representation of a monaural signal with the previously mentioned structure may consist of the following components:

1. Some coded spectral information for disjoint frequency bands up to a certain upper frequency, which can be considered a basic compressed representation.

2. Some basic auxiliary information specifying the coded spectral information (e.g. by number and width of coded frequency bands).

3. Consisting of parameters of so-called Spectral Band Replication (SBR), which describes how to parametrically reconstruct from the basic compressed representation the spectral information for upper frequency bands not considered in the basic compressed representation. Any enhancement auxiliary information.

Next, a method for layered coding of a fully compressed sound (or sound field) representation with the previously mentioned structure will be described.

Compression is said to be frame-based, in the sense that it provides compressed representations (e.g., in the form of data packets or equivalently frame payloads) for consecutive time intervals, e.g., equally sized time intervals. It is assumed. These data packets are assumed to contain a validity flag, a value indicating their size, as well as the actual compressed representation data. Throughout the following description, the focus will primarily be on the processing of single frames, and thus frame indices will be omitted.

The payload of each frame of the considered complete compressed sound (or sound field) representation 1100 is J data packets, denoted by BSRC _j , j = 1, ..., J - each representing the basic compressed sound (or It is assumed to include - one component (1110-1, ..., 1110-J) of the sound field) expression. In addition, the frame payload is assumed to contain a packet with independent basic auxiliary information 1120, denoted by BSI _I, which specifies certain components of the basic compressed sound representation (BSRC _j ) independently of other components. . Optionally, the frame payload is, in addition, a packet with dependent basic auxiliary information marked by BSI _D that specifies certain components of the basic compressed sound representation (BSRC _j ) depending on the other components. It is assumed. Information contained within two data packets (BSI _I and BSI _D ) may be arbitrarily grouped into a single data packet (BSI).

Ultimately, the frame payload contains an enhancement auxiliary information payload marked by ESI with a description of how to improve the sound (or sound field) reconstructed from the complete underlying compressed representation.

The described scheme for layered coding covers the steps required to enable both the receiver and decompression parts as well as the compression part, which includes packing of data packets for transmission. Each part will be described in detail below.

First, compression and packing for transmission will be described. In the case of layered coding (assuming a total of M layers, i.e. one base layer and M-1 enhancement layers), each component of the complete compressed sound (or sound field) representation 1100 is processed as follows: do:

The basic compressed sound (or sound field) representation is subdivided into parts that are assigned to individual layers. Without loss of generality, for J _m-1 ≤ j < J _m , _the grouping is M+1 numbers (J _m , m = 0, .., M, and J ₀ = 1 and J _M = J+1).

Due to its small size, it makes sense to assign complete primary auxiliary information to the base layer to avoid its unnecessary fragmentation. The independent basic auxiliary information (BSI _I ) remains unchanged for assignment, but on the one hand to enable correct decoding at the receiver and on the other hand to reduce the size of the dependent basic auxiliary information to be transmitted. The information must be handled especially for layered coding. It is proposed to decompose the dependent basic auxiliary information into M parts (1130-1, ..., 1130-M) denoted by BSI _D,m , m = 1,...,M, where each dependent If auxiliary information exists, the mth part includes dependent auxiliary information for each of the components (BSRC _j , J _m-1 ≤ j <J _m ) of the basic compressed sound expression assigned to the mth layer. If the respective dependent auxiliary information does not exist, BSI _D,m is assumed to be empty. The auxiliary information (BSI _D,m ) depends on all of the components (BSRC _j , 1 ≤ j < J _m ) included in all layers up to the mth layer.

In the case of layered coding, it is important to realize that additional enhancement auxiliary information must be computed for each layer, since it enhances the preliminarily decompressed sound (or sound field) - but this does not require decompression. This is because it is intended - to depend on the available layers. Accordingly, compression must provide M individual enhanced auxiliary information data packets 1140-1, ..., 1140-M, denoted by ESI _m , m = 1, ..., M, where the m The enhancement assistance information in the data packet ESI _m is calculated to improve the sound (or sound field) representation obtained, for example, from all of the data included in the base layer and enhancement layers with indices lower than m.

In summary, at the compression stage, a frame data packet must be provided, denoted by FRAME, with the following composition:

It is understood that the ordering of individual payloads with frame data packets is generally arbitrary.

The allocation of individual payloads to the base and enhancement layers, already described, is achieved by a so-called transport layers packer and is schematically illustrated in Figure 1.

Next, reception and decompression will be described. The corresponding receiver and decompression stage are illustrated in Figure 2.

First, received frame packets of the complete compressed sound (or sound field) representation, which are then passed to decompressor 2100.

To provide, individual layer packets (1200, 1300-1, ..., 1300-(M-1)) are multiplexed. If the transmission of an individual layer is error-free, it is assumed that at least the validity flag of the included enhancement auxiliary information payload is set to “true”. In case of errors due to the transmission of an individual layer, the validity flag in at least the enhancement auxiliary information payload in this layer is set to "false". Accordingly, the validity of a layer packet can be determined from the validity of the included enhancement auxiliary information payload.

In decompressor 2100, received frame packets are first demultiplexed. To this end, information about the size of each payload can be used to avoid unnecessary parsing of the data of the individual payloads.

In the next step, the number of the top layer (N _B ) that will actually be used for decompressing the basic sound representation is selected. The highest enhancement layer that will actually be used for decompressing the basic sound representation is given by N _B - 1. Since each layer includes exactly one enhancement auxiliary information payload, whether the containing layer is valid can be known from each enhancement auxiliary information payload. Accordingly, selection can be achieved using all of the enhancement auxiliary information payloads (ESI _m , m = 1, ..., M). In addition, the index (N _E ) of the enhancement auxiliary information payload to be used for decompression is determined, and the index (N _E ) is always equal to N _B or 0. This means that enhancement is always achieved based on the underlying sound representation or not achieved at all. A more detailed explanation of the selection is given further below.

Successively, the payloads of the basic compressed sound representation components (BSRC ₁ , ..., BSRC _j ) are combined with the basic auxiliary information payloads (i.e. BSI _I and BSI _D,m , m = 1, ..., M ₎ are passed to the base representation decompression processing unit 2200 along with all and _the value _N Reconstruct the basic sound (or sound field) representation using only the basic compressed sound representation components contained within the layer. The required information regarding which components of the basic compressed sound (or sound field) representation are included in the individual layers is assumed to be known to the decompressor 2100 from a data packet with configuration information, which is It is assumed that data packets are previously transmitted and received. The actual decoding of each individual dependent basic auxiliary information payload (BSI _D,m , m = 1,..., N _B ) can be split into two parts as follows:

1. First J _m - 1 basic compressed sound representation components included in the first m layers, assumed in the encoding stage ( Preliminary decoding of each payload (BSI _D,m , m = 1, ..., N _B ) by exploiting its dependency on ).

2. The basic sound components are included in the first N _B > m layers, which are more components than assumed for preliminary decoding. The basic compressed sound representation components ( ) successive corrections of each payload (BSI _D,m , m = 1,...,N _B ) by considering that it is finally reconstructed from. Therefore, correction can be achieved by discarding old information, which means that when certain complementary components are added to the basic compressed sound (or sound field) representation, the dependent basic auxiliary information for each individual complementary component is changed from the original. This is possible due to the initially assumed characteristic of the dependent basic auxiliary information that it is a subset of the dependent basic auxiliary information of .

Ultimately, the reconstructed basic sound (or sound field) representation consists of all of the enhancement auxiliary information payloads (ESI ₁ , ..., ESI _M ), the basic auxiliary information payloads (BSI _I and BSI _D,m , m = 1,...,M), together with the value _N ) and discarding all other enhancement auxiliary information payloads, calculate the final enhanced sound (or sound field) representation. If the value of N _E is 0, all enhancement auxiliary information payloads are discarded and the final reconstructed enhanced sound (or sound field) representation is the same as the reconstructed basic sound (or sound field) representation.

Next, layer selection will be described. In the case where all of the frame data packets can be decompressed independently of each other, the number of the top layer that will actually be used for decompressing the basic sound representation (N _B ) and the index of the enhancement auxiliary information payload to be used for decompression (N _E ) are both set to the highest number (L) of the valid enhancement auxiliary information payload, and the valid enhancement auxiliary information payload itself can be determined by evaluating the validity flags in the enhancement auxiliary information payloads. By utilizing knowledge of the size of each enhancement auxiliary information payload, complex parsing of the payloads' actual data to determine their validity can be avoided.

If differential decompression with inter-frame dependencies is used, decisions from the previous frame must also be considered. In differential decompression, independent frame data packets are transmitted at regular time intervals, making it possible to start decompression from time instants, wherein the determination of the values N _B and N _E becomes frame independent and is performed as previously described.

To elaborate the frame-dependent decision, first, for the kth frame,

The highest number of valid enhancement auxiliary information payloads is L(k).

The highest layer number (e.g. layer index) to be selected and used for decompression of the underlying sound representation is N _B (k).

The number of enhancement auxiliary information payloads to be used for decompression is N _E (k).

Mark it. Using this technique, the highest layer number to be used for decompressing the basic sound representation in N _B (k) is

It is calculated according to

By choosing N _B (k) not greater than L (k) and N _B (k-1), it is ensured that all of the information required for differential decompression of the basic sound representation is available.

The number (N _E (k)) of the enhancement auxiliary information payload to be used for decompression is

It is decided according to

This means in detail that the same corresponding enhancement layer number is selected as long as the top layer number (N _B (k)) to be used for decompression of the basic sound representation does not change. However, in case of a change in N _B (k), the enhancement is disabled by setting N _E (k) to 0. Due to the assumed differential decompression of the enhancement auxiliary information, its change according to N _B (k) is not possible since it requires decompression of the corresponding enhancement auxiliary information layer in the previous frame, which is assumed not to have been performed. Because they will demand it.

Alternatively, if all of the enhancement auxiliary information payloads with numbers up to N _E (k) in the decompression are decompressed in parallel, the selection rule (Equation 4) becomes

can be replaced with

Finally, it should be noted that in case of differential decompression the number of the highest used layer can only be increased for independent frame data packets, whereas it can be decreased in every frame.

Next, embodiments of the present disclosure relate to layered coding of frames of compressed sound representations and to data structures (e.g., bitstreams) representing encoded frames of compressed sound representations in the case of compressed HOA representations. will be described. In detail, proposed changes to the layered coding scheme of compressed HOA representation will be described.

As a correction of the layered coding mode for HOA-based content, configuration of HOA decoding tools such as spatial signal prediction, subband directional signal synthesis and parametric ambience replication (PAR) decoder and frame payloads are transferred to the corresponding HOA enhancement layer. To better adapt, a new usacExtElementType is defined. If the layered coding mode for HOA-based content is activated - this is signaled by SingleLayer==0 -, the corresponding bitstream elements of these tools are used for each layer (including the base layer and one or more enhancement layers). It is proposed to move to a new type of additional HOA extension payload.

Extensions to these tools must be made as auxiliary information is generated to enhance specific HOA representations. In the current definition of layered HOA coding, the provided data only appropriately extends the HOA representation of the top layer. For lower layers, these tools do not adequately enhance the partially reconstructed HOA representation.

Therefore, it would be better to provide auxiliary information for these tools for each layer to better adapt them to the reconstructed HOA representation of the corresponding layer.

In addition, tools such as subband directional signal synthesis and parametric ambience replica decoder are specifically designed for low data rates, where only a few transmit signals are available. The proposed extension will therefore provide an optimal adaptation of the auxiliary information of these tools depending on the number of transmitted signals in the layer. Accordingly, the sound quality of the reconstructed HOA representation for low bit rate layers, such as the base layer, can be significantly increased compared to the existing layered approach.

Furthermore, if CodedVVecLength equal to 1 is signaled in HOADecoderConfig(), the bitstream syntax for encoded V-vector elements for vector-based signals must be adapted to HOA layered coding. In this vector coding mode, V-vector elements for HOA coefficient indices included in the ContAddHoaCoeff set are not transmitted. This set contains all HOA coefficient indices AmbCoeffIdx[i] with AmbCoeffTransitionState equal to 0. Since the original HOA coefficient sequence for these indices is transmitted explicitly, there is no need to add a weighted V-vector signal. Therefore, for these indices the V-vector elements in the conventional approach are set to 0.

However, in layered coding mode, the set of consecutive HOA coefficient indices depends on the transport channels that are currently part of the active layer. This means that the additional HOA coefficient indices transmitted in the upper layer are missing in the lower layers. Then, for HOA coefficient indices belonging to HOA coefficient sequences included in upper layers, the assumption that the vector signal should not contribute to the HOA coefficient sequence is incorrect. Therefore, it is proposed to (explicitly) signal the V-vector elements for these missing coefficient indices.

As a result, it is proposed to define a set of ContAddHoaCoeff for each layer and to use the set of layers to which the V-vector signal is added (to which the transmission signal of the V-vector signal belongs) for the selection of active V-vector elements. do. Nonetheless, it is suggested that the V-vector data stay in HOAFrame() and not be moved to HOAEnhFrame().

Next, integration into MPEG-H bitstream syntax will be described. A corresponding encoding method (e.g., a layered encoding method of a frame of a compressed HOA representation of a sound or sound field) according to embodiments of the present disclosure will be described with reference to FIG. 3 . Proposed changes to the MPEG-H 3D bitstream will be described in the appendix below.

In layered coding mode, the SingleLayer flag in HOADecoderConfig() is inactive (SingleLayer==0) and the number of layers and their corresponding number of assigned HOA transmission signals are defined. In general, a compressed HOA representation may include a plurality of transmitted signals.

Accordingly, in S3010 of FIG. 3, a plurality of transmission signals are assigned to a plurality of hierarchical layers. In other words, transmission signals are distributed to multiple layers. Each layer can be said to contain its own transmission signals assigned to that layer. Each layer may be assigned more than one transmission signal. The plurality of layers may include a base layer and one or more hierarchical enhancement layers. The layers can be ordered from the base layer through the enhancement layers to the overall top enhancement layer (overall top layer).

An additional HOA configuration extension payload with a newly defined usacExtElementType, ID_EXT_ELE_HOA_ENH_LAYER, to carry one payload of spatial signal prediction, subband directional signal synthesis, and PAR decoder data for each HOA enhancement layer (including the base layer). It is proposed to add LOAD and HOA frame extension payloads to the MPEG-H bitstream. These additional payloads will immediately follow a payload of type ID_EXT_ELE_HOA in mpegh3daExtElementConfig() and correspondingly in mpegh3daFrame().

Therefore, in the case of SingleLayer == 0, the components for spatial signal prediction, subband directional signal synthesis and PAR decoder are derived from HOADecoderConfig() to the newly defined HOADecoderEnhConfig() and correspondingly HOAPredictionInfo(), HOADirectionalPredictionInfo() and It is proposed to move HOAParInfo() from HOAFrame() to the newly defined HOAEnhFrame().

Accordingly, in S3020 , respective HOA extension payloads for each layer are generated. The generated HOA extension payload is an aid for parametrically improving the reconstructed HOA representation obtainable from the transmission signals assigned to (e.g., included in) each layer and any layers lower than the respective layer. May contain information. As discussed above, HOA extension payloads may include bitstream elements for one or more of the HOA spatial signal prediction decoding tool, the HOA subband directional signal synthesis decoding tool, and the HOA parametric ambience replication decoding tool. Additionally, HOA extension payloads may have a usacExtElementType of ID_EXT_ELE_HOA_ENH_LAYER.

In S3030 , the generated HOA extension payloads are assigned to their respective layers.

In addition (not shown in Figure 3), an HOA configuration extension page containing bitstream elements for configuring an HOA spatial signal prediction decoding tool, an HOA subband directional signal synthesis decoding tool, and/or an HOA parametric ambience replica decoding tool. A load can be created.

Additionally (not shown in Figure 3), an HOA decoder configuration payload may be generated that includes information indicating the assignment of HOA extension payloads to a plurality of layers.

Next, the transmission of layered bitstreams (eg, MPEG-H bitstreams) will be described. Since all of the extension payloads of the MPEG-H bitstream are byte-aligned and their sizes are explicitly signaled, if the elementLengthPresent flag is assumed to be 1, the de-packer parses the MPEG-H bitstream and Payloads for layers higher than 1 can be extracted and transmitted separately over different transport channels. The base layer contains (eg, consists of) the MPEG-H bitstream excluding data for the upper layers. Missing extension payloads are signaled as empty or inactive. For payloads of type ID_USAC_SCE, ID_USAC_CPE and ID_USAC_LFE, an empty payload is signaled by an elementLength of 0, where elementLengthPresent needs to be set to 1. An empty payload of type ID_USAC_EXT can be signaled by setting the usacExtElementPresent flag to 0 (false).

Accordingly, at S3040 , the generated HOA extension payloads are signaled (eg, transmitted or output) in the output bitstream. Typically, a plurality of layers and their assigned payloads are signaled (eg, transmitted, or output) in an output bitstream. Additionally, an HOA decoder configuration payload and/or HOA configuration extension payload may be signaled (e.g., transmitted, or output) in the output bitstream.

The HOA base layer (layer index = 1) is assumed to be transmitted using the highest error protection and to have a relatively small bitrate. Error protection for subsequent layers (one or more HOA enhancement layers) steadily decreases as the bit rate of the enhancement layers increases. Due to bad transmission conditions and lower error tolerance, the transmission of upper layers may fail and in the worst case only the base layer is transmitted correctly. It is assumed that combined error prevention for all payloads of one layer is applied. Therefore, if transmission of a layer fails, all payloads of the corresponding layer are missing.

In other words, data payloads for multiple layers may be transmitted with respective error protection levels, where the base layer has the highest error protection and one or more enhancement layers have sequentially decreasing error protection.

It is understood that the foregoing steps may be performed in any order and that the example order illustrated in FIG. 3 is non-limiting, unless the steps require certain other steps as prerequisites.

As seen previously, if CodedVVecLength equal to 1 is signaled in HOADecoderConfig(), the bitstream syntax for encoded V-vector elements for vector-based signals must be adapted to HOA layered coding. A corresponding encoding method (e.g., a layered encoding method of a frame of a compressed HOA representation of a sound or sound field) according to embodiments of the present disclosure will be described with reference to FIG. 4 .

In S4010 of FIG. 4, a plurality of transmission signals are assigned to a plurality of hierarchical layers. This step can be performed in the same manner as S3010 described previously.

At S4020 , it is determined whether the vector coding mode is active. This may include determining whether CodedVVecLength==1.

As seen above, in the conventional approach, in vector coding mode, V-vector elements for HOA coefficient indices included in the ContAddHoaCoeff set are not transmitted. This set contains all HOA coefficient indices AmbCoeffIdx[i] with AmbCoeffTransitionState equal to 0. Since the original HOA coefficient sequence for these indices is transmitted explicitly, there is no need to add a weighted V-vector signal. Therefore, for these indices the V-vector elements in the conventional approach are set to 0.

However, in layered coding mode, the set of consecutive HOA coefficient indices depends on the transport channels that are currently part of the active layer. This means that the additional HOA coefficient indices transmitted in the upper layer are missing in the lower layers. Then, for HOA coefficient indices belonging to HOA coefficient sequences included in upper layers, the assumption that the vector signal should not contribute to the HOA coefficient sequence is incorrect.

Accordingly, when the vector coding mode is active, at S4030 , a set of consecutive HOA coefficient indices (e.g., ContAddHoaCoeff) for each layer is determined (e.g., defined) based on the transmission signals assigned to each layer.

If the vector coding mode is active, at S4040 , for each transmitted signal, a V-vector is generated based on the determined set of consecutive HOA coefficient indices for the layer to which the respective transmitted signal is assigned. Each generated V-vector may include elements for arbitrary transmission signals assigned to layers higher than the layer to which each transmission signal is assigned. This step may include using the set of continuous HOA coefficient indices determined for the layer to which the V-vector signal is added (the layer to which the transmission signal of the V-vector signal belongs) for selection of active V-vector elements. Nonetheless, it is suggested that the V-vector data stay in HOAFrame() and not be moved to HOAEnhFrame().

Then, at S4050 , the generated V-vectors (V-vector signals) are signaled in the output bitstream. This may include (explicitly) signaling the V-vector elements for the missing coefficient indices mentioned above.

Steps S4020 to S4050 of FIG. 4 may also be used in connection with the encoding method illustrated in FIG. 3, for example, after S3010. In this case, S3040 and S4050 can be combined into a single signaling step.

It is understood that the foregoing steps may be performed in any order and that the example order illustrated in FIG. 4 is non-limiting, unless the steps require certain other steps as prerequisites.

On the receiver side, the MPEG-H bitstream packer can reinsert correctly received payloads into the base layer MPEG-H bitstream and deliver them to the MPEG-H 3D audio decoder.

Next, HOA decoding initialization (configuration) will be described. HOA configuration payloads of type ID_EXT_ELE_HOA and ID_EXT_ELE_HOA_ENH_LAYER with corresponding sizes in bytes are input to the HOA decoder for its initialization. HOA coding tools are constructed according to the bitstream elements defined in HOAConfig(), which are parsed from a payload of type ID_EXT_ELE_HOA. Additionally, this payload includes the use of layered coding mode, the number of layers and the corresponding number of transmitted signals per layer. Subsequently, if layered coding is enabled (SingleLayer==0), HOAEnhConfig() uses a page of type ID_EXT_ELE_HOA_ENH_LAYER to configure the corresponding spatial signal prediction, subband directional signal synthesis, and parametric ambience replication decoder of each layer. Parsed from loads.

The LayerIdx element from HOAEnhConfig() indicates the order of the HOA enhancement layers along with the order of the HOA enhancement layer configuration payloads in mpegh3daExtElementConfig(). The order of HOA enhancement layer frame payloads of type ID_EXT_ELE_HOA_ENH_LAYER in mpegh3daFrame() is the same as the order of configuration payloads in mpegh3daExtElementConfig() to clearly assign frame payloads to corresponding layers.

In the case of SingleLayer==1 (single layer coding), payloads of type ID_EXT_ELE_HOA_ENH_LAYER are ignored, and the spatial signal prediction, subband directional signal synthesis and parametric ambience replication decoders use the corresponding data from HOADecoderConfig() for their configuration. Use .

Next, HOA frame decoding in layered mode will be described. A corresponding decoding method (eg, a method of decoding a frame of a compressed HOA representation of a sound or sound field) according to embodiments of the present disclosure will be described with reference to FIG. 5 . It is understood that the compressed HOA representation (eg, the output of the methods of FIG. 3 or FIG. 4 described above) is encoded in a plurality of hierarchical layers, including a base layer and one or more enhancement layers.

At S5010 of Figure 5, a bitstream associated with a frame of the compressed HOA representation is received.

The 3D audio core decoder decodes the correctly transmitted HOA transmission signals and generates transmission signals with all zero samples for the corresponding invalid payloads. Decoded transmission signals are input to the HOA decoder along with usacExtElementPresent flags, data and sizes of HOA payloads of type ID_EXT_ELE_HOA and ID_EXT_ELE_HOA_ENH_LAYER. Extension payloads from type ID_USAC_EXT with the usacExtElementPresent flag set to false must be signaled to the HOA decoder as missing payloads to ensure assignment of payloads to the corresponding layers.

In S5020 , payloads for a plurality of layers are extracted. Each payload may include transmission signals assigned to each layer.

At this stage, the HOA decoder can parse HOAFrame() from a payload of type ID_EXT_ELE_HOA.

Valid payloads of type ID_EXT_ELE_HOA_ENH_LAYER and invalid payloads of type ID_EXT_ELE_HOA_ENH_LAYER are then determined by evaluating the corresponding usacExtElementPresent flag of the payloads, where invalid payloads are indicated by the usacExtElementPresent flag being false and Assigning HOA enhancement payloads to enhancement layer indices can be seen from the HOA decoder configuration.

At S5030 , the highest available layer among the plurality of layers for decoding is determined.

Because the layers depend on each other in terms of transmitted signals, the HOA decoder can decode a layer only when all of the layers with lower indices are received correctly. The highest available layer can be selected at this stage so that all layers up to the highest available layer are received correctly. Details of this step will be described below.

In S5040 , the HOA extension payload assigned to the highest available layer is extracted. As seen above, the HOA extension payload may contain auxiliary information to parametrically enhance the reconstructed HOA representation corresponding to the highest available layer. There, the reconstructed HOA representation corresponding to the highest available layer may be obtainable based on transmission signals assigned to the highest available layer and any layers lower than the highest available layer.

In addition, HOA extension payloads assigned to the remaining layers of the plurality of layers may be extracted. Each HOA extension payload may include auxiliary information to parametrically enhance the reconstructed HOA representation corresponding to its respective assigned layer. The reconstructed HOA representation corresponding to each assigned layer may be obtainable from transmission signals assigned to that layer and any layers lower than that layer.

Additionally (not shown in Figure 5), the decoding method may include extracting the HOA configuration extension payload. This can be done by parsing the bitstream. The HOA configuration extension payload may include bitstream elements for configuring the HOA spatial signal prediction decoding tool, the HOA subband directional signal synthesis decoding tool, and/or the HOA parametric ambience replica decoding tool.

At S5050 , a (partially) reconstructed HOA representation corresponding to the highest available layer is generated based on transmission signals assigned to the highest available layer and any layers lower than the highest available layer.

The number of actually used transmission signals (I _ADD,LAY (k)) is set according to the index (M LAY (k)) of the highest available layer (M _LAY (k)), and the first preliminary HOA representation is from HOAFrame() and Decoded from the corresponding transmission signals of the layer and any lower layers.

Then, at S5060 , the reconstructed HOA representation is enhanced (e.g., parametrically enhanced) using auxiliary information contained in the HOA extension payload assigned to the highest available layer.

That is, the HOA representation obtained in S5050 is then used to predict spatial signals using the HOAEnhFrame() data parsed from the HOA enhancement layer extension payload of type ID_EXT_ELE_HOA_ENH_LAYER of the currently active layer (M _LAY (k)), i.e. the highest available layer; It is enhanced by subband directional signal synthesis and parametric ambience replication decoder.

Information used in steps S5020 to S5060 may be known as layer information.

It is understood that the above-mentioned steps may be performed in any order and that the example order illustrated in FIG. 5 is non-limiting, unless the steps require certain other steps as prerequisites.

Next, details about determining (eg, selecting) the highest available layer in S5030 will be described.

As seen previously, the HOA decoder can decode a layer only when all of the layers with lower indices are received correctly, because the layers are dependent on each other in terms of transmitted signals.

For selection of the highest decodable layer, the HOA decoder can generate a set of invalid layer indices, where subtracting 1 from the smallest index from this set gives the index of the highest decodable enhancement layer (M _LAY ). Lose. The set of invalid layer indices can be determined by evaluating the validity flags of the corresponding HOA extension payloads.

In other words, determining the highest available layer may include determining a set of invalid layer indices that indicate layers that were not validly received. This may further include determining the highest available layer as a layer that is one layer below the layer indicated by the smallest index in the set of invalid layer indices. Thereby, it is ensured that all layers below the highest available layer are received validly.

In case of differential encoding of frames, the index of the highest available layer of the previous (eg immediately preceding) frame would have to be taken into account. First, a situation will be described where the index of the highest available layer of the previous (eg preceding) frame is maintained.

If the index of the highest available layer (e.g., highest decodable layer) for the current frame is the same as the index of the previous frame (M _LAY (k-1)), then the layer index (M _LAY (k)) of the current frame is M It is set to _LAY (k-1).

Subsequently, as seen above, the number of actually used transmission signals (I _ADD,LAY (k)) is set according to M _LAY (k), and the first preliminary HOA representation is obtained from HOAFrame() and the layer and random is decoded from the corresponding transmission signals of the lower layers of . As seen earlier, this HOA representation then uses the HOAEnhFrame() data parsed from the HOA enhancement layer extension payload of type ID_EXT_ELE_HOA_ENH_LAYER of the currently active layer (M _LAY (k)) for spatial signal prediction, subband directional signal synthesis, and It is improved by a parametric ambience replication decoder.

Next, the situation of switching to an index lower than that of the highest available layer of the previous (eg, preceding) frame will be described. That is, if the index of the highest decodable layer for the current frame is smaller than the index of the layer of the previous frame (M _LAY (k-1)), the HOA decoder can use M _LAY (k) as the highest decodable layer for the current frame. Set as the index of the layer. For the new layer, decoding of payloads for spatial signal prediction, subband directional signal synthesis and parametric ambience replica decoder can only start in the next HOA frame with hoaIndependencyFlag equal to 1. Until such HOAFrame() is received, the HOA representation of the layer at index (M _LAY (k)) is reconstructed without performing spatial signal prediction, subband directional signal synthesis, and parametric ambience replica decoder. This means that the number of actually used transmission signals (I _ADD,LAY (k)) is set according to M _LAY (k), and only the first preliminary HOA representation is drawn from HOAFrame() and the corresponding layer and any lower layers. This means that it is decoded from transmission signals. Then, when a HOAFrame() with hoaIndependencyFlag equal to 1 is received, spatial signal prediction, subband directional signals are used to improve the preliminary HOA representation, such that the full quality of the currently active layer is provided for this frame. Payloads for synthetic and parametric ambience replica decoders are parsed and decoded.

Therefore, the proposed method uses the HOA extension fee assigned to the highest available layer if the highest available layer of the current frame is lower than the highest available layer of the previous frame (if the current frame was coded differentially with respect to the previous frame). A step (not shown in Figure 5) may include deciding not to perform parametric enhancement of the reconstructed HOA representation using the auxiliary information included in the load.

Generally, determining the highest available layer for the current frame may include determining a set of invalid layer indices that indicate layers that have not been validly received for the current frame. This may further include determining the highest available layer of the previous frame preceding the current frame. This also means (if the current frame was coded differentially with respect to the previous frame) the highest available layer is the highest available layer of the previous frame and the one below the layer indicated by the lowest index in the set of invalid layer indices. It may additionally include a step of determining a lower layer among the layers of .

An alternative solution could always parse all valid enhancement layer payloads (eg, HOA expansion payloads) in parallel, even if they are currently inactive. This will enable direct switching to lower index layers with maximum quality, where spatial signal prediction, subband directional signal synthesis and parametric ambience replication (PAR) decoders can be applied directly on the switched frames.

Next, the situation of switching to an index higher than the index of the highest available layer of the previous (eg, preceding) frame will be described. This transition to a layer with a higher index can only be applied if mpegh3daFrame() has usacIndependencyFlag equal to 1 (e.g., if the frame is an independent frame) because the corresponding payloads of previous frames or decoding Because all the states are missing. Therefore, until an mpegh3daFrame() (e.g., independent frame) with usacIndependencyFlag equal to 1 that contains valid data for the upper decodable layer is received, the HOA decoder sets the HOA layer index (M _LAY (k)) to M _LAY Keep it equal to (k-1). M _LAY (k) is then set to the highest decodable layer index for the current frame, and the number of actually used transmission signals (I _ADD,LAY (k)) is determined accordingly. A preliminary HOA representation of that layer is generated using HOAFrame() and HOAEnhFrame() data decoded from the corresponding transmitted signals and parsed from the HOA enhancement layer extension payload of type ID_EXT_ELE_HOA_ENH_LAYER of the currently active layer (M _LAY (k)). It is enhanced by spatial signal prediction, subband directional signal synthesis, and parametric ambience replication decoder.

It is understood that the proposed method of layered encoding of compressed sound representations can be implemented by an encoder for layered encoding of compressed sound representations. This encoder may comprise respective units adapted to perform the respective steps described above. An example of such an encoder 6000 is schematically illustrated in FIG. 6. For example, this encoder 6000 may include a transmission signal allocation unit 6010 adapted to perform S3010 mentioned above, an HOA expansion layer payload generation unit 6020 adapted to perform S3020 mentioned above, It may include an HOA extension payload allocation unit 6030 adapted to perform the mentioned S3030, and a signaling unit or output unit 6040 adapted to perform the previously mentioned S3040. The respective units of this encoder are adapted to perform the processing performed by each of the respective units, i.e. to perform some or all of the above-mentioned steps of the proposed encoding method schematically illustrated in Figure 3. It is further understood that the processor 6100 may be implemented by a processor 6100 of a customized computing device. Additionally or alternatively, processor 6100 may be adapted to perform each of the steps of the encoding method schematically illustrated in FIG. 4 . For this purpose, the processor 6100 can be adapted to implement the respective units of the encoder. The encoder or computing device may further include memory 6200 accessible by processor 6100.

It is further understood that the proposed method for decoding a compressed sound representation encoded in a plurality of hierarchical layers can be implemented by a decoder for decoding a compressed sound representation encoded in a plurality of hierarchical layers. This decoder may comprise respective units adapted to perform the respective steps described above. An example of such a decoder 7000 is schematically illustrated in FIG. 7. For example, such a decoder 7000 may include a receiving unit 7010 adapted to perform the previously mentioned S5010, a payload extraction unit 7020 adapted to perform the previously mentioned S5020, and a payload extraction unit 7020 adapted to perform the previously mentioned S5030. a top usable layer determination unit 7030 adapted to perform the aforementioned S5040, an HOA extension payload extraction unit 7040 adapted to perform the aforementioned S5050, and a reconstructed HOA representation generation unit adapted to perform the aforementioned S5050. 7050, and an enhancement unit 7060 adapted to perform S5060 mentioned above. A processor of the computing device, wherein the respective units of this decoder are adapted to perform the processing performed by each of the respective units, i.e. to perform some or all of the above-mentioned steps of the proposed decoding method. It is further understood that the embodiment may be implemented by (7100). The decoder or computing device may further include memory 7200 accessible by processor 7100.

Next, a data structure (e.g., bitstream) for accommodating (e.g., representing) a compressed HOA representation in a layered coding mode will be described. This data structure may result from using the proposed encoding methods and may be decoded (e.g., decompressed) using the proposed decoding method.

The data structure may include a plurality of HOA frame payloads corresponding to respective layers of the plurality of hierarchical layers. The plurality of transmission signals may be assigned to (eg, belong to) respective layers of the plurality of layers. The data structure includes a respective HOA extension payload containing auxiliary information to parametrically improve the reconstructed HOA representation obtainable from the transmission signals assigned to the respective layer and any layers lower than the respective layer. can do. As seen above, HOA frame payloads and HOA extension payloads for multiple layers may be provided with respective error prevention levels. In addition, HOA extension payloads may include the bitstream elements discussed earlier and may have a usacExtElementType of ID_EXT_ELE_HOA_ENH_LAYER. The data structure may also further include an HOA configuration extension payload and/or an HOA decoder configuration payload including the bitstream elements discussed above.

It should be noted that the description and drawings merely illustrate the principles of the proposed methods and devices. Accordingly, it will be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope. Moreover, all examples listed herein are expressly intended for educational purposes only, primarily to assist the reader in understanding the principles of the proposed methods and devices and the concepts contributed by the inventors to advance the technology. and should not be construed as being limited to these specifically enumerated examples and conditions. Moreover, all statements herein reciting the principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to include equivalents thereof.

The methods and devices described herein may be implemented as software, firmware, and/or hardware. Certain components may be implemented as software running on, for example, a digital signal processor or microprocessor. Other components may be implemented, for example, as hardware and/or as application specific integrated circuits (ASICs). Signals from the described methods and devices may be stored on a medium such as random access memory or optical storage media. They may be transmitted via networks, such as radio networks, satellite networks, wireless networks or wired networks, such as the Internet.

Appendix:

Proposed MPEG-H 3D bitstream changes

Changes are indicated by highlighting them in gray:

NumOfDirSigsPerLayer[lay] This element determines the number of active directional signals in the current HOAFrame() that are actually used in the HOA enhancement layer ( lay ).

AddHoaCoeffPerLayer[lay] This array contains the HOA coefficient index for each additional surrounding HOA coefficient actually used in the HOA enhancement layer (lay ).

NumOfAddHoaChansPerLayer[lay] This element signals the total number of additional surrounding HOA coefficients actually used in the HOA enhancement layer ( lay ).

add this table

Update this table:

codedLayerCh This element indicates, for the first (i.e. base) layer, the number of included transmission signals, given by codedLayerCh + MinNumOfCoeffsForAmbHOA. For higher (i.e. enhancement) layers, this element indicates the number of additional signals included in the enhancement layer compared to the next lower layer, given by codedLayerCh + 1.

HOALayerChBits This element indicates the number of bits for reading codedLayerCh .

NumLayers This element indicates the total number of layers in the bitstream (after reading from HOADecoderConfig()).

NumHOAChannelsLayer This element is an array composed of NumLayers elements, of which the i th element indicates the number of transmission signals included in all layers up to the i th layer.

12.4.1.x Frame and user-dependent parameters

M _LAY (k) Number of all layers actually used for the kth frame (to be specified) at the decoder side. In case of layered coding (indicated by SingleLayer==0), this number must be less than or equal to the total number of layers present in the bitstream, i.e. M _LAY = NumLayers. In case of single layered coding (indicated by SingleLayer==1), M _LAY is set to 1.

Depending on the choice of M _LAY (k), the number of additional transport channels actually used (I _ADD,LAY (k)) for spatial HOA decoding (i.e. in addition to the O _MIN channels that are implicitly always used) ) is calculated as follows:

if( SingleLayer | (! SingleLayer & M _LAY (k) == NumLayers))

{

I _ADD,LAY (k) = NumOfAdditionalCoders;

}

else

{

I _ADD,LAY (k) = NumHOACannelsLayer[M _LAY (k) - 1] - MinNumOfCoeffsForAmbHOA;

}

VVecLength and VVecCoeffId

The codedVVecLength word indicates:

0) Total vector length (NumOfHoaCoeffs elements). Indicates that all coefficients (NumOfHoaCoeffs) for dominance vectors are specified.

1) Vector elements 1 to MinNumOfCoeffsForAmbHOA and all elements defined in ContAddHoaCoeff[lay] of the currently active layer with index lay=0??NumLayers-1 are not transmitted. For single layer mode (SingleLayer==1), the variable NumLayers must be set to 1. Indicates that only those coefficients of the dominance vector corresponding to numbers greater than MinNumOfCoeffsForAmbHOA are specified. The additional NumOfContAddAmbHoaChan[lay] coefficients identified in ContAddAmbHoaChan[lay] are subtracted. The ContAddAmbHoaChan[lay] list specifies additional channels corresponding to orders exceeding the MinAmbHoaOrder order.

2) Vector elements 1 to MinNumOfCoeffsForAmbHOA are not transmitted. Indicates that the coefficients of dominance vectors corresponding to numbers greater than MinNumOfCoeffsForAmbHOA are specified.

In the case of codedVVecLength==1, both the VVecLength[i] array as well as the VVecCoeffId[i][m] 2D array are valid for the VVector at index i, and in other cases, both the VVecLength element as well as the VVecCoeffId[m] array are valid for the VVector at index i. It is valid for all VVectors in the HOAFrame. For the assignment algorithm below, the helper function is defined as follows:

switch CodedVVecLength {

case 0:

VVecLength = NumOfHoaCoeffs;

for (m=0; m<VVecLength; ++m) {

VVecCoeffId[m] = m;

}

break;

case 1:

for (i=0; i < NumOfVecSigs; ++i) {

lay = VecSigLayerIdx[i];

VVecLength[i] = NumOfHoaCoeffs

-.MinNumOfCoeffsForAmbHOA

-NumOfContAddHoaChans[lay];

CoeffIdx = MinNumOfCoeffsForAmbHOA+1;

for (m=0; m<VVecLength[i]; ++m) {

bIsInArray = isMemberOf(CoeffIdx,

ContAddHoaCoeff[lay];

NumOfContAddHoaChans[lay]);

while (bIsInArray) {

CoeffIdx++;

bIsInArray = isMemberOf(CoeffIdx,

ContAddHoaCoeff[lay];

NumOfContAddHoaChans[lay]);

}

VVecCoeffId[i][m] = CoeffIdx-1;

}

}

break;

case 2:

VVecLength = NumOfHoaCoeffs - MinNumOfCoeffsForAmbHOA;

for (m=0; m< VVecLength; ++m) {

VVecCoeffId[m] = m + MinNumOfCoeffsForAmbHOA;

}

}

The first switch statement with three cases (case 0 to case 2) thus provides a way to determine the dominant vector length with a number (VVecLength) and indices of coefficients (VVecCoeffId).

Conversion to 12.4.1.X VVec elements

The type of inverse quantization of Vvector is signaled by the NbitsQ word. An NbitsQ value of 4 indicates vector quantization. When NbitsQ is 5, uniform 8 bit scalar dequantization is performed. In contrast, an NbitsQ value greater than or equal to 6 indicates application of Huffman decoding of the scalar quantized Vvector. The prediction mode is denoted as PFlag, while CbFlag represents the Huffman Table information bit.

if (CodedVVecLength == 1) {

VVecLengthUsed = VVecLength[i];

VVecCoeffIdUsed = VVecCoeffId[i];

} else {

VVecLengthUsed = VVecLength;

VVecCoeffIdUsed = VVecCoeffId;

}

if (NbitsQ(k)[i] == 4) {

if (NumVvecIndices == 1) {

for (m=0; m< VVecLengthUsed; ++m) {

idx = VVecCoeffIdUsed[m];

= WeightVal[0] * VecDict[900].[VvecIdx[0]][idx];

}

} else {

cdbLen = O ;

if (N==4) {

cdbLen = 32;

}

for (m=0; m<O; ++m) {

TmpVVec[m] = 0;

for (j=0; j< NumVvecIndices; ++j) {

TmpVVec[m] += WeightVal[j] * VecDict[cdbLen].[VvecIdx[j]][m];

}

}

FNorm = 0.0;

for (m=0; m<O; ++m) {

FNorm += TmpVVec[m] * TmpVVec[m];

}

FNorm = (N+1)/sqrt(FNorm);

for (m=0; m< VVecLengthUsed; ++m) {

idx = VVecCoeffIdUsed[m];

= TmpVVec[idx] * FNorm;

}

}

}

elseif (NbitsQ(k)[i] == 5) {

for (m=0; m< VVecLengthUsed; ++m) {

(N+1)*aVal[i][m];

}

}

elseif (NbitsQ(k)[i] >= 6) {

for (m=0; m< VVecLengthUsed; ++m) {

= (N+1) * (2^(16 - NbitsQ(k)[i])*aVal[i][m])/2^15;

if (PFlag(k)[i] == 1) {

+= ;

}

}

}

Claims (11)

A method for decoding a compressed Higher Order Ambisonics (HOA) representation of a sound or sound field, comprising:
Receiving a bitstream including the compressed HOA representation, the bitstream including a plurality of hierarchical layers including a base layer and one or more hierarchical enhancement layers;
determining a highest available layer among the plurality of hierarchical layers for decoding;
Determining that the parameter CodedVVecLength is CodedVVecLength = 2 - based on this determination, determine that vector elements from 1 to MinNumOfCoeffsForAmbHOA have not been transmitted and that coefficients of dominant vectors corresponding to numbers greater than MinNumOfCoeffsForAmbHOA are specified. and the VVecCoeffId array is determined based on MinNumOfCoeffsForAmbHOA -;
extracting an HOA extension payload assigned to the highest available layer, wherein the HOA extension payload includes auxiliary information to parametrically enhance a reconstructed HOA representation corresponding to the highest available layer, the reconstructed HOA representation corresponding to an available layer is based on transmission signals assigned to the highest available layer and any layers lower than the highest available layer;
Decoding the compressed HOA representation corresponding to the highest available layer based on layer information, wherein the layer information indicates an active enhancement layer, the active enhancement layer being an active directional signal in the current frame of the active enhancement layer. Can be used to determine the number of -; and
Parametrically enhancing the decoded HOA representation using the auxiliary information included in the HOA extension payload assigned to the highest available layer.
Method, including. The method of claim 1, wherein the layer information includes enhancement information including at least one of spatial signal prediction, subband directional signal synthesis, and Parametric Ambience Replication (PAR) decoder. According to paragraph 1,
ContAddHoaCoeff The method further comprising v-vector elements that are not transmitted for indices identical to the additional HOA coefficient indices included in the set. The method of claim 1, wherein the layer information includes NumLayers elements, and each element indicates the number of transmission signals included in all layers up to the i-th layer. The method of claim 1, wherein the layer information includes an indicator of all layers actually used for the kth frame. delete delete delete A non-transitory computer-readable medium comprising instructions that, when executed by a processor, perform the method according to claim 1. A device for decoding a compressed higher-order ambisonics (HOA) representation of a sound or sound field, comprising:
A receiver configured to receive a bitstream comprising the compressed HOA representation, the bitstream comprising a plurality of hierarchical layers including a base layer and one or more hierarchical enhancement layers, and
decoder
, wherein the decoder:
determine the highest available layer among the plurality of hierarchical layers for decoding;
Determine that the parameter CodedVVecLength is CodedVVecLength = 2 - and, based on this determination, determine that vector elements from 1 to MinNumOfCoeffsForAmbHOA have not been transmitted, and that the coefficients of the dominant vectors corresponding to numbers greater than MinNumOfCoeffsForAmbHOA are specified, and that the VVecCoeffId array is MinNumOfCoeffsForAmbHOA Determined on the basis of -;
Extract an HOA extension payload assigned to the highest available layer, wherein the HOA extension payload includes auxiliary information to parametrically enhance a reconstructed HOA representation corresponding to the highest available layer, and the reconstructed HOA representation corresponding to an available layer is based on transmission signals assigned to the highest available layer and any layers lower than the highest available layer;
Decode the compressed HOA representation corresponding to the highest available layer based on layer information, wherein the layer information indicates an active enhancement layer, the active enhancement layer comprising: Can be used to determine number -;
Parametrically enhance the decoded HOA representation using the auxiliary information contained in the HOA extension payload assigned to the highest available layer.
configured device. delete