[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

EP3706444A1 - Improved rendering of immersive audio content - Google Patents

Improved rendering of immersive audio content Download PDF

Info

Publication number
EP3706444A1
EP3706444A1 EP20167910.7A EP20167910A EP3706444A1 EP 3706444 A1 EP3706444 A1 EP 3706444A1 EP 20167910 A EP20167910 A EP 20167910A EP 3706444 A1 EP3706444 A1 EP 3706444A1
Authority
EP
European Patent Office
Prior art keywords
audio
rendering
audio object
speaker
gains
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP20167910.7A
Other languages
German (de)
French (fr)
Other versions
EP3706444B1 (en
Inventor
Michael William Mason
Juan Felix TORRES
Antonio MATEOS SOLÉ
Daniel Arteaga
Adam Mills
Mark David DE BURGH
Andrew Owen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dolby International AB
Dolby Laboratories Licensing Corp
Original Assignee
Dolby International AB
Dolby Laboratories Licensing Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dolby International AB, Dolby Laboratories Licensing Corp filed Critical Dolby International AB
Priority to EP23219882.0A priority Critical patent/EP4333461A3/en
Publication of EP3706444A1 publication Critical patent/EP3706444A1/en
Application granted granted Critical
Publication of EP3706444B1 publication Critical patent/EP3706444B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S7/00Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30Control circuits for electronic adaptation of the sound field
    • H04S7/302Electronic adaptation of stereophonic sound system to listener position or orientation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S7/00Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30Control circuits for electronic adaptation of the sound field
    • H04S7/307Frequency adjustment, e.g. tone control
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04RLOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R3/00Circuits for transducers, loudspeakers or microphones
    • H04R3/04Circuits for transducers, loudspeakers or microphones for correcting frequency response
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04RLOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R3/00Circuits for transducers, loudspeakers or microphones
    • H04R3/12Circuits for transducers, loudspeakers or microphones for distributing signals to two or more loudspeakers
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04RLOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R5/00Stereophonic arrangements
    • H04R5/04Circuit arrangements, e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S7/00Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30Control circuits for electronic adaptation of the sound field
    • H04S7/302Electronic adaptation of stereophonic sound system to listener position or orientation
    • H04S7/303Tracking of listener position or orientation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S2400/00Details of stereophonic systems covered by H04S but not provided for in its groups
    • H04S2400/07Generation or adaptation of the Low Frequency Effect [LFE] channel, e.g. distribution or signal processing
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S2400/00Details of stereophonic systems covered by H04S but not provided for in its groups
    • H04S2400/11Positioning of individual sound objects, e.g. moving airplane, within a sound field
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S2420/00Techniques used stereophonic systems covered by H04S but not provided for in its groups
    • H04S2420/03Application of parametric coding in stereophonic audio systems
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S2420/00Techniques used stereophonic systems covered by H04S but not provided for in its groups
    • H04S2420/11Application of ambisonics in stereophonic audio systems

Definitions

  • the present document relates to methods and apparatus for rendering of object-based audio content.
  • the present document relates to methods and apparatus for improved immersive rendering of audio objects having associated metadata specifying extent (e.g., size) of the audio objects, diffusion, and/or divergence.
  • extent e.g., size
  • These methods and apparatus are applicable to cinema sound reproduction systems and home cinema sound reproduction systems, for example.
  • audio object may refer to a stream of audio object signals and associated audio object metadata.
  • the metadata may indicate at least the position of the audio object.
  • the metadata also may indicate decorrelation data, rendering constraint data, content type data (e.g. dialog, effects, etc.), gain data, trajectory data, etc.
  • Some audio objects may be static, whereas others may have time-varying metadata: such audio objects may move, may change extent (e.g., size) and/or may have other properties that change over time.
  • audio objects may be humans, animals or any other elements serving as sound sources.
  • the Audio Definition Model formalizes the description of the structure of metadata that can be applied in the rendering of audio data to one of the loudspeaker configurations specified in Recommendation ITU-R BS.2051.
  • the ADM specifies a metadata model that describes the relationship between a group or groups of raw audio data and how they should be interpreted so that when reproduced, the original or authored audio experience is recreated.
  • an emphasis on flexibility provides multiple ways to describe the variety of immersive experiences which may be on offer.
  • the present document frequently makes reference to the ADM, the subject matter described therein is equally applicable to other specifications of metadata and other metadata models.
  • B-chain processing or playback system In order to reproduce an immersive audio experience, the description must be interpreted in the context of a playback environment to create speaker specific feeds. This process can typically be split into two steps, of which the second step is sometimes referred to as B-chain processing or playback system:
  • the renderer (rendering apparatus, e.g., baseline renderer) described in the present document addresses the first step of interpreting the description of the audio, e.g., in ADM, to create ideal speaker feeds-which can themselves be captured as a simpler ADM that does not require further rendering before reproduction.
  • the present document addresses the above issues related to treatment of metadata and describes methods and apparatus for improved rendering of object-based audio content for playback, in particular of object-based audio content including audio objects for which one or more of extent, diffusion, and divergence are specified by the associated metadata.
  • the input audio may include at least one audio object and associated metadata.
  • the associated metadata may indicate at least a location (e.g., position) of the audio object.
  • the method may optionally comprise referring to the metadata for the audio object and determining whether a phantom object at the location of the audio object is to be created.
  • the method may comprise creating two additional audio objects associated with the audio object such that respective locations of the two additional audio objects are evenly spaced from the location of the audio object, on opposite sides of the location of the audio object when seen from an intended listener's position in the playback environment.
  • the additional audio objects may be located in the horizontal plane in which the audio object is located.
  • the additional audio objects' locations may be fixed with respect to the location of the audio object.
  • the additional audio objects may be evenly spaced from the intended listener's position, e.g., at equal radius.
  • the additional audio objects may be referred to as virtual audio objects.
  • the method may further comprise determining respective weight factors for application to the audio object and the two additional audio objects.
  • the weight factors may be mixing gains.
  • the weight factors (e.g., mixing gains) may impose a desired relative importance (e.g., relative weight) across the three objects.
  • the two additional audio objects may have equal weight factors.
  • the method may yet further comprise rendering the audio object and the two additional audio objects to one or more speaker feeds in accordance with the determined weight factors.
  • the rendering of the audio object and the two additional audio objects to the one or more speaker feeds may result in a gain coefficient for each of the one or more speaker feeds (e.g., for an audio object signal of the audio object).
  • the proposed method allows efficient and accurate generation of a phantom object for the audio object at the location of the audio object.
  • audio power may be more equally distributed among speakers of a speaker layout, thus avoiding overload at particular speakers of the speaker layout.
  • the associated metadata may further indicate a distance measure indicative of a distance between the two additional audio objects.
  • the distance measure may be indicative of a distance between each of the additional audio objects and the audio object, such as an angular distance, or a Euclidean distance.
  • the distance may be indicative of the distance between the two additional audio objects themselves, such as an angular distance or a Euclidean distance.
  • the associated metadata may further indicate a measure of relative importance (e.g., relative weight) of the two additional audio objects compared to the audio object.
  • the measure of relative importance may be referred to as divergence, and be defined by a divergence parameter (divergence value), for example a divergence parameter d ⁇ [0, 1], with 0 indicating zero relative importance of the additional audio objects and 1 indicating zero relative importance of the audio object-i.e., full relative importance of the additional audio objects.
  • the weight factors may be determined based on said measure of relative importance.
  • the method may further comprise normalizing the weight factors based on said distance measure.
  • the weight factors may be normalized (e.g., scaled) such that a function f ( g 1 , g 2 , D ) of the weight factors g 1 , g 2 and the distance measure D attains a predetermined value, e.g., 1.
  • the perceptible loudness for the audio object matches the artistic intent of the content creator.
  • the normalization may represent an amplitude preserving pan to account for coherent summation of the signals of the additional audio objects.
  • the normalization may represent a power preserving pan.
  • the weight factors may be normalized such that a sum of equal powers of the normalized weight factors is equal to a predetermined value.
  • An exponent of the normalized weight factors in said sum may be determined based on the distance measure.
  • the weight factors may be mixing gains.
  • the predetermined value may be 1, for example.
  • normalization of the weight factors may be performed on a (frequency) sub-band basis, in dependence on frequency. That is, normalization may be performed for each of a plurality of sub-bands.
  • the exponent of the normalized weight factors in said sum may be determined on the basis of a frequency of the respective sub-band.
  • the exponent may be a function of the distance measure and the frequency, p ( D,f ).
  • the aforementioned first and second thresholds may be lower than for lower frequencies. That is, the first threshold may be a monotonically decreasing function of frequency, and the second threshold may be a monotonically decreasing function of frequency.
  • the frequency may be the center frequency of a respective sub-band or may be any other frequency suitably chosen within the respective sub-band.
  • the method may further comprise determining a set of rendering gains for mapping (e.g., panning) the audio object and the two additional audio objects to the one or more speaker feeds.
  • the method may yet further comprise normalizing the rendering gains based on said distance measure.
  • the normalization of the rendering gains may represent an amplitude preserving pan. Otherwise, for sufficient distance between the additional audio objects, the normalization may represent a power preserving pan.
  • the rendering gains may be normalized such that a sum of equal powers of the normalized rendering gains for all of the one or more speaker feeds and for all of the audio objects and the two additional audio objects is equal to a predetermined value.
  • An exponent of the normalized rendering gains in said sum may be determined based on said distance measure.
  • the predetermined value may be 1, for example.
  • normalization of the rendering gains may be performed on a (frequency) sub-band basis and in dependence on frequency. That is, normalization may be performed for each of a plurality of sub-bands.
  • the exponent of the rendering gains in said sum may be determined on the basis of a frequency of the respective sub-band.
  • the exponent may be a function of the distance measure and the frequency, p ( D, f ) .
  • the aforementioned first and second thresholds may be lower than for lower frequencies. That is, the first threshold may be a monotonically decreasing function of frequency, and the second threshold may be a monotonically decreasing function of frequency.
  • the frequency may be the center frequency of a respective sub-band or may be any other frequency suitably chosen within the respective sub-band.
  • the input audio may include at least one audio object and associated metadata.
  • the associated metadata may indicate at least a location (e.g., position) of the at least one audio object and a three-dimensional extent (e.g., size) of the at least one audio object.
  • the method may comprise rendering the audio object to one or more speaker feeds in accordance with its three-dimensional extent. Said rendering of the audio object to one or more speaker feeds in accordance with its three-dimensional extent may be performed by determining locations of a plurality of virtual audio objects within a three-dimensional volume defined by the location of the audio object and its three-dimensional extent.
  • the virtual audio objects may be referred to as virtual sources.
  • Candidates for the virtual audio objects may be arranged in a grid (e.g., a three-dimensional rectangular grid) across the playback environment. Determining said locations may involve imposing a respective minimum extent for the audio object in each of the three dimensions (e.g., ⁇ x,y,z ⁇ or ⁇ r, ⁇ , ⁇ ⁇ ) . Said rendering of the audio object to one or more speaker feeds in accordance with its three-dimensional extent may be performed by further, for each virtual audio object, determining a weight factor that specifies the relative importance of the respective virtual audio object.
  • Said rendering of the audio object to one or more speaker feeds in accordance with its three-dimensional extent may be performed by further rendering the audio object and the plurality of virtual audio objects to the one or more speaker feeds in accordance with the determined weight factors.
  • the rendering of the audio object and the virtual audio objects to the one or more speaker feeds may be performed by a so-called point panner, i.e., the audio object and the plurality of virtual audio objects may be treated as respective point sources.
  • the rendering of the audio object and the virtual audio objects to the one or more speaker feeds may result in a gain coefficient for each of the one or more speaker feeds (e.g., for an audio object signal of the audio object).
  • the proposed method allows for efficient and accurate rendering of audio objects having extent, e.g., a three-dimensional size.
  • the proposed method allows for efficient and accurate rendering of audio objects that take a three-dimensional volume in the reproduction environment.
  • the audio object When seen from the intended listener's position, the audio object thus not only features width and height, but can additionally feature depth.
  • the proposed method provides for independent control of each of the three spatial dimensions of extent (e.g., ⁇ x,y,z ⁇ or ⁇ r, ⁇ , ⁇ ⁇ ), and thus provides for a rendering framework that allows for greater flexibility at the time of content creation. In consequence, the proposed method provides the rendering framework for more immersive, more realistic rendering of audio objects with extent.
  • the method may further comprise, for each virtual audio object and for each of the one or more speaker feeds, determining a gain for mapping the respective virtual audio object to the respective speaker feed.
  • the gains may be point gains.
  • the gains may be determined based on the location of the respective virtual audio object and the location of the respective speaker feed (i.e., the location of a speaker for playback of the respective speaker feed).
  • the method may yet further comprise, for each virtual object and for each of the one or more speaker feeds, scaling the respective gain with the weight factor of the respective virtual audio object.
  • the method may further comprise, for each speaker feed, determining a first combined gain depending on the gains of those virtual audio objects that lie within a boundary of the playback environment.
  • the method may further comprise, for each speaker feed, determining a second combined gain depending on the gains of those virtual audio objects that lie on said boundary.
  • the first and second combined gains may be normalized.
  • the method may yet further comprise, for each speaker feed, determining a resulting gain for the plurality of virtual audio objects based on the first combined gain, the second combined gain, and a fade-out factor indicative of the relative importance of the first combined gain and the second combined gain.
  • the fade-out factor may depend on the three-dimensional extent (e.g., size) of the audio object and the location of the audio object.
  • the fade-out factor may depend on a fraction of the overall extent (e.g., of the overall three-dimensional volume) of the audio object that is within the boundary of the playback environment.
  • the method may further comprise, for each speaker feed, determining a final gain based on the resulting gain for the plurality of virtual audio objects, a respective gain for the audio object, and a cross-fade factor depending on the three-dimensional extent (e.g. size) of the audio object.
  • the associated metadata may indicate a first three-dimensional extent (e.g., size) of the audio object in a spherical coordinate system by respective ranges of values for a radius, an azimuth angle, and an elevation angle.
  • the method may further comprise determining a second three-dimensional extent (e.g., size) in a Cartesian coordinate system as dimensions of a cuboid that circumscribes the part of a sphere that is defined by said respective ranges of the values for the radius, the azimuth angle, and the elevation angle.
  • the method may yet further comprise using the second three-dimensional extent as the three-dimensional extent of the audio object.
  • the associated metadata may further indicate a measure of a fraction of the audio object that is to be rendered isotropically (e.g., from all directions with equal powers) with respect to an intended listener's position in the playback environment.
  • the method may further comprise creating an additional audio object at a center of the playback environment and assigning a three-dimensional extent (e.g. size) to the additional audio object such that a three-dimensional volume defined by the three-dimensional extent of the additional audio object fills out the entire playback environment.
  • the method may further comprise determining respective overall weight factors for the audio object and the additional audio object based on the measure of said fraction.
  • the method may yet further comprise rendering the audio object and the additional audio object, weighted by their respective overall weight factors, to the one or more speaker feeds in accordance with their respective three-dimensional extents.
  • Each speaker feed may be obtained by summing respective contributions from the audio object and the additional audio object.
  • the proposed method provides for perceptually appealing delocalization of part or all of an audio object.
  • the proposed method enables to achieve diffuseness of the audio object regardless of actual speaker layout of the reproduction environment.
  • diffuseness can be realized in an efficient manner, essentially without introducing new components/modules into a renderer for performing the proposed method.
  • the method may further comprise applying decorrelation to the contribution from the additional audio object to the one or more speaker feeds
  • renderers e.g., rendering apparatus
  • Such rendering apparatus may be configured to perform the methods described in the present document and/or may comprise respective modules (or blocks, units) for performing one or more of the processing steps of the methods described in the present document. Any statements made above with respect to such methods are understood to likewise apply to apparatus for rendering input audio for playback in a playback environment.
  • an apparatus for rendering input audio for playback in a playback environment.
  • the input audio may include at least one audio object and associated metadata.
  • the associated metadata may indicate at least a location (e.g., position) of the audio object.
  • the apparatus may comprise a metadata processing unit (e.g., a metadata pre-processor).
  • the metadata processing unit may be configured to create two additional audio objects associated with the audio object such that respective locations of the two additional audio objects are evenly spaced from the location of the audio object, on opposite sides of the location of the audio object when seen from an intended listener's position in the playback environment.
  • the metadata processing unit may be further configured to determine respective weight factors for application to the audio object and the two additional audio objects.
  • the apparatus may further comprise a rendering unit configured to render the audio object and the two additional audio objects to one or more speaker feeds in accordance with the determined weight factors.
  • the rendering unit may comprise a panning unit (e.g., point panner) and may further comprise a mixer.
  • the associated metadata may further indicate a distance measure indicative of a distance between the two additional audio objects.
  • the associated metadata may further indicate a measure of relative importance of the two additional audio objects compared to the audio object.
  • the weight factors may be determined based on said measure of relative importance.
  • the metadata processing unit may be further configured to normalize the weight factors based on said distance measure.
  • the weight factors may be normalized such that a sum of equal powers of the normalized weight factors is equal to a predetermined value.
  • An exponent of the normalized weight factors in said sum may be determined based on the distance measure (e.g., the metadata processing unit may be configured to determine said exponent based on the distance measure).
  • normalization of the weight factors may be performed on a sub-band basis, in dependence on frequency.
  • the rendering unit may be further configured to determine a set of rendering gains for mapping the audio object and the two additional audio objects to the one or more speaker feeds.
  • the rendering unit may be yet further configured to normalize the rendering gains based on said distance measure.
  • the rendering gains may be normalized such that a sum of equal powers of the normalized rendering gains for all of the one or more speaker feeds and for all of the audio objects and the two additional audio objects is equal to a predetermined value.
  • An exponent of the normalized rendering gains in said sum may be determined based on said distance measure (e.g., the metadata processing unit may be configured to determine said exponent based on the distance measure).
  • normalization of the rendering gains may be performed on a sub-band basis, in dependence on frequency.
  • an apparatus for rendering input audio for playback in a playback environment.
  • the input audio may include at least one audio object and associated metadata.
  • the associated metadata may indicate at least a location (e.g., position) of the at least one audio object and a three-dimensional extent (e.g., size) of the at least one audio object.
  • the apparatus may comprise a rendering unit for rendering the audio object to one or more speaker feeds in accordance with its three-dimensional extent.
  • the rendering unit may be configured to determine locations of a plurality of virtual audio objects within a three-dimensional volume defined by the location of the audio object and its three-dimensional extent.
  • the rendering unit may be further configured to for each virtual audio object, determine a weight factor that specifies the relative importance of the respective virtual audio object.
  • the rendering unit may be further configured to render the audio object and the plurality of virtual audio objects to the one or more speaker feeds in accordance with the determined weight factors.
  • the rendering unit may comprise a panning unit (e.g., extent panner, or size panner) and may further comprise a mixer.
  • the rendering unit may be further configured to, for each virtual audio object and for each of the one or more speaker feeds, determine a gain for mapping the respective virtual audio object to the respective speaker feed.
  • the rendering unit may be yet further configured to, for each virtual object and for each of the one or more speaker feeds, scale the respective gain with the weight factor of the respective virtual audio object.
  • the rendering unit may be further configured to, for each speaker feed, determine a first combined gain depending on the gains of those virtual audio objects that lie within a boundary of the playback environment.
  • the rendering unit may be further configured to, for each speaker feed, determine a second combined gain depending on the gains of those virtual audio objects that lie on said boundary.
  • the rendering unit may be yet further configured to, for each speaker feed, determine a resulting gain for the plurality of virtual audio objects based on the first combined gain, the second combined gain, and a fade-out factor indicative of the relative importance of the first combined gain and the second combined gain.
  • the rendering unit may be further configured to, for each speaker feed, determine a final gain based on the resulting gain for the plurality of virtual audio objects, a respective gain for the audio object, and a cross-fade factor depending on the three-dimensional extent (e.g., size) of the audio object.
  • the associated metadata may indicate a first three-dimensional extent (e.g., size) of the audio object in a spherical coordinate system by respective ranges of values for a radius, an azimuth angle, and an elevation angle.
  • the apparatus may further comprise a metadata processing unit (e.g., a metadata pre-processor) configured to determine a second three-dimensional extent (e.g., size) in a Cartesian coordinate system as dimensions of a cuboid that circumscribes the part of a sphere that is defined by said respective ranges of the values for the radius, the azimuth angle, and the elevation angle.
  • the rendering unit may be configured to use the second three-dimensional extent as the three-dimensional extent of the audio object.
  • the associated metadata may further indicate a measure of a fraction of the audio object that is to be rendered isotropically with respect to an intended listener's position in the playback environment.
  • the apparatus may further comprise a metadata processing unit (e.g., a metadata pre-processor) configured to create an additional audio object at a center of the playback environment and assigning a three-dimensional extent (e.g., size) to the additional audio object such that a three-dimensional volume defined by the three-dimensional extent of the additional audio object fills out the entire playback environment.
  • the metadata processing unit may be further configured to determine respective overall weight factors for the audio object and the additional audio object based on the measure of said fraction.
  • the metadata processing unit may be yet further configured to output the audio object and the additional audio object, weighted by their respective overall weight factors, to the rendering unit for rendering the audio object and the additional audio object to the one or more speaker feeds in accordance with their respective three-dimensional extents.
  • the rendering unit may be configured to obtain each speaker feed by summing respective contributions from the audio object and the additional audio object.
  • the rendering unit may be further configured to apply decorrelation to the contribution from the additional audio object to the one or more speaker feeds.
  • a software program is described.
  • the software program may be adapted for execution on a processor and for performing the method steps outlined in the present document when carried out on a computing device.
  • the storage medium may comprise a software program adapted for execution on a processor and for performing the method steps outlined in the present document when carried out on a computing device.
  • the computer program may comprise executable instructions for performing the method steps outlined in the present document when executed on a computer.
  • the present document describes several schemes (methods) and corresponding apparatus for addressing the above issues. These schemes, directed to rendering of audio objects with extent, diffusion, and divergence (e.g., audio objects having extent metadata, diffuseness metadata, and divergence metadata), respectively, may be employed individually or in conjunction with each other.
  • extent, diffusion, and divergence e.g., audio objects having extent metadata, diffuseness metadata, and divergence metadata
  • the renderer e.g., baseline renderer
  • the renderer may be suitable to (see, e.g., ITU-R Document 6C/511-E (annex 10) to chairman's report for continuation of the RG):
  • the renderer specifies algorithms for rendering a subset of ADM and is not meant as a complete product.
  • the algorithms and architecture described in the baseline renderer is designed to be easily extended to completely cover the ADM specification.
  • the renderer described in this document is not to be understood to be limited to ADM and may likewise be applied to other specifications of object-based audio content.
  • ADM allows for the grouping of audio elements into programs and can capture multiple programs in a single ADM tree. This ability to capture multiple ways of compositing audio primarily addresses content management aspects for the broadcast ecosystem, and has little influence on how individual elements are rendered. With this in mind the renderer does not address the logic components required to select the input audio to the rendering process, and assumes a production system using the renderer would provide this functionality.
  • the ADM supports several formats to represent a spatial audio description (SAD).
  • SAD spatial audio description
  • a fundamental component of the SAD is the means to specify the nominal locations of sounds. This requires establishing a frame of reference.
  • a frame of reference In order to specify locations in a space (e.g., in a playback environment), a frame of reference (FoR) is required.
  • FoR frame of reference
  • Fig. 1 and Fig. 2 schematically illustrate examples of an egocentric frame of reference and an allocentric frame of reference, respectively.
  • the egocentric location is 56° azimuth and 2m from the listener.
  • the allocentric location is 1/4 of the way from left to right wall, 1/3 of the way from front to back wall.
  • An egocentric reference is commonly used for the study and description of perception; the underlying physiological and neurological processes of acquisition and coding most directly relate to the egocentric reference.
  • an egocentric representation is appropriate in scenarios when the sound scene is captured from a single point (such as with an Ambisonics microphone array, or other "scene-based" models), or when the sound scene is intended for a single, isolated listener (such as listening to music over headphones).
  • a spherical coordinate system is often well suited for specifying locations when using an egocentric frame of reference.
  • most scene-based spatial audio descriptions are based on a decomposition that utilizes circular or spherical coordinates, as in the example of Fig.
  • FIG. 3 which illustrates a simplified single-band in-phase B-format decoder for a square loudspeaker layout.
  • Fig. 3 illustrates a naive example which does not fulfil the psychoacoustic criteria for Ambisonics decoding.
  • the ADM supports scene-based, egocentric representations and spherical coordinates.
  • An allocentric reference is well suited for audio scene descriptions that are independent of a single observer position, and when the relationship between elements in the playback environment is of interest.
  • a rectangular or Cartesian coordinate system is often used for specifying locations when using an allocentric frame of reference.
  • the ADM supports specifying location using an allocentric frame of reference, and Cartesian coordinates.
  • All direct speaker and dynamic object channels are accompanied by metadata (associated metadata) that specifies at least a location.
  • Spherical coordinates indicate the location of an object, as a direction of arrival, in terms of azimuth and elevation, relative to one listening position.
  • a (relative) distance parameter e.g., in the range 0...1
  • Cartesian coordinates indicate the location of an object, as a position relative to a normalized listening space, in terms of X, Y and Z coordinates of a unit cube (the "Cartesian cube", defined by
  • the X index corresponds to the left-right dimension;
  • the Y index corresponds to the rear-front dimension; and
  • the cornerstones for the allocentric model are the corners of the unit cube and the loudspeakers that define these corners.
  • the ADM supports both egocentric spherical coordinates and allocentric Cartesian coordinates.
  • the panning function defined in section 3.2.1 "Rendering Point Objects" below may be based on Cartesian coordinates to specify the location of audio sources in space.
  • a translation is required.
  • a change of coordinate systems could be achieved using simple trigonometry.
  • translation of the frame of reference is more complicated, and requires that the space be "warped" to preserve the artistic intent.
  • an allocentric frame of reference is constructed based on key channel locations. That is, the object location is defined relative to landmark channels. This ensures that the relative location of channels and objects remains consistent, and that the most important spatial aspects of an audio program (from the mixer's perspective) are preserved. For example, an object that moves across the front sound stage from "full left” to “full right” will do so in every playback environment.
  • mapping function from spherical to Cartesian, the following principles will generally be adhered to:
  • the author When an audio scene is authored, the author will generally have a specific playback environment in mind. This will generally coincide with the playback environment used by the author during the content-creation process.
  • the playback environment that is deemed, by the author, to be preferred for playback of the audio file will be referred to as the reference rendering environment.
  • Map SC Map SC Az El R Flag
  • Map SC () function The definition of the Map SC () function can be found in section 3.3.2 "Object and Channel Location Transformations" below.
  • the renderer (e.g., baseline renderer) supports a subset of the formats and features specified by ADM. In limiting the ADM input format the focus has been on defining new Object, DirectSpeaker and HOA behavior as these represent the core of the new experiences enabled by ADM. Matrix content and Binaural content are not addressed by the baseline renderer.
  • structures in ADM aimed at supporting the cataloguing and compositing of multiple elements are also set aside in the baseline renderer, in favor of describing the rendering process for the programme elements themselves.
  • the ADM input content and format must conform to the reduced UML model illustrated in Fig. 4 , which an example of an input ADM format.
  • This subset of the full model is sufficient to express all the features supported in the renderer (e.g., baseline renderer). If the input metadata contains objects and references between objects beyond those depicted in the UML diagram above, such metadata shall be ignored by the renderer.
  • the renderer will only attempt to parse the first audioPackFormatIDRef that it encounters inside an audioObject. Therefore, it is recommended that an audioObject only reference a single audioPackFormat.
  • the renderer will also assume that audioObjects persist throughout the duration of the audioProgramme (i.e., audioObject start time will be assumed to be 0 and duration attributes shall be ignored). This implies that the list of Track Numbers in the BWF File .chna chunk must be non-repeating, as shown in Fig. 4 .
  • a common audioPackFormat reference in an audioObject instance shall be interpreted by the renderer to indicate the speaker layout that was used during content creation. Only one reference to an audioPackFormat from the common definitions is therefore allowed to exist in the file. However, multiple instances of non-common audioPackFormats may be present.
  • an audioStreamFormat instance may refer to either an audioPackFormat or audioChannelFormat instance, but not both.
  • an audioStreamFormat instance refers to audioPackFormat, but not audioTrackFormat
  • the renderer loses the ability to link an audio track to the specific audioChannelFormat instance containing its metadata. Therefore, while audioPackFormat instances may be present in the .xml chunk, they shall not be referenced from audioStreamFormat instances.
  • the renderer shall associate audio tracks to their corresponding audioPackFormat (if any) through the audioPackFormat reference in the .chna chunk.
  • the output from the renderer may be passed through a B-chain for reproduction in a studio environment.
  • the output could be captured as new ADM content, however before writing to a file the signal overload protection (i.e., peak limiting) which the B-chain would provide in a studio environment may need to be simulated in software.
  • the output is captured as ADM, it is recommended that it should only contain common audioObjectIDs, matching the waveform information to the BS.2051-0 speaker configuration specified.
  • Fig. 5 illustrates the reduced model which the output of the renderer may conform to as an example of the output ADM format. This output may be ready for presentation to a reproduction system which conforms to what is specified in Recommendation ITU-R BS.1116. It is recommended that reproduction systems used to evaluate rendered ADM content are calibrated to provide level and time alignment within 0.25 dB and 100 ⁇ s respectively at the listening position.
  • FIG. 6 An example of the system architecture of the renderer 600 is schematically illustrated in Fig. 6 .
  • the renderer 600 is constructed in three major blocks:
  • the ADM reader 300 parses ADM content 10 to extract the metadata 25 into an internal representation and aligns the metadata 25 with associated audio data 20 to feed, in blocks, to the rendering engines.
  • the ADM reader 300 also validates the metadata 25 to ensure a consistent and complete set of metadata is present, for example the ADM reader 300 ensures all components of an HOA scene are present before attempting to render the scene.
  • the scene renderer 200 consumes scene-based channels and renders them to the desired speaker layout. Details of the scene formats supported by the renderer and the rendering methods are detailed in section 4 "Scene Renderer" below.
  • the object and channel renderer 100 consumes DirectSpeaker channels and Object channels and renders them to the desired speaker layout. Details of the metadata features supported by the baseline renderer and the rendering methods are detailed in section 3 "Channel and Object Renderer" below.
  • the speaker renders created by the two render stages are mixed (summed) at mixing stage 400 and the resulting speaker feeds are passed to the reproduction system 500.
  • the renderer algorithm adds no latency to the audio signal path.
  • the maximum delay between the time when the metadata is presented to the rendering algorithm, and when its effect is represented on the output may be 64 samples.
  • the delay incurred between the control surface and the renderer depends on the hardware/software integration encapsulating the baseline renderer, and the delay incurred after the output is updated before it is reproduced by the speakers depends on the latency of the B-chain processing and the software/hardware interfaces linking the system to the speakers. These delays should be minimized when integrating the renderer into a studio environment.
  • the renderer algorithm (e.g., baseline renderer algorithm) described in this document supports ADM content with homogenous sampling rates. It is recommended that content with mixed sampling rates be converted to the highest common sampling rate and aligned as a pre-step to the rendering stage in order to avoid timing complexities introduced when combining sample rate conversion and rendering into a single stage of processing.
  • Updates to the mixing matrices are not limited to the 32 sample boundaries and may be updated on a per-sample basis-section 3.4 "Ramping Mixer" below details how the mixing matrices may be updated and applied in the channel and object renderer.
  • the object and channel renderer 100 comprises a metadata pre-processor (embodying an example of a metadata processing unit) 110, a source panner 120, a ramping mixer 130, a diffuse ramping mixer 140, a speaker decorrelator 150, and a mixing stage 160.
  • the object and channel renderer 100 may receive metadata (e.g., ADM metadata) 25, audio data (e.g., PCM audio data) 20, and optionally a speaker layout 30 of the reproduction environment as inputs.
  • the object and channel renderer 100 may output one or more speaker feeds 50.
  • the metadata pre-processor 110 converts existing direct speaker and dynamic object metadata, implementing the channelLock, divergence and screenEdgeLock features. It also takes the speaker layout 30 and implements the zoneExclusion metadata features to create a virtual room.
  • the Source Panner 120 takes the new virtual source metadata, and virtual room metadata and pans the sources to create speaker gains, and diffuse speaker gains.
  • the source panner 120 may implement the extent and diffuseness features respectively described in section 3.2.2 “Rendering Object Locations with Extents” and section 3.2.5 “Diffuse” below.
  • the Ramping Mixer 130 mixes the audio data 20 with the speaker gains to create the speaker feeds 50.
  • the ramping mixer 130 may implement the jumpPosition feature. There are two ramping mixer paths. The first path implements the direct speaker feeds, while the second path implements the diffuse speaker feeds.
  • the per-object gains are speaker independent, so the diffuse ramping mixer 140 produces a mono downmix. This downmix feeds the Speaker Decorrelator 150 where the diffuse speaker dependent gains are applied. Finally the two paths are mixed together at the mixing stage 160 to produce the final speaker feeds.
  • the source panner 120 and the ramping mixer(s) 130, 140, and optionally the speaker decorrelator 150 may be said to form a rendering unit.
  • the source panner 120 comprises a point panner 810, an extent panner (size panner) 820 and a diffusion block (diffusion unit) 830.
  • the source panner 120 may receive the virtual sources 812 and virtual rooms 814 as inputs.
  • Outputs 832, 834, 836 of the source panner 120 may be provided to the ramping mixer 130, the diffuse ramping mixer 140, and the speaker decorrelator 150, respectively.
  • the source panner 120 receives the pre-processed objects, and virtual room metadata from the metadata pre-processor 110, and first pans them to speaker gains, assuming no extent or diffusion using the point panner 810. The resulting speaker gains are then processed by the extent panner 820, adding source extent and producing a new set of speaker gains. Finally these speaker gains pass to the diffusion block 830.
  • the diffusion block 830 maps these gains to speaker gains for the ramping mixer 130, the diffuse ramping mixer 140 and the speaker decorrelator 150.
  • the purpose of the point panner 810 is to calculate a gain coefficient for each speaker in the output speaker layout, given an object position.
  • the point panning algorithm may consist of a 3D extension of the 'dual-balance' panner concept that is widely used in 5.1- and 7.1-channel surround sound production.
  • One of the main requirements of the point panner 810 is that it is able to create the impression of an auditory event at any point inside the room.
  • the advantage of using this approach is that it provides a logical extension to the current surround sound production tools used today.
  • the inputs to the point panner 810 comprise (e.g., consist of) an object's position [ p ox ,p oy ,p oz ] and the positions of the output speakers, all in Cartesian coordinates, for example.
  • [ p sx ( j ), p sy ( j ) ,p sz ( j )] denote the position of the j -th speaker.
  • N denote the number of speakers in the layout.
  • the point panner 810 requires that the following conditions are satisfied in order to be able to accurately place a phantom image of the object anywhere in the room (i.e., in the playback environment):
  • the purpose of the extent panner 820 is to calculate a gain coefficient for each speaker in the output speaker layout, given an object position and object extent (e.g., object size).
  • object extent e.g., object size
  • the intention of extent is to make the object appear larger so that when the extent is at the maximum the object fills the room, while when it is set to zero the object is rendered as a point object.
  • the extent panner 820 considers a grid (e.g., a three-dimensional rectangular grid) of many virtual sources in the room. Each virtual source fires speakers exactly in the same way any object rendered with the point panner 810 would.
  • the extent panner 820 when given an object position and object extent, determines which (and how many) of those virtual sources will contribute. That is, candidates for the contributing virtual sources may be arranged in a grid (e.g., a three-dimensional rectangular grid) across the playback environment (e.g., room).
  • Fig. 24 is a flowchart schematically illustrating an example of a method (e.g., algorithm) for rendering object locations with extents as an example for a method of rendering input audio for playback in a playback environment.
  • the input audio includes at least one audio object and associated metadata.
  • the associated metadata indicates (e.g., specifies) at least a location (e.g., position) of the at least one audio object and a three-dimensional extent (e.g., size) of the at least one audio object.
  • the method comprises rendering the audio object to one or more speaker feeds in accordance with its three-dimensional extent.
  • step S2410 locations of a plurality of virtual audio objects (virtual sources) within a three-dimensional volume defined by the location of the audio object and its three-dimensional extent are determined. Determining said locations may involve imposing a respective minimum extent for the audio object in each of the three dimensions (e.g., ⁇ x,y,z ⁇ or ⁇ ⁇ , ⁇ ,r ⁇ ). Further, said determining may involve selecting a subset of locations of (active) virtual audio objects among a predetermined set of fixed potential locations of virtual audio objects in the reproduction environment. The fixed potential positions may be arranged in a three-dimensional grid, as explained below.
  • a weight factor is determined for each virtual audio object that specifies the relative importance (e.g., relative weight) of the respective virtual audio object.
  • the "relative importance" dealt with in this section is not to be confused with the metadata feature relating to ⁇ importance> and ⁇ obj_importance> described in section 3.3.9 "Importance” below.
  • the audio object and the plurality of virtual audio objects are rendered to the one or more speaker feeds in accordance with the determined weight factors.
  • Performing step S2430 results in a gain coefficient for each of the one or more speaker feeds that may be applied to (e.g., mixed with) the audio data for the audio object.
  • the audio data for the audio object may be the audio data (e.g., audio signal) of the original audio object.
  • Step S2430 may comprise the following further steps:
  • An apparatus for rendering input audio for playback in a playback environment (e.g., for performing the method of Fig. 24 ) may comprise a rendering unit.
  • the rendering unit may comprise a panning unit and a mixer (e.g., the source panner 120 and either or both of the ramping mixer(s) 130, 140).
  • Step S2410, step S2420 and step S2430 may be performed by the rendering unit.
  • the method may comprise steps S2510 and S2520 illustrated in the flowchart of Fig. 25 and steps S2610 to S2640 illustrated in the flowchart of Fig. 26 .
  • Said steps may be said to be sub-steps of step S2430. Accordingly, steps S2510 and S2520 as well as steps S2610 to S2640 may be performed by the aforementioned rendering unit.
  • a gain is determined, for each virtual audio object and for each of the one or more speaker feeds, for mapping the respective virtual audio object to the respective speaker feed. These gains may be the point gains referred to above.
  • respective gains determined at step S2510 are scaled, for each virtual object and for each of the one or more speaker feeds, with the weight factor of the respective virtual audio object.
  • a first combined gain is determined for each speaker feed depending on the gains of those virtual audio objects that lie within a boundary of the playback environment (e.g., room).
  • the first combined gains determined at step S2610 may be the inside extent gains (one for each speaker feed) referred to above.
  • a second combined gain is determined for each speaker feed depending on the gains of those virtual audio objects that lie on said boundary.
  • the second combined gains determined at step S2620 may be the boundary extent gains (one for each speaker feed) referred to above.
  • a resulting gain for the plurality of virtual audio objects is determined for each speaker feed based on the first combined gain, the second combined gain, and a fade-out factor indicative of the relative importance of the first combined gain and the second combined gain.
  • the resulting gains determined at step S2630 may be the final extent gains (one for each speaker feed) referred to above.
  • the fade-out factor may depend on the three-dimensional extent of the audio object and the location of the audio object. For example, the fade-out factor may depend on a fraction of the overall extent of the audio object that is within the boundary of the playback environment (e.g., the fraction of the overall three-dimensional volume of the audio object that is that is within the boundary of the playback environment).
  • the first and second combined gains may be normalized before performing step S2630.
  • a final gain is determined for each speaker feed based on the resulting gain for the plurality of virtual audio objects, a respective gain for the audio object, and a cross-fade factor depending on the three-dimensional extent of the audio object. This may relate to combining the final extent gains with the point gains for the object.
  • the extent value (e.g., size value) may be scaled up to a larger range. That is, the first step may be to scale up the ADM extent value to a larger range.
  • the user is exposed to extent values s ⁇ [0,1], which may be mapped into the actual extent used by the algorithm to the range [0, 5.6].
  • the mapping may be done by a piecewise linear function, for example a piecewise linear function defined by the value pairs (0, 0), (0.2, 0.6), (0.5, 2.0), (0.75, 3.6), (1, 5.6), as shown in Fig. 9 .
  • the maximum value of 5.6 ensures that when extent is set to maximum, it truly occupies the whole room.
  • the variables s x ⁇ , s y ⁇ , s z ⁇ refer to the extent values after conversion. Notably, each of the three dimensions of the extent may be independently controlled when employing the presently described method.
  • the renderer may clip (i.e., increase) small, non-zero extent values to respective minimum values as needed. That is, determining said locations at step S2410 may involve imposing a respective minimum extent for the audio object in each of the three dimensions (e.g., ⁇ x,y,z ⁇ or ⁇ ⁇ , ⁇ ,r ⁇ ).
  • These restricted values s x ,s y ,s z may be used throughout the algorithm, except for the computation of effective size s eff below, which uses the unrestricted values s x ⁇ , s y ⁇ , s z ⁇ .
  • the grid of virtual sources referred to in step S2410 may be defined as a static rectangular uniform grid of N x ⁇ N y ⁇ N z points.
  • the grid may span the range of positions [-1, 1] in each dimension. That is, the grid may span the entire reproduction environment (e.g., room).
  • the range of virtual sources in the z dimension may be limited to [0, 1], and the recommended value of N z is 8.
  • the notation ( x s ,y s ,z s ) will be used to denote the possible coordinates of the virtual sources.
  • the object position and extent ( x o ,y o ,z o ,s x ,s y ,s z ) may be used to calculate a set of weights that determine how much each virtual source will contribute to the final gains. Accordingly, the set of weights may be determined based on the object position (location) and extent. This calculation may be performed at step S2420.
  • the weights for each virtual source are denoted w ( x s , y s ,z s , x o , y o , z o ,s x ,s y ,s z ) and may be used to scale the gains (e.g., point gains) for each virtual source at step S2520.
  • the gains (e.g., point gains) may have been determined at step S2510.
  • Virtual sources with zero weight may be considered as not having been selected at step S2410, i.e., their locations are not among the locations determined at step S2410.
  • the extent-dependent exponent p controls the smoothness of the gains across loudspeakers.
  • the extent-dependent exponent p may be determined (e.g., calculated) as follows: First sort s x ⁇ s y ⁇ s z ⁇ in descending order, and label the resulting ordered triad as ⁇ s 1 , s 2 , s 3 ⁇ .
  • an effective extent e.g., effective size
  • s eff s x ⁇ , for example.
  • gains e.g., point gains
  • g j point x y z g j point x ⁇ g j point y ⁇ g j point z
  • the weight function can also treat each axis separately and the whole extent computation simplifies.
  • the chosen weight functions may look like something between circles and squares (or spheres and cubes, in 3D).
  • ITU ITU
  • a normalization step may be applied to g j inside , i.e., the first combined gains may be normalized.
  • dim 1 for ITU-R BS.2051-0 System A
  • dim 2 for System B
  • dim 4 for Systems E and H
  • dim 3 otherwise in the calculations below.
  • boundary extent gains g j bound may be determined depending on the gains of those virtual sources that lie on the boundary of the reproduction environment (e.g., room).
  • a normalization step may be applied to the boundary extent gains g j bound , i.e., the second combined gains may be normalized.
  • the boundary extent gains (second combined gains) may now be combined with the inside extent gains (first combined gains).
  • the fade-out factor may indicate a relative importance of the inside extent gains and boundary extent gains.
  • the fade-out factor may depend on the location and extent of the audio object. Combination of the inside extent gains and boundary extent gains may be performed at step S2630.
  • the fade-out factor may be determined such that, as part of the sized object starts moving outside the room, all virtual sources inside the object start fading out, except for those at the boundaries.
  • d bound may be the minimum distance to a boundary.
  • a normalization step may be applied to the final extent gains g j size (resulting gains).
  • the extent contributions may then be combined with the gains for the audio object (e.g., point gains of the audio object-assuming zero extent for the audio object), and a crossfade between them may be applied as a function of extent.
  • Combination of the final extent gains and the gains of the audio object may be performed at step S2640 and may result in a set of final gains (total gains), one for each speaker feed.
  • a last normalization may be applied to the final gains.
  • the final gains G j s may be provided to the diffusion block 830 if present, or otherwise directly to the ramping mixer 130.
  • the final gains may be the outcome of the rendering at step S2430.
  • any associated extent metadata given in spherical coordinates i.e., width, height, and depth ADM parameters, in degrees
  • Cartesian extent metadata i.e., X-width, Y-width, Z-width ADM parameters, e.g., in the range [0, 1]
  • Extent metadata may be converted from spherical to Cartesian coordinates by finding the size of a cuboid that encompasses the angular extents.
  • the Cartesian cuboid can be found by determining the extremities in each dimension of the shape described by the spherical extent angles and depth.
  • Two examples are shown in Fig. 10A and Fig. 10B , limited to the x and y plane, for simplicity.
  • Fig. 10A illustrates the case of an extent defined by acute angles
  • Fig. 10B illustrates the case of an extent defined by obtuse angles.
  • the distance will be halved to match the range of extent given in the Cartesian coordinate system and these parameters can then be used by the extent panner to render an object.
  • a method for converting the extent from spherical coordinates to Cartesian coordinates may comprise the steps illustrated in the flowchart of Fig. 27 .
  • This method is applicable to any audio object whose associated metadata indicates a first three-dimensional extent (e.g., size) of the audio object in a spherical coordinate system by respective ranges of values for a radius, an azimuth angle, and an elevation angle.
  • a second three-dimensional extent (e.g., size) in a Cartesian coordinate system is determined as dimensions (e.g., lengths along the X, Y, and Z coordinate axes, i.e., X-width, Y-width, and Z-width) of a cuboid that circumscribes the part of a sphere that is defined by said respective ranges of the values for the radius, the azimuth angle, and the elevation angle.
  • the second three-dimensional extent is used as the three-dimensional extent of the audio object in the above method for rendering object locations with extents as an example for a method of rendering input audio for playback in a playback environment.
  • the aforementioned apparatus for rendering input audio for playback in a playback environment (e.g., for performing the method of Fig. 24 ) may further comprise a metadata processing unit (e.g., metadata pre-processor 110).
  • Step S2710 may be performed by the metadata processing unit.
  • Step S2720 may be performed by the rendering unit.
  • the following pseudocode defines an example of an algorithm for calculating X-width, Y-width, and Z-width from spherical width, height, and depth :
  • the renderer takes the following strategy to render channel-based content:
  • the position ranges specified in the Tables 1 to 4 below were derived from the ranges specified in ITU-R BS.2051-0 for Sound Systems B, F, G, and H. Because the specification gives no ranges to the speakers in Systems A, C, D, and E, the ranges for the System B surround speakers are used for all these systems, but the upper-layer speakers in systems C, D, and E are given no ranges (i.e., they will always be panned to the position specified in the metadata). In the case of System F, the M+/-90 and M+/-135 speakers overlap in azimuth range, so a boundary between them was set at the midpoint of +/-112.5 degrees azimuth.
  • the position adjustment strategy defined herein ensures that channel-based content that was authored using a Sound System conformant to ITU-R BS.2051-0 will be sent entirely to the correct loudspeaker when rendered to the same system, even when there is not an exact match between the speaker positions used during content creation and during playback (because different positions were chosen within the ranges allowed by the BS.2051 specification).
  • channel-based content will still be sent to a single loudspeaker if the position specified in metadata is within the allowed range for a speaker in the output layout. Otherwise, in order to preserve the approximate position of the sound during content creation, the channel-based content will be panned to the location specified in its metadata.
  • LFE Low Frequency Effects
  • sub-woofer speaker feeds The distinction between Low Frequency Effects (LFE) channels and sub-woofer speaker feeds is subtle, and understanding this with respect to how the renderer (e.g., baseline renderer) treats LFE content requires some clarification.
  • Recommendation ITU-R BS.775-3 has more detail and recommended use of the LFE channel.
  • Sub-woofer speakers are specialized speakers in a reproduction system with the purpose of reproducing low-frequency signals or content. They may require other signal processing (e.g. bass management, overload protection) in the B-chain of a reproduction system. As such the renderer (e.g., baseline renderer) does not include any effort to perform these functions.
  • the renderer e.g., baseline renderer
  • ITU-R BS.2051-0 includes speakers labelled as LFE, which are intended to carry the audio expected to be output by sub-woofers.
  • ADM may contain DirectSpeaker content labelled as LFE.
  • the baseline renderer ensures input LFE content is directed to the LFE output channels, with minimal processing. The following cases are described explicitly:
  • the renderer shall consider LFE input content to be either any common audioChannelFormat with an ID equal to AC_00010004 (LFE), AC_00010020 (LFEL), or AC_00010021 (LFER), or any input audioChannelFormat of type DirectSpeakers with an active audioBlockFormat sub-element containing 'LFE' as the first three characters in its speakerLabel element.
  • LFE common audioChannelFormat with an ID equal to AC_00010004
  • LFEL AC_00010020
  • LFER AC_00010021
  • the associated metadata of the audio object may further or alternatively indicate (e.g., specify) a degree of diffuseness for the audio object.
  • the associated metadata may indicate a measure of a fraction of the audio object that is to be rendered isotropically (i.e., with equal energies from all directions) with respect to the intended listener's position in the playback environment.
  • the degree of diffuseness (or equivalently, said measure of a fraction) may be indicated by a diffuseness parameter p , for example ranging from 0 (no diffuseness, full directionality) to 1 (full diffuseness, no directionality).
  • may be used to determine the fraction of signal power sent to the direct path and to the decorrelated paths.
  • 1, an object is mixed completely to the diffuse path.
  • objects are processed by the extent panner 820 to produce the direct gains G ij s .
  • the diffuse ramping mixer 140 pans a fraction of the audio object (the fraction being determined by the diffuseness of the audio object) to the center of the reproduction environment (e.g., room). This fraction may be considered as an additional audio object. Further, the ramping mixer assigns an extent (e.g., three-dimensional size) to the additional object such that the three-dimensional volume of the additional object (located at the center of the reproduction environment) fills the entire reproduction environment.
  • an extent e.g., three-dimensional size
  • FIG. 28 A summary of an example of a method for rendering an audio object with diffuseness is illustrated in the flowchart of Fig. 28 .
  • the method may comprise the steps of Fig. 28 either as stand-alone or in combination with the method illustrated in Fig. 24 , Fig. 25 , and Fig. 26 .
  • an additional audio object is created at a center of the playback environment (e.g., room). Further, an extent (e.g., three-dimensional size) is assigned to the additional audio object such that a three-dimensional volume defined by the extent of the additional audio object fills out the entire playback environment.
  • respective overall weight factors are determined for the audio object and the additional audio object based on a measure of a fraction of the audio object that is to be rendered isotropically with respect to the intended listener's position in the playback environment. That is, said two overall weight factors may be determined based on the diffuseness of the audio object, e.g., based on the diffuseness parameter ⁇ .
  • the overall weight factor for the direct fraction (direct part) of the audio object may be given by 1 ⁇ ⁇
  • the overall weight factor for the diffuse fraction (diffuse part) of the audio object i.e., for the additional audio object
  • the audio object and the additional audio object are rendered to the one or more speaker feeds in accordance with their respective three-dimensional extents. Rendering of an object in accordance with its extent may be performed as described above in section 3.2.2 "Rendering Object Locations with Extents", and may be performed by the size panner 820 in conjunction with the diffuse ramping mixer 140, for example.
  • the direct fraction of the audio object is rendered at its actual location with its actual extent.
  • the diffuse fraction of the audio object is rendered at the center of the room, with an extent chosen such that it fills the entire room.
  • the resulting gains for the diffuse fraction of the audio object may be determined beforehand, when initializing a new room configuration (reproduction environment).
  • Each speaker feed may be obtained by summing respective contributions from the direct and diffuse fractions of the audio object (i.e., from the audio object and the additional audio object).
  • decorrelation is applied to the contribution from the additional audio object to the one or more speaker feeds. That is, the contributions to the speaker feeds stemming from the additional audio object are decorrelated from each other.
  • An apparatus for rendering input audio for playback in a playback environment (e.g., for performing the method of Fig. 27 ) may comprise a metadata processing unit (e.g., metadata pre-processor 110) and a rendering unit.
  • the rendering unit may comprise a panning unit and a mixer (e.g., the source panner 120 and either or both of the ramping mixer(s) 130, 140), and optionally, a decorrelation unit (e.g., the speaker decorrelator 150).
  • Steps S2810 and S2820 may be performed by the metadata processing unit.
  • Steps S2830 and S2840 may be performed by the rendering unit.
  • the apparatus may be the further configured to perform the method of Fig. 24 (optionally, with the sub-steps illustrated in Fig. 25 and Fig. 26 ), and optionally, the method of Fig. 27 .
  • the metadata pre-processor 110 is the component that achieves this for the renderer by either reducing the number of speakers available for render or modifying the positional metadata.
  • Metadata features An example for the processing order of metadata (metadata features) is schematically illustrated in Fig. 11 .
  • metadata parameters are processed in a very specific order. Importance is processed first for efficiency reasons as it may result in fewer sources to process. screenEdgeLock and screenRef are mutually exclusive. zoneExclusion must happen prior to channelLock to prevent locking to speakers that will not be part of the panning layout. Finally divergence is placed after channelLock to allow the mixer to produce a phantom image that remains centered at the location of the locked channel.
  • the outputs of the Map SC () function will be the ( X,Y,Z ) values, as produced by the procedure above.
  • Map CS () converts an ( X,Y,Z ) position to ( ⁇ , ⁇ ,r ) and may be achieved through a step-by-step inversion of Map SC () .
  • zoneExclusion is an ADM metadata parameter that allows an object to specify a spatial region of speakers that should not be used to pan the object.
  • An audioChannelFormat of type "Objects” may include a set of "zoneExclusion” sub-elements to describe a set of cuboids. Speakers inside this set of cuboids shall not be used by the renderer to pan the object.
  • the metadata pre-processor 110 may handle zone exclusion by removing speakers from the virtual room layout that is generated for each object. Exclusion zones are applied to speakers before spherical speaker coordinates are transformed to Cartesian coordinates by the warping function described in section 3.3.2 "Object and Channel Location Transformations".
  • Step 1 For each of the N speakers in the virtual speaker layout, check if the speaker lies inside any of the M exclusion zone rectangular cuboids. If so, remove it from the layout by setting its mask value to zero.
  • the mask values will then be used by the point panner 810 to select which speakers are considered part of the output layout for the object, as described in section 3.2.1 "Rendering Point Objects”.
  • Step 2 ensures that the resulting speaker layout does not lead to undesired panning behavior. For example, consider the System F layout from ITU-R BS.2051, where only the M-90 speaker has been removed. If we then pan an object from the front right to the back right of the room, the panner will pan the object entirely to the left (speaker M+90) as the object crosses the middle of the room. To correct this, we also remove the M+90 speaker, and now the object renders correctly from front to back on the right side, by panning between the M-30 and M-135 speakers.
  • Gain metadata thus receives the same cross-fade defined by the objects jumpPosition metadata.
  • channelLock metadata is implemented inside the metadata pre-processor 110 component described in section 3.1 "Architecture". If the channelLock flag is set to 1 in an audioBlockFormat element contained by an audioChannelFormat instance of type Objects, the virtual source renderer component will modify the position sub-elements of the audioBlockFormat to ensure that the object's audio is panned entirely to a single output channel.
  • the optional maxDistance attribute controls whether the channelLock effect is applied to the object, based on the unweighted Euclidean distance between an object's position and the output speaker closest to it. If maxDistance is undefined, the renderer assumes a default value of infinity, meaning that the object always "snaps" to the closest speaker.
  • channelLock processing is performed after the object's position has been transformed into Cartesian coordinates, as described in section 3.3.2 “Object and Channel Location Transformations”.
  • the distances between the object and the speakers are calculated using the speaker positions after they have been transformed from spherical to Cartesian coordinates, as described in section 3.3.2 “Object and Channel Location Transformations”.
  • a weighted Euclidean distance measure has been designed to yield rectangular cuboid "lock” regions around each speaker in Cartesian space. Dividing the snap regions in this way improves the intuitiveness of the snap feature during content creation in a mixing studio, and is consistent with the allocentric rendering philosophy behind the point panner 810.
  • Channel Lock may be applied as follows:
  • the speakers 1 to N are pre-sorted as follows: center is always placed at the head of the list if it is present. The remaining speakers are then ordered first by decreasing z-value, then by increasing y-value and finally by increasing x-value, such that when there are multiple speakers with exactly the same weighted distance to the object, the object is locked to the speaker that is closest to the top-front-left of the room.
  • This section relates to a method for controlling constraints when rendering audio objects with divergence.
  • a power preserving pan is used to distribute a source to the left and right channels, based on the expectation that this power preserving pan will cause an acoustic summing in the room to create a source of the correct level at the correct location.
  • Section 9.6 of the ADM standard specifies a way to express the concept of divergence in metadata and provides what could be considered an obvious approach to phantom source panning in an effort to provide the same functionality as legacy mixing through objects.
  • One detail provided within the ADM specification is that in order to create a phantom image, a power preserving pan should be created between two virtual objects (additional audio objects) and an original audio object-as would be expected when using left and right speakers to create a phantom center channel. Needless to say, the phantom image to be created is located at the position of the original audio object.
  • Fig. 12 illustrates an example of two virtual objects (additional audio objects) 1220, 1230 that are provided for an (original) audio object 1210 for purposes of phantom source panning.
  • each virtual object 1220, 1230 is spaced from the audio object 1210 by an angular distance 1240.
  • the two virtual objects 1220, 1230 are spaced from each other by twice the angular distance 1240.
  • This angular distance 1240 may be referred to as an angle of divergence.
  • the first problem comes from the ability to specify the angle of divergence, and the second problem from how objects are rendered to speakers in an object audio renderer.
  • the freedom (e.g., in ADM) for object based divergence to specify an angle that dictates where the new pair of virtual objects are created relative to the desired phantom image location means that the new virtual objects can be located very close to the phantom location.
  • the location of these virtual objects close to the phantom location is analogous to placing speakers close together when rendering a phantom center-if this is realized in practice, a power preserving pan would result in inappropriate level of the phantom image (e.g., increased loudness), due to the coherent summation of the new sources.
  • Section 9.6 of the ADM standard provides a definition of the divergence metadata's behavior in terms of two parameters: objectDivergence (0, 1) and azimuthRange. While this is not the only way such a behavior could be described, it will be used to help explain the context and formulation of this invention.
  • the metadata may be said to indicate (e.g., specify), apart from a location of the audio object, a distance measure (e.g., the azimuthRange) indicative of a distance between the virtual sources.
  • the distance measure may be expressed by a distance parameter D .
  • the distance measure may indicate an angular distance or a Euclidean distance. In the examples below, the distance measure indicates an angular distance.
  • the distance measure may directly indicate a distance between the virtual sources themselves, or a distance between each of the virtual sources and the original audio object. As will be appreciated by the person of skill in the art, such distance measures can be easily converted into each other.
  • the metadata may indicate (e.g., specify) a measure of relative importance of the virtual sources and the original audio object (e.g., the objectDivergence). This measure of relative importance may be referred to as divergence and may be expressed by a divergence parameter (divergence value) d .
  • the divergence parameter d may range from 0 to 1, with 0 indicating zero divergence (i.e., no power is provided to the virtual sources-zero relative importance of the virtual sources), and 1 indicating full divergence (i.e., no power is provided to the original audio object-full relative importance of the virtual sources).
  • the renderer e.g., virtual object renderer
  • the renderer creates two additional audio objects O i + , O i - at the locations controlled by the distance measure D (e.g., by the azimuthRange element) and calculates three gains g di , g di + , g di - to ensure the power across the three new objects is equivalent to the original object.
  • the locations for the virtual objects (additional audio objects) are determined by the location of the original audio object and the distance measure D.
  • the distance measure e.g., azimuthRange
  • the distance measure may be reduced to ensure both virtual objects are within the rendering region (e.g., within the reproduction environment).
  • the need to recalculate the position of both virtual objects is to ensure the phantom image created remains at the correct location.
  • locations for the virtual objects may be determined first by transforming the Cartesian location to spherical coordinates using the mapping function Map SC (), described in section 3.3.2 "Object and Channel Location Transformations”. Then the spherical locations of Oi + and O i - are determined, e.g., in accordance with the above formula, and finally the locations may be transformed to Cartesian coordinates with the inverse transformation function Map SC ( ).
  • Map SC mapping function
  • g d and g v are weight factors (e.g., mixing gains) to be applied to the (original) audio object and the virtual (additional) audio objects.
  • the ADM specification also provides a specification for how these gains vary as the objectDivergence changes.
  • the gains to be applied to the original object and the two new virtual objects provide a power preserving spread across the three sources with the divergence (e.g., objectDivergence value) d controlling the distribution of the power between the sources.
  • the divergence (e.g., objectDivergence value) d varies between 0 and 1, where a value of 1 represents all the power coming from the virtual objects, and the original object made silent.
  • weight factors e.g., mixing gains
  • g di ⁇ 1 4 d + 1 0 ⁇ d ⁇ 0.5 1 ⁇ d 2 ⁇ d 0.5 ⁇ d ⁇ 1
  • g di ⁇ ⁇ 2 d 4 d + 1 0 ⁇ d ⁇ 0.5 1 4 ⁇ 2 d 0.5 ⁇ d ⁇ 1
  • Fig. 13 schematically illustrates a speaker layout comprising plural speakers 1342, 1344, 1346, 1348, among them a Left-surround speaker (Ls) 1342 and a front-left speaker (L) 1344.
  • the figure further illustrates an audio object 1310 and two virtual objects 1320, 1330 for phantom source rendering.
  • the virtual objects 1320, 1330 are created based on divergence metadata.
  • the rendering algorithm is to determine how to mix these objects in order to create the speaker feeds.
  • both virtual objects 1320, 1330 in the example of Fig. 13 are closer to the L speaker 1342 than to the Ls speaker 1344 it is expected that the gains for creating the speaker feed L[n] for the L speaker 1342 would direct the majority of each of their power to the L speaker 1342. Since the mixing is done in the renderer, the virtual objects 1320, 1330 will be summed coherently-hence the power preserving gains generated as part of creating the virtual objects will be summed inappropriately.
  • the distance measure e.g., azimuthRange
  • the present disclosure describes methods for controlling the constraints applied to render objects with divergence in order to tune their signal power or perceived loudness.
  • the present disclosure describes two methods for rendering audio objects with divergence metadata that address the aforementioned issues and that could be applied independently or in combination with each other.
  • Fig. 15 illustrates, as a general overview, a block diagram of an example of a renderer (rendering apparatus) 1500 according to embodiments of the disclosure that is capable of rendering audio objects with divergence metadata.
  • the renderer 1500 comprises a divergence metadata processing block (metadata processing unit) 1510, a point panner 1520, and a mixer block (mixer unit) 1530.
  • the divergence metadata processing block 1510 may correspond to, or be included in, the metadata pre-processor 110 in Fig. 7 .
  • the point panner 1520 may correspond to the point panner 810 in Fig. 8 .
  • the mixer block 1530 may correspond to the ramping mixer 130 in Fig. 7 .
  • the renderer 1500 receives an object (x[n]) 1512 and associated (divergence) metadata 1514 as input.
  • the metadata 1514 may include an indication of divergence d and the distance measure D. Further, the renderer 1500 may receive the speaker layout 1524 as an input. If the object 1512 has divergence metadata 1514 (e.g., divergence d and distance measure D ) associated with it, first the divergence metadata preprocessing block 1510 will interpret that metadata 1514 to create three audio objects 1522, namely virtual object sources (y V1 [n] and y V2 [n]) and the modified original object (y[n]).
  • divergence metadata 1514 e.g., divergence d and distance measure D
  • the point panner 1520 then will calculate the gain matrix G ij M 1534 which contains the gain applied to object i to create the signal for speaker j.
  • the point panner 1520 may further modify the signals associated with the three audio objects to thereby create three modified audio objects 1532, namely y'[n], y' V1 [n], and y' V2 [n].
  • the final stage of rendering is to apply the gain matrix created in the point panner 1520 to object signals in order create the speaker feeds 1542-this is the function of the mixer block 1530.
  • Both the aforementioned methods for rendering audio objects with divergence metadata can be performed by the renderer 1500, for example.
  • the first method describes a control function which can be added during the creation of the virtual objects, which compensates for the variation in how these virtual sources would be summed acoustically if rendered to speakers at their virtual locations. This could be integrated within the divergence metadata processing block 1510 of the renderer 1500.
  • the second method describes how the rendering gains can be normalized (for example in the point panner 1520) to ensure that a desired signal level is produced from the speakers in a specific layout. Both methods will now be described in detail.
  • the first element of the present method is to incorporate a distance (e.g., an angle of separation) into the calculation of the gains to allow for the effective panning to vary between an amplitude preserving pan and a power preserving pan.
  • a distance e.g., an angle of separation
  • an angle of separation ( ⁇ ) may be defined as the angle between the two virtual sources (more generally, as the distance, or distance measure).
  • the virtual sources will be located symmetrically about the original source, and in such cases, the angle of separation may easily be derived from the angle between the original source and either of the virtual sources (for example, the angle of separation of the virtual sources may be equal to twice the angle between the original source and either of the virtual sources).
  • the control function p is a function of the distance measure D, p ( D ).
  • the control function p being a function of the angle of separation ⁇ , p ( ⁇ ).
  • the range of p ( ⁇ ) may vary from 1, where the above equation represents the constraints of an amplitude preserving pan, to 2 where the above equation is equivalent to enforcing constraints of a power preserving pan.
  • Fig. 29 is a flowchart illustrating an overview of the first method of rendering audio objects with divergence as an example of method of rendering input audio for playback in a playback environment.
  • Input audio received by the method includes at least one audio object and associated metadata.
  • the associated metadata indicates at least a location of the audio object.
  • the metadata further indicates that the audio object is to be rendered with divergence, and may also indicate a degree of divergence (divergence parameter, divergence value) d and a distance measure D.
  • the degree of divergence may be said to be a measure of relative importance of virtual objects (additional audio objects) compared to the audio object.
  • the method comprises steps S2910 to S2930 described below.
  • the method may comprise, as an initial step, referring to the metadata for the audio object and determining whether a phantom object at the location of the audio object is to be created. If so, steps S2910 to S2930 may be executed. Otherwise, the method may end.
  • two additional audio objects associated with the audio object are created such that respective locations of the two additional audio objects are evenly spaced from the location of the audio object, on opposite sides of the location of the audio object when seen from an intended listener's position in the playback environment.
  • the additional audio objects may be referred to as virtual audio objects.
  • respective weight factors for application to the audio object and the two additional audio objects are determined.
  • the weight factors may be the mixing gains g d and g v described above.
  • the weight factors gains may impose a desired relative importance across the three objects.
  • the two additional audio objects may have equal weight factors.
  • the weight factors e.g., mixing gains g d and g v ; without intended limitation, reference may be made to the mixing gains g d and g v in the following
  • the majority of energy may be provided by the original object, while for high values of the divergence parameter, the majority of energy may be provided by the virtual objects.
  • the values of the divergence parameter may vary between 0 and 1. A divergence value of 0 indicates that all energy will be provided by the original object, so that g d will be equal to 1. Conversely, a divergence value of 1 indicates that all energy will be provided by the virtual objects. In this case, g d will be 0. Further, the weight factors may depend on the distance measure D. Examples of this dependence will be provided below.
  • the audio object and the two additional audio objects are rendered to one or more speaker feeds in accordance with the determined weight factors.
  • application of the weight factors to the audio object and the additional audio objects may yield the three new audio objects y [ n ], y V 1 [ n ], and y V 2 [ n ] described above, which may be rendered to the speaker feeds, for example by the point panner 1520 and the mixer block 1530 of the renderer 1500.
  • the rendering of the audio object and the two additional audio objects to the one or more speaker feeds may result in a gain coefficient for each of the one or more speaker feeds (e.g., for an audio object signal x [ n ] of the original audio object).
  • An apparatus for rendering input audio for playback in a playback environment (e.g., for performing the method of Fig. 29 ) may comprise a metadata processing unit (e.g., metadata pre-processor 110) and a rendering unit.
  • the rendering unit may comprise a panning unit and a mixer (e.g., the source panner 120 and either or both of the ramping mixer(s) 130, 140).
  • Step S2910 and step S2920 may be performed by the aforementioned metadata processing unit (e.g., metadata pre-processor 110).
  • Step S2930 may be performed by the rendering unit.
  • the method may further comprise normalizing the weight factors based on the distance measure D. That is, initial weight factors may be determined, for example in accordance with the divergence parameter d, and the initial weight factors may subsequently be normalized based on the distance measure D.
  • An example of such a method is illustrated in the flowchart of Fig. 30 .
  • Step S3010 , step S3020 , and step S3040 in Fig. 30 may correspond to steps S2910, S2920, and S2930, respectively, in Fig. 29 , wherein the weight factors determined at step S3020 may be referred to as initial weight factors.
  • the (initial) weight factors determined at step S3020 are normalized based on the distance measure.
  • Step S3030 may be performed by the metadata processing unit.
  • the weight factors may be normalized such that a sum of equal powers of the normalized weight factors is equal to a predetermined value (e.g., 1).
  • an exponent of the normalized weight factors in said sum may be determined based on the distance measure.
  • this normalization may be performed in accordance with the control function p ( ⁇ ).
  • the control function p ( ⁇ ) may be used as said exponent.
  • normalizing a set of quantities is understood to relate to uniformly scaling an initial set of quantities (i.e., using the same scaling factor for each quantity of the set) so that the set of scaled quantities satisfies a normalization condition, such as equation [6].
  • the control function p ( ⁇ ) may be a smooth monotonic function of the distance measure (e.g., angle of separation ⁇ ; without intended limitation, reference may be made to the angle of separation ⁇ in the following).
  • the function p ( ⁇ ) may yield 1 for the distance measure below a first threshold value and may yield 2 for the distance measure above a second threshold value.
  • the image range of p ( ⁇ ) extends from 1, where equation [6] represents the constraints of an amplitude preserving pan, to 2 where equation [6] is equivalent to enforcing constraints of a power preserving pan, as in equation [3].
  • p ( ⁇ ) For values of the distance measure between the first and second threshold values, p ( ⁇ ) varies between 1 and 2 (i.e., takes on intermediate values) as the distance measure (e.g., the angle of separation ⁇ ) increases. p ( ⁇ ) may have zero slope at the first and second threshold values. Further, p ( ⁇ ) may have an inflection point at an intermediate value between the first and second threshold values.
  • Fig. 16A illustrates an example of the general characteristic expected of p ( ⁇ ).
  • the control function p ( ⁇ ) follows the guiding principles that the panning function should tend to favor amplitude preservation if the virtual sources are close to the phantom image location, and should provide for power preservation once the sources become sufficiently separated.
  • the values of the weight factors may also depend on the divergence parameter. For small values of the divergence parameter, the majority of energy will be provided by the original object, while for high values of the divergence parameter, the majority of energy will be provided by the virtual objects. In one example, the values of the divergence parameter may vary between 0 and 1. A divergence value of 0 indicates that all energy will be provided by the original object. In this case, g v will be equal to 0 and g d will be equal to 1, regardless of the value of p ( ⁇ ). Conversely, a divergence value of 1 indicates that all energy will be provided by the virtual objects. In this case, g d will be 0, the value 2 g v p ⁇ will be equal to 1, and the value of g v will vary between 1 2 and 2 2 as p ( ⁇ ) varies between 1 and 2.
  • control function p ( ⁇ ) as a pure function of the distance measure (e.g., angle of separation) still constrains the weight factors (e.g., mixing gains) generated to be wideband-i.e. they apply the same gain to all frequencies. This may not fully agree with the guiding principle that the perception of phantom images varies across frequencies.
  • Fig. 16B illustrates an example of the general characteristic expected of p ( ⁇ , f ), i.e., how the control function p ( ⁇ , f ) varies across frequencies.
  • the amplitude panning constraint is preserved for larger distances (e.g., larger angles of separation) than for high frequencies. That is, for lower frequencies, the aforementioned first and second thresholds may be higher than for higher frequencies.
  • the first threshold may be a monotonically decreasing function of frequency
  • the second threshold may be a monotonically decreasing function of frequency
  • normalization of the weight factors may be performed on a sub-band basis, depending on frequency. That is, normalization of the weight factors may be performed for each of a plurality of sub-bands. Then, said exponent of the normalized weight factors in said sum mentioned above may be determined on the basis of a frequency of the frequency sub-band, so that the exponent is a function of the distance measure (e.g., angle of separation) and the frequency.
  • the frequency that is used for determining said exponent may be the center frequency of a respective sub-band or may be any other frequency suitably chosen within the respective sub-band.
  • the exponent may be the control function p ( ⁇ ,f ) .
  • the method described in the foregoing section addresses the issues that would arise through blindly applying a power preserving set of gains (weight factors) prior to rendering. However it does not address the issues which may arise within an object renderer where divergence is allowed to be applied to an object located anywhere in the immersive space. These issues arise primarily because rendering of the final speaker feeds occurs in the playback environment, rather than in the controlled environment of the content creator, and are intrinsic to the object renderer paradigm of immersive audio. Thus, under certain conditions, using the second method that will now be described in more detail may be of advantage. As noted above, the second method may be employed either as a stand alone or in combination with the first method that has been described in the foregoing section.
  • Fig. 31 is a flowchart illustrating an overview of the second method of rendering audio objects with divergence as an example of method of rendering input audio for playback in a playback environment.
  • Input audio received by the method includes at least one audio object and associated metadata.
  • the associated metadata indicates at least a location of the audio object.
  • the metadata further indicates that the audio object is to be rendered with divergence, and may also indicate a degree of divergence (divergence parameter, divergence value) d and a distance measure D.
  • the degree of divergence may be said to be a measure of relative importance of virtual objects (additional audio objects) compared to the audio object.
  • the method comprises steps S3110 to S3150 described below.
  • the method may comprise, as an initial step, referring to the metadata for the audio object and determining whether a phantom object at the location of the audio object is to be created. If so, steps S3110 to S3150 may be executed. Otherwise, the method may end.
  • Step S3110 and step S3120 in Fig. 31 may correspond to step S2910 and step S2920, respectively, in Fig. 29 .
  • a set of rendering gains for mapping (e.g., panning) the audio object and the two additional audio objects to the one or more speaker feeds is determined.
  • This step may be performed by the point panner 1520, for example. Setting aside the details of the internal algorithms used by the point panner 1520, its purpose is to determine how to steer an audio object, given the audio object's location, to the set of speakers it is currently rendering for.
  • step S3130 determines a rendering matrix G ij M (i.e., a set of rendering gains) which dictates the gains (rendering gains) applied to each object's content when mixing it into each speaker signal.
  • G ij M i.e., a set of rendering gains
  • the rendering gains are normalized based on the distance measure (e.g., angle of separation).
  • Step S3140 may be performed by the point panner 1520, for example.
  • the control functions p ( ⁇ ) and p ( ⁇ , f ) can be introduced, for example to replace p in equation [9].
  • the rendering gains may be normalized (e.g., re-scaled) such that a sum of equal powers of the normalized rendering gains for all of the one or more speaker feeds and for all of the audio objects and the two additional audio objects is equal to a predetermined value (such as 1, for example).
  • An exponent of the normalized rendering gains in said sum may be determined based on said distance measure. Said exponent may be the control function p ( ⁇ ) described above.
  • the normalization of the rendering gains may be performed on a sub-band basis and in dependence on frequency.
  • the audio object and the two additional audio objects are rendered to the one or more speaker feeds in accordance with the determined weight factors and the (normalized) rendering gains.
  • the method of Fig. 31 may additionally include a step of normalizing the weight factors, in analogy to step S3030 in Fig. 30 .
  • equations [7] and [10] recite a function p ( ⁇ ,f ). While these functions may typically be the same, in some cases they may be defined independently of one another, such that p ( ⁇ , f ) in equation [7] may not necessarily be equivalent to p ( ⁇ , f ) in equation [10].
  • An apparatus for rendering input audio for playback in a playback environment (e.g., for performing the method of Fig. 31 ) may comprise a metadata processing unit (e.g., metadata pre-processor 110) and a rendering unit.
  • the rendering unit may comprise a panning unit and a mixer (e.g., the source panner 120 and either or both of the ramping mixer(s) 130, 140).
  • Step S3110 and step S3120 may be performed by the aforementioned metadata processing unit (e.g., metadata pre-processor 110).
  • Step S3130, step S3140, and step S3150 may be performed by the rendering unit.
  • the screenScaling feature allows objects in the front half of the room (e.g., the playback environment) to be panned relative to the screen.
  • the screenRef flag in the object's metadata is used to indicate whether the object is screen related. If the flag is set to 1, the renderer will use metadata about the reference screen that was used during authoring (e.g., contained in the audioProgramme element) and the playback screen (e.g., given to the renderer as configuration parameters) to warp the azimuth and elevation of the objects in order to account for differences in the location and size of the screens.
  • ITU-R BS.2076-0 provides default screen specification for the reference screen for use when such information is not contained in the input file. The renderer shall use default values for the playback screen, e.g., these same default values, when no configuration data is provided.
  • the following conditions should be satisfied by the attributes of the audioProgrammeReferenceScreen sub-element of the audioProgramme element.
  • the same conditions apply to the corresponding renderer configuration parameters that specify the properties of the playback screen.
  • Step 1 If the screen position and size values are given in Cartesian coordinates, convert to spherical coordinates using the warping function described in section 3.3.2 "Object and Channel Location Transformations”. Step 2. Apply limits to the screen position and size metadata, as follows:
  • the warp function begins to warp angles at +/- 50 degrees. This is because the screen edges are allowed to be at +/- 45 degrees, and there needs to be a bit of "slack" space to prevent the warping function from producing line segments with zero slope, which would result in panning "dead zones”.
  • the angle-warping strategy naturally causes the displacement of objects due to screen scaling to be greater near the front of the room than in the center of the room.
  • the screen distance is purposely not considered in this strategy, as this allows a small screen near the center of the room to be treated the same as a larger screen near the front wall-i.e., the algorithm always considers the projection of the screen to the front wall of the room.
  • Fig. 17 in which the screen is projected to the front wall of the room in accordance with its width azimuth angle 1710 (screenWidth.azimuth).
  • Fig. 18A and Fig. 18B schematically show the resulting warping functions for azimuth and elevation for the following screen configurations:
  • ADM specifies screenEdgeLock for both channels and objects.
  • screenEdgeLock ensures that an audioObject is rendered at the edge of a playback screen.
  • the playback screen size will be an input to the command line of the renderer and will be in the audioProgrammeReferenceScreen format.
  • the ADM metadata provides for the specification of importance both of an audioPackFormat and an audioObject.
  • the ADM baseline renderer takes inputs related to importance called ⁇ importance> and ⁇ obj_importance>, both ranging from 0 to 10. AudioPackFormats with an importance value less than the ⁇ importance> parameter will be ignored by the metadata pre-processor 110. Within audio packs that will be rendered, objects with audioObject.importance less than ⁇ obj_importance> will be ignored by the metadata pre-processor 110.
  • ADM allows audioChannelFormat elements to contain optional frequency parameters specifying frequency ranges of audio data.
  • the baseline renderer treats this element of ADM as purely informational as has no direct influence on the renderer output. Explicitly no frequency information is required for LFE channels and no low pass characteristic is enforced on sub-woofer speaker outputs. However, because future processing stages in the playback system may choose to do something with this information, frequency metadata shall be passed through to the output LFE channels. See section Error! Reference source not found .3.2.4 "LFE Channels and Sub-Woofer Speakers " for more details regarding LFE channels and sub-woofer speaker rendering.
  • the ramping mixer combines the input object audio PCM samples to create speaker feeds using the gains calculated in the source panner 120.
  • the gains are crossfaded from their previous values over a length of time determined by the object's metadata.
  • the metadata update for object i is represented by a new vector of speaker gains, G ij M , and the number of slots remaining before the metadata update should be completed, ⁇ i , whose calculation is described in the next section.
  • each active object's PCM data is mixed into the speaker feeds y j .
  • This metadata feature controls the cross-fade of an object's position from its previous position.
  • the cross-fade is implemented directly by the ramping mixers 130, 140. This section details the calculation of ⁇ i .
  • F s SL otherwise.
  • ⁇ i is forced to be at least 1, to ensure no audio glitches occur.
  • the diffuse ramping mixer 140 combines the input object audio PCM samples using the gains calculated in the source panner 120 to feed the speaker decorrelator 150.
  • the gains may be crossfaded from their previous values over a length of time determined by the object's metadata.
  • the speaker-dependent part of the gain G j ⁇ is fixed by the speaker layout and so is applied directly in the decorrelator block.
  • the diffuse ramping mixer 140 thus down-mixes all the objects to a single mono channel y D using the gains g i M ⁇ .
  • the equations for the diffuse ramping mixer 140 are identical to the ramping mixer 130 except there is no-longer any speaker dependence.
  • the Speaker Decorrelator 150 takes the down-mixed channel y D from the diffuse ramping mixer 140, and the diffuse speaker gains G j ⁇ and creates the diffuse speaker feeds y j ⁇ .
  • the design makes use of one decorrelation filter per speaker pair.
  • a large number of orthogonal decorrelation filters may lead to audible decorrelation artefacts. Therefore, a maximum of four unique decorrelation filters are implemented. For larger numbers of speakers the decorrelation filter outputs are re-used.
  • Each decorrelation filter consists of four all-pass filter sections AP ns in series, where n indexes over the decorrelation filters, and s indexes over the all-pass sections within a decorrelation filter.
  • Fig. 20 illustrates an example of the four decorrelation filters and their respective all-pass filter sections.
  • Each all-pass filter section consists of a single parameter C DS and a delay line with delay d s .
  • the transient response of the decorrelators is improved by ducking the input upon detecting a quick rise in the signal envelope, and ducking the output upon detecting a quick fall in envelope.
  • An example of the full decorrelator structure is shown in Fig. 22 .
  • the decorrelator blocks are fed by a look-ahead delay to compensate for the ducking calculation latency.
  • the look-ahead delay is 2ms.
  • the ducking calculation first works by creating fast and slow smoothed envelope estimates.
  • the result is then smoothed with a single-pole smoother with slow time constant of 80ms, and a fast time constant of 5ms to produce e slow and e fast , respectively.
  • the original downmix signal y D is mixed with the ducked decorrelation filter signal, with y D receiving a mix coefficient of 0.9 and the ducked decorrelation filter signal receiving a mix coefficient of 0.3.
  • each y D mix block gives another decorrelated output. These decorrelated outputs are then multiplied by the appropriate speaker gain G j ⁇ and distributed to the speakers.
  • the section describes how the decorrelated outputs will map to speakers for specific speaker layouts.
  • Symbol 'D1' will denote the output of the decorrelator 1 block and '-D1' the negated output of the decorrelator 1 block. Since there are only up to 8 outputs from the decorrelator blocks, some outputs are re-used on the larger speaker layouts. On the smaller speaker layouts some decorrelator blocks will not be required.
  • the scene renderer 200 comprises a HOA panner 2310 and a mixer (e.g., HOA mixer) 2320.
  • the scene renderer 200 is presented with input audio objects, i.e., with metadata (e.g., ADM metadata) 25 and audio data (e.g., PCM audio data) 20, and with the speaker layout 30.
  • the scene renderer 200 outputs speaker feeds 2350 that can be combined (e.g., by addition) with the speaker feeds output by the object and channel renderer 100 and provided to the reproduction system 500.
  • Any LFE inputs are passed through or mixed to output LFE channels following the same rules as the channel and object renderer uses as set out in section 3.2.4 "LFE Channels and Sub-Woofer Speakers".
  • HOA Higher Order Ambisonics
  • the HOA Panner is responsible for generating a ( N + 1) 2 ⁇ N S matrix of gain coefficients, in the matrix G i , j M , where N S is the number of speakers in the playback system (excluding LFE channels): G i , j M : 1 ⁇ i ⁇ N + 1 2 1 ⁇ j ⁇ N S
  • This panner matrix is computed by first selecting the Reference HOA Matrix from the set of predefined matrices described in Appendix B.
  • Each row of this matrix is scaled by a scale factor that depends on the HOA Scaling Mode. This scaling is performed by the following procedure:
  • the methods and apparatus described in the present document may be implemented as software, firmware and/or hardware. Certain components may e.g. be implemented as software running on a digital signal processor or microprocessor. Other components may e.g. be implemented as hardware and or as application specific integrated circuits.
  • the signals encountered in the described methods and apparatus may be stored on media such as random access memory or optical storage media. They may be transferred via networks, such as radio networks, satellite networks, wireless networks or wireline networks, e.g. the Internet.
  • EEEs enumerated example embodiments

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Otolaryngology (AREA)
  • Stereophonic System (AREA)

Abstract

The present document relates to methods and apparatus for rendering input audio for playback in a playback environment. The input audio includes at least one audio object and associated metadata, and the associated metadata indicates at least a location of the audio object. A method for rendering input audio including divergence metadata for playback in a playback environment comprises creating two additional audio objects associated with the audio object such that respective locations of the two additional audio objects are evenly spaced from the location of the audio object, on opposite sides of the location of the audio object when seen from an intended listener's position in the playback environment, determining respective weight factors for application to the audio object and the two additional audio objects, and rendering the audio object and the two additional audio objects to one or more speaker feeds in accordance with the determined weight factors. The present document further relates to methods and apparatus for rendering audio input including extent metadata and/or diffuseness metadata for playback in a playback environment.

Description

    CROSS-REFERENCE TO RELATED APPLICATION
  • This application is a European divisional application of Euro-PCT patent application EP 16834241.8 (reference: D15128EP01), filed 18 November 2016.
  • TECHNICAL FIELD OF THE INVENTION
  • The present document relates to methods and apparatus for rendering of object-based audio content. In particular, the present document relates to methods and apparatus for improved immersive rendering of audio objects having associated metadata specifying extent (e.g., size) of the audio objects, diffusion, and/or divergence. These methods and apparatus are applicable to cinema sound reproduction systems and home cinema sound reproduction systems, for example.
  • BACKGROUND OF THE INVENTION
  • The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which in and of themselves may also be inventions.
  • As used herein, the term "audio object" may refer to a stream of audio object signals and associated audio object metadata. The metadata may indicate at least the position of the audio object. However, the metadata also may indicate decorrelation data, rendering constraint data, content type data (e.g. dialog, effects, etc.), gain data, trajectory data, etc. Some audio objects may be static, whereas others may have time-varying metadata: such audio objects may move, may change extent (e.g., size) and/or may have other properties that change over time. For example, audio objects may be humans, animals or any other elements serving as sound sources.
  • Recommendation ITU-R BS.2076 The Audio Definition Model (ADM) formalizes the description of the structure of metadata that can be applied in the rendering of audio data to one of the loudspeaker configurations specified in Recommendation ITU-R BS.2051. The ADM specifies a metadata model that describes the relationship between a group or groups of raw audio data and how they should be interpreted so that when reproduced, the original or authored audio experience is recreated. Importantly there is not a single audio format dictated by ADM, instead an emphasis on flexibility provides multiple ways to describe the variety of immersive experiences which may be on offer. Whereas the present document frequently makes reference to the ADM, the subject matter described therein is equally applicable to other specifications of metadata and other metadata models.
  • In order to reproduce an immersive audio experience, the description must be interpreted in the context of a playback environment to create speaker specific feeds. This process can typically be split into two steps, of which the second step is sometimes referred to as B-chain processing or playback system:
    1. 1. Rendering the immersive content to ideal speakers, and
    2. 2. Processing the ideal speaker signals to match a reproduction system (i.e. corrections for the room, actual speaker placement, DACs, Amplifiers and other equipment used during playback).
  • The renderer (rendering apparatus, e.g., baseline renderer) described in the present document addresses the first step of interpreting the description of the audio, e.g., in ADM, to create ideal speaker feeds-which can themselves be captured as a simpler ADM that does not require further rendering before reproduction.
  • In creating those ideal speaker feeds, it is desirable to have an improved treatment of the features extent (e.g., size), diffusion, and/or divergence that may be specified by the metadata for associated audio objects.
  • The present document addresses the above issues related to treatment of metadata and describes methods and apparatus for improved rendering of object-based audio content for playback, in particular of object-based audio content including audio objects for which one or more of extent, diffusion, and divergence are specified by the associated metadata.
  • SUMMARY OF THE INVENTION
  • According to an aspect of the disclosure, a method of rendering input audio for playback in a playback environment is described. The input audio may include at least one audio object and associated metadata. The associated metadata may indicate at least a location (e.g., position) of the audio object. The method may optionally comprise referring to the metadata for the audio object and determining whether a phantom object at the location of the audio object is to be created. The method may comprise creating two additional audio objects associated with the audio object such that respective locations of the two additional audio objects are evenly spaced from the location of the audio object, on opposite sides of the location of the audio object when seen from an intended listener's position in the playback environment. The additional audio objects may be located in the horizontal plane in which the audio object is located. The additional audio objects' locations may be fixed with respect to the location of the audio object. The additional audio objects may be evenly spaced from the intended listener's position, e.g., at equal radius. The additional audio objects may be referred to as virtual audio objects. The method may further comprise determining respective weight factors for application to the audio object and the two additional audio objects. The weight factors may be mixing gains. The weight factors (e.g., mixing gains) may impose a desired relative importance (e.g., relative weight) across the three objects. The two additional audio objects may have equal weight factors. The method may yet further comprise rendering the audio object and the two additional audio objects to one or more speaker feeds in accordance with the determined weight factors. The rendering of the audio object and the two additional audio objects to the one or more speaker feeds may result in a gain coefficient for each of the one or more speaker feeds (e.g., for an audio object signal of the audio object).
  • Configured as above, the proposed method allows efficient and accurate generation of a phantom object for the audio object at the location of the audio object. Thereby, audio power may be more equally distributed among speakers of a speaker layout, thus avoiding overload at particular speakers of the speaker layout.
  • In embodiments, the associated metadata may further indicate a distance measure indicative of a distance between the two additional audio objects. For example, the distance measure may be indicative of a distance between each of the additional audio objects and the audio object, such as an angular distance, or a Euclidean distance. Alternatively, the distance may be indicative of the distance between the two additional audio objects themselves, such as an angular distance or a Euclidean distance.
  • In embodiments, the associated metadata may further indicate a measure of relative importance (e.g., relative weight) of the two additional audio objects compared to the audio object. The measure of relative importance may be referred to as divergence, and be defined by a divergence parameter (divergence value), for example a divergence parameter d ∈ [0, 1], with 0 indicating zero relative importance of the additional audio objects and 1 indicating zero relative importance of the audio object-i.e., full relative importance of the additional audio objects. The weight factors may be determined based on said measure of relative importance.
  • In embodiments, the method may further comprise normalizing the weight factors based on said distance measure. For example, the weight factors may be normalized (e.g., scaled) such that a function f(g 1,g 2,D) of the weight factors g 1, g 2 and the distance measure D attains a predetermined value, e.g., 1. For example, the weight factors may be normalized such that f(g 1 ,g 2 ,D) = 1.
  • By normalizing the weight factors (e.g., mixing gains) based on the distance measure, it can be ensured that the perceptible loudness (signal power) for the audio object matches the artistic intent of the content creator. Moreover, for an audio object that is moving across the reproduction environment along a trajectory, consistent perceived loudness can be achieved by the proposed method, even if the speaker feeds to which the audio object and the additional audio objects are primarily rendered, respectively, changes along the trajectory. For example, for the additional audio objects being spaced close to each other, the normalization may represent an amplitude preserving pan to account for coherent summation of the signals of the additional audio objects. On the other hand, for the additional audio objects being sufficiently spaced from each other, the normalization may represent a power preserving pan.
  • In embodiments, the weight factors may be normalized such that a sum of equal powers of the normalized weight factors is equal to a predetermined value. An exponent of the normalized weight factors in said sum may be determined based on the distance measure. The weight factors may be mixing gains. The predetermined value may be 1, for example. The weight factors (e.g., mixing gains) may be normalized to satisfy (g 1) p(D) + 2(g 2) p(D) = 1, where g 1 is the weight factor (e.g., mixing gain) to be applied to the audio object (e.g., multiplying the audio object signal of the (original) audio object), g 2 is the weight factor (e.g., mixing gain) to be applied to each of the two additional audio objects (e.g., multiplying the audio object signal of the (original) audio object), D is the distance measure, and p is a (smooth) monotonic function that yields p(D) = 1 for the distance measure below a first threshold and that yields p(D) = 2 for the distance measure above a second threshold.
  • In embodiments, normalization of the weight factors may be performed on a (frequency) sub-band basis, in dependence on frequency. That is, normalization may be performed for each of a plurality of sub-bands. The exponent of the normalized weight factors in said sum may be determined on the basis of a frequency of the respective sub-band. The exponent may be a function of the distance measure and the frequency, p(D,f). For example, for higher frequencies, the aforementioned first and second thresholds may be lower than for lower frequencies. That is, the first threshold may be a monotonically decreasing function of frequency, and the second threshold may be a monotonically decreasing function of frequency. The frequency may be the center frequency of a respective sub-band or may be any other frequency suitably chosen within the respective sub-band.
  • Thereby, different characteristics of audio signals at different frequencies with respect to the perception of their summation can be accounted for. In particular, different distance thresholds within which signals of audio objects sum coherently can be taken into account, to thereby achieve a desired or intended loudness of the audio object in each frequency sub-band.
  • In embodiments, the method may further comprise determining a set of rendering gains for mapping (e.g., panning) the audio object and the two additional audio objects to the one or more speaker feeds. The method may yet further comprise normalizing the rendering gains based on said distance measure.
  • By normalizing the rendering gains based on the distance measure, it can be ensured that the perceptible loudness (level, signal power) for the audio object matches the artistic intent of the content creator, even if two or more of the audio object and the additional audio object are located close to each other and/or would be rendered to the same speaker feed. For this case, the normalization of the rendering gains may represent an amplitude preserving pan. Otherwise, for sufficient distance between the additional audio objects, the normalization may represent a power preserving pan.
  • In embodiments, the rendering gains may be normalized such that a sum of equal powers of the normalized rendering gains for all of the one or more speaker feeds and for all of the audio objects and the two additional audio objects is equal to a predetermined value. An exponent of the normalized rendering gains in said sum may be determined based on said distance measure. The predetermined value may be 1, for example. The rendering gains may be normalized to satisfy ∑ i j (Gij ) p(D) = 1, where index i indicates a respective one among the audio object and the two additional audio objects, j indicates a respective one among the speaker feeds, Gij are the rendering gains, D is the distance measure, and p is a (smooth) monotonic function that yields p(D) = 1 for the distance measure below a first threshold and that yields p(D) = 2 for the distance measure above a second threshold.
  • In embodiments, normalization of the rendering gains may be performed on a (frequency) sub-band basis and in dependence on frequency. That is, normalization may be performed for each of a plurality of sub-bands. The exponent of the rendering gains in said sum may be determined on the basis of a frequency of the respective sub-band. The exponent may be a function of the distance measure and the frequency, p(D, f). For example, for higher frequencies, the aforementioned first and second thresholds may be lower than for lower frequencies. That is, the first threshold may be a monotonically decreasing function of frequency, and the second threshold may be a monotonically decreasing function of frequency. The frequency may be the center frequency of a respective sub-band or may be any other frequency suitably chosen within the respective sub-band.
  • According to another aspect of the disclosure, a method of rendering input audio for playback in a playback environment is described. The input audio may include at least one audio object and associated metadata. The associated metadata may indicate at least a location (e.g., position) of the at least one audio object and a three-dimensional extent (e.g., size) of the at least one audio object. The method may comprise rendering the audio object to one or more speaker feeds in accordance with its three-dimensional extent. Said rendering of the audio object to one or more speaker feeds in accordance with its three-dimensional extent may be performed by determining locations of a plurality of virtual audio objects within a three-dimensional volume defined by the location of the audio object and its three-dimensional extent. The virtual audio objects may be referred to as virtual sources. Candidates for the virtual audio objects may be arranged in a grid (e.g., a three-dimensional rectangular grid) across the playback environment. Determining said locations may involve imposing a respective minimum extent for the audio object in each of the three dimensions (e.g., {x,y,z} or {r,θ,ϕ}). Said rendering of the audio object to one or more speaker feeds in accordance with its three-dimensional extent may be performed by further, for each virtual audio object, determining a weight factor that specifies the relative importance of the respective virtual audio object. Said rendering of the audio object to one or more speaker feeds in accordance with its three-dimensional extent may be performed by further rendering the audio object and the plurality of virtual audio objects to the one or more speaker feeds in accordance with the determined weight factors. The rendering of the audio object and the virtual audio objects to the one or more speaker feeds may be performed by a so-called point panner, i.e., the audio object and the plurality of virtual audio objects may be treated as respective point sources. The rendering of the audio object and the virtual audio objects to the one or more speaker feeds may result in a gain coefficient for each of the one or more speaker feeds (e.g., for an audio object signal of the audio object).
  • Configured as above, the proposed method allows for efficient and accurate rendering of audio objects having extent, e.g., a three-dimensional size. In other words, the proposed method allows for efficient and accurate rendering of audio objects that take a three-dimensional volume in the reproduction environment. When seen from the intended listener's position, the audio object thus not only features width and height, but can additionally feature depth. The proposed method provides for independent control of each of the three spatial dimensions of extent (e.g., {x,y,z} or {r,θ,ϕ}), and thus provides for a rendering framework that allows for greater flexibility at the time of content creation. In consequence, the proposed method provides the rendering framework for more immersive, more realistic rendering of audio objects with extent.
  • In embodiments, the method may further comprise, for each virtual audio object and for each of the one or more speaker feeds, determining a gain for mapping the respective virtual audio object to the respective speaker feed. The gains may be point gains. The gains may be determined based on the location of the respective virtual audio object and the location of the respective speaker feed (i.e., the location of a speaker for playback of the respective speaker feed). The method may yet further comprise, for each virtual object and for each of the one or more speaker feeds, scaling the respective gain with the weight factor of the respective virtual audio object.
  • In embodiments, the method may further comprise, for each speaker feed, determining a first combined gain depending on the gains of those virtual audio objects that lie within a boundary of the playback environment. The method may further comprise, for each speaker feed, determining a second combined gain depending on the gains of those virtual audio objects that lie on said boundary. The first and second combined gains may be normalized. The method may yet further comprise, for each speaker feed, determining a resulting gain for the plurality of virtual audio objects based on the first combined gain, the second combined gain, and a fade-out factor indicative of the relative importance of the first combined gain and the second combined gain. The fade-out factor may depend on the three-dimensional extent (e.g., size) of the audio object and the location of the audio object. For example, the fade-out factor may depend on a fraction of the overall extent (e.g., of the overall three-dimensional volume) of the audio object that is within the boundary of the playback environment.
  • In embodiments, the method may further comprise, for each speaker feed, determining a final gain based on the resulting gain for the plurality of virtual audio objects, a respective gain for the audio object, and a cross-fade factor depending on the three-dimensional extent (e.g. size) of the audio object.
  • In embodiments, the associated metadata may indicate a first three-dimensional extent (e.g., size) of the audio object in a spherical coordinate system by respective ranges of values for a radius, an azimuth angle, and an elevation angle. The method may further comprise determining a second three-dimensional extent (e.g., size) in a Cartesian coordinate system as dimensions of a cuboid that circumscribes the part of a sphere that is defined by said respective ranges of the values for the radius, the azimuth angle, and the elevation angle. The method may yet further comprise using the second three-dimensional extent as the three-dimensional extent of the audio object.
  • In embodiments, the associated metadata may further indicate a measure of a fraction of the audio object that is to be rendered isotropically (e.g., from all directions with equal powers) with respect to an intended listener's position in the playback environment. The method may further comprise creating an additional audio object at a center of the playback environment and assigning a three-dimensional extent (e.g. size) to the additional audio object such that a three-dimensional volume defined by the three-dimensional extent of the additional audio object fills out the entire playback environment. The method may further comprise determining respective overall weight factors for the audio object and the additional audio object based on the measure of said fraction. The method may yet further comprise rendering the audio object and the additional audio object, weighted by their respective overall weight factors, to the one or more speaker feeds in accordance with their respective three-dimensional extents. Each speaker feed may be obtained by summing respective contributions from the audio object and the additional audio object.
  • Configured as above, the proposed method provides for perceptually appealing delocalization of part or all of an audio object. In particular, by panning the additional audio object to the center of the reproduction environment (e.g., room) and letting it fill out the entire reproduction environment, the proposed method enables to achieve diffuseness of the audio object regardless of actual speaker layout of the reproduction environment. Further, by employing the rendering of extent for the additional audio object, diffuseness can be realized in an efficient manner, essentially without introducing new components/modules into a renderer for performing the proposed method.
  • In embodiments, the method may further comprise applying decorrelation to the contribution from the additional audio object to the one or more speaker feeds
  • It should be noted that the methods described in the present document may be applied to renderers (e.g., rendering apparatus). Such rendering apparatus may be configured to perform the methods described in the present document and/or may comprise respective modules (or blocks, units) for performing one or more of the processing steps of the methods described in the present document. Any statements made above with respect to such methods are understood to likewise apply to apparatus for rendering input audio for playback in a playback environment.
  • Consequently, according to another aspect of the disclosure, an apparatus (e.g., renderer, rendering apparatus) for rendering input audio for playback in a playback environment is described. The input audio may include at least one audio object and associated metadata. The associated metadata may indicate at least a location (e.g., position) of the audio object. The apparatus may comprise a metadata processing unit (e.g., a metadata pre-processor). The metadata processing unit may be configured to create two additional audio objects associated with the audio object such that respective locations of the two additional audio objects are evenly spaced from the location of the audio object, on opposite sides of the location of the audio object when seen from an intended listener's position in the playback environment. The metadata processing unit may be further configured to determine respective weight factors for application to the audio object and the two additional audio objects. The apparatus may further comprise a rendering unit configured to render the audio object and the two additional audio objects to one or more speaker feeds in accordance with the determined weight factors. The rendering unit may comprise a panning unit (e.g., point panner) and may further comprise a mixer.
  • In embodiments, the associated metadata may further indicate a distance measure indicative of a distance between the two additional audio objects.
  • In embodiments, the associated metadata may further indicate a measure of relative importance of the two additional audio objects compared to the audio object. The weight factors may be determined based on said measure of relative importance.
  • In embodiments, the metadata processing unit may be further configured to normalize the weight factors based on said distance measure.
  • In embodiments, the weight factors may be normalized such that a sum of equal powers of the normalized weight factors is equal to a predetermined value. An exponent of the normalized weight factors in said sum may be determined based on the distance measure (e.g., the metadata processing unit may be configured to determine said exponent based on the distance measure).
  • In embodiments, normalization of the weight factors may be performed on a sub-band basis, in dependence on frequency.
  • In embodiments, the rendering unit may be further configured to determine a set of rendering gains for mapping the audio object and the two additional audio objects to the one or more speaker feeds. The rendering unit may be yet further configured to normalize the rendering gains based on said distance measure.
  • In embodiments, the rendering gains may be normalized such that a sum of equal powers of the normalized rendering gains for all of the one or more speaker feeds and for all of the audio objects and the two additional audio objects is equal to a predetermined value. An exponent of the normalized rendering gains in said sum may be determined based on said distance measure (e.g., the metadata processing unit may be configured to determine said exponent based on the distance measure).
  • In embodiments, normalization of the rendering gains may be performed on a sub-band basis, in dependence on frequency.
  • According to another aspect of the disclosure, an apparatus (e.g., renderer, rendering apparatus) for rendering input audio for playback in a playback environment is described. The input audio may include at least one audio object and associated metadata. The associated metadata may indicate at least a location (e.g., position) of the at least one audio object and a three-dimensional extent (e.g., size) of the at least one audio object. The apparatus may comprise a rendering unit for rendering the audio object to one or more speaker feeds in accordance with its three-dimensional extent. The rendering unit may be configured to determine locations of a plurality of virtual audio objects within a three-dimensional volume defined by the location of the audio object and its three-dimensional extent. The rendering unit may be further configured to for each virtual audio object, determine a weight factor that specifies the relative importance of the respective virtual audio object. The rendering unit may be further configured to render the audio object and the plurality of virtual audio objects to the one or more speaker feeds in accordance with the determined weight factors. The rendering unit may comprise a panning unit (e.g., extent panner, or size panner) and may further comprise a mixer.
  • In embodiments, the rendering unit may be further configured to, for each virtual audio object and for each of the one or more speaker feeds, determine a gain for mapping the respective virtual audio object to the respective speaker feed. The rendering unit may be yet further configured to, for each virtual object and for each of the one or more speaker feeds, scale the respective gain with the weight factor of the respective virtual audio object.
  • In embodiments, the rendering unit may be further configured to, for each speaker feed, determine a first combined gain depending on the gains of those virtual audio objects that lie within a boundary of the playback environment. The rendering unit may be further configured to, for each speaker feed, determine a second combined gain depending on the gains of those virtual audio objects that lie on said boundary. The rendering unit may be yet further configured to, for each speaker feed, determine a resulting gain for the plurality of virtual audio objects based on the first combined gain, the second combined gain, and a fade-out factor indicative of the relative importance of the first combined gain and the second combined gain.
  • In embodiments, the rendering unit may be further configured to, for each speaker feed, determine a final gain based on the resulting gain for the plurality of virtual audio objects, a respective gain for the audio object, and a cross-fade factor depending on the three-dimensional extent (e.g., size) of the audio object.
  • In embodiments, the associated metadata may indicate a first three-dimensional extent (e.g., size) of the audio object in a spherical coordinate system by respective ranges of values for a radius, an azimuth angle, and an elevation angle. The apparatus may further comprise a metadata processing unit (e.g., a metadata pre-processor) configured to determine a second three-dimensional extent (e.g., size) in a Cartesian coordinate system as dimensions of a cuboid that circumscribes the part of a sphere that is defined by said respective ranges of the values for the radius, the azimuth angle, and the elevation angle. The rendering unit may be configured to use the second three-dimensional extent as the three-dimensional extent of the audio object.
  • In embodiments, the associated metadata may further indicate a measure of a fraction of the audio object that is to be rendered isotropically with respect to an intended listener's position in the playback environment. The apparatus may further comprise a metadata processing unit (e.g., a metadata pre-processor) configured to create an additional audio object at a center of the playback environment and assigning a three-dimensional extent (e.g., size) to the additional audio object such that a three-dimensional volume defined by the three-dimensional extent of the additional audio object fills out the entire playback environment. The metadata processing unit may be further configured to determine respective overall weight factors for the audio object and the additional audio object based on the measure of said fraction. The metadata processing unit may be yet further configured to output the audio object and the additional audio object, weighted by their respective overall weight factors, to the rendering unit for rendering the audio object and the additional audio object to the one or more speaker feeds in accordance with their respective three-dimensional extents. The rendering unit may be configured to obtain each speaker feed by summing respective contributions from the audio object and the additional audio object.
  • In embodiments, the rendering unit may be further configured to apply decorrelation to the contribution from the additional audio object to the one or more speaker feeds.
  • According to another aspect, a software program is described. The software program may be adapted for execution on a processor and for performing the method steps outlined in the present document when carried out on a computing device.
  • According to another aspect, a storage medium is described. The storage medium may comprise a software program adapted for execution on a processor and for performing the method steps outlined in the present document when carried out on a computing device.
  • According to a further aspect, a computer program product is described. The computer program may comprise executable instructions for performing the method steps outlined in the present document when executed on a computer.
  • It should be noted that the methods and apparatus including its preferred embodiments as outlined in the present document may be used stand-alone or in combination with the other methods and systems disclosed in this document. Furthermore, all aspects of the methods and apparatus outlined in the present document may be arbitrarily combined. In particular, the features of the claims may be combined with one another in an arbitrary manner.
  • DESCRIPTION OF THE DRAWINGS
  • Example embodiments are explained below with reference to the accompanying drawings, wherein:
    • Fig. 1 and Fig. 2 illustrate examples of different frames of references for playback environments;
    • Fig. 3 illustrates an example of a sound field decomposition in a spherical coordinate system;
    • Fig. 4 illustrates an example of an input ADM format;
    • Fig. 5 illustrates an example of an output ADM format;
    • Fig. 6 schematically illustrates an example of an architecture of a renderer according to embodiments of the disclosure;
    • Fig. 7 schematically illustrates an example of an architecture of an object and channel renderer of the renderer according to embodiments of the disclosure;
    • Fig. 8 schematically illustrates an example of an architecture of source panner of the object and channel renderer;
    • Fig. 9 illustrates an example of a piece-wise linear mapping between extent values;
    • Fig. 10A and Fig. 10B illustrate examples of extents in a spherical coordinate system;
    • Fig. 11 schematically illustrates an example of a processing order of metadata processing in the renderer according to embodiments of the disclosure;
    • Fig. 12 schematically illustrates an example of an audio object and two virtual objects for phantom source panning in the renderer according to embodiments of the disclosure;
    • Fig. 13 schematically illustrates an example of a speaker layout in which phantom source panning can be performed;
    • Fig. 14A, Fig. 14B, and Fig. 14C illustrate examples of relative arrangements of virtual object locations and speaker locations for a given speaker layout;
    • Fig. 15 schematically illustrates an example of an architecture of a renderer that is capable of rendering audio objects with divergence metadata according to embodiments of the disclosure;
    • Fig. 16A and Fig. 16B show examples of control functions for gain normalization;
    • Fig. 17 schematically illustrates an example of projecting a screen to the front wall of a room;
    • Fig. 18A and Fig. 18B show examples of screen scaling warping functions for azimuth and elevation, respectively;
    • Fig. 19A and Fig19B show examples of audio objects to which the screen edge lock feature is applied;
    • Fig. 20 schematically illustrates an example of a core decorrelator in the renderer according to embodiments of the disclosure;
    • Fig. 21 schematically illustrates an example of an all-pass filter structure in the renderer according to embodiments of the disclosure;
    • Fig. 22 schematically illustrates an example of an architecture of a transient-compensated decorrelator in the renderer according to embodiments of the disclosure;
    • Fig. 23 schematically illustrates an example of a scene renderer of the renderer according to embodiments of the disclosure;
    • Fig. 24 is a flowchart schematically illustrating a method (e.g., algorithm) for rendering audio objects with extent according to embodiments of the disclosure;
    • Fig. 25 and Fig. 26 are flowcharts schematically illustrating details of the method of Fig. 24 ;
    • Fig. 27 is a flowchart schematically illustrating a method for transforming an extent of the audio object from spherical coordinates to Cartesian coordinates according to embodiments of the disclosure;
    • Fig. 28 is a flowchart schematically illustrating a method (e.g., algorithm) for rendering audio objects with diffusion according to embodiments of the disclosure;
    • Fig. 29 is a flowchart schematically illustrating a method (e.g., algorithm) for rendering audio objects with divergence according to embodiments of the disclosure;
    • Fig. 30 is a flowchart schematically illustrating a modification of the method of Fig. 29 ; and
    • Fig. 31 is a flowchart schematically illustrating another method (e.g., algorithm) for rendering audio objects with divergence according to embodiments of the disclosure;
    DETAILED DESCRIPTION
  • The present document describes several schemes (methods) and corresponding apparatus for addressing the above issues. These schemes, directed to rendering of audio objects with extent, diffusion, and divergence (e.g., audio objects having extent metadata, diffuseness metadata, and divergence metadata), respectively, may be employed individually or in conjunction with each other.
  • 1. INTRODUCTION 1.1 Baseline Renderer Scope
  • The renderer (e.g., baseline renderer) described in this document may be suitable to (see, e.g., ITU-R Document 6C/511-E (annex 10) to chairman's report for continuation of the RG):
    • Be used during production of advanced sound programs
    • Be used for monitoring, e.g. content authoring and quality assessment
    • Be used, in listening experiments and evaluations, for
      ∘ Making assessment of different audio systems independent of the renderer component
    • Be used as a renderer to evaluate other renderers.
  • Within the itemized scope above, the renderer specifies algorithms for rendering a subset of ADM and is not meant as a complete product. The algorithms and architecture described in the baseline renderer is designed to be easily extended to completely cover the ADM specification. Moreover, the renderer described in this document is not to be understood to be limited to ADM and may likewise be applied to other specifications of object-based audio content.
  • ADM allows for the grouping of audio elements into programs and can capture multiple programs in a single ADM tree. This ability to capture multiple ways of compositing audio primarily addresses content management aspects for the broadcast ecosystem, and has little influence on how individual elements are rendered. With this in mind the renderer does not address the logic components required to select the input audio to the rendering process, and assumes a production system using the renderer would provide this functionality.
  • 1.2 Spatial Audio Description
  • The ADM supports several formats to represent a spatial audio description (SAD). In all cases, a fundamental component of the SAD is the means to specify the nominal locations of sounds. This requires establishing a frame of reference.
  • 1.2.1 Frame of Reference
  • In order to specify locations in a space (e.g., in a playback environment), a frame of reference (FoR) is required. There are many ways to classify reference frames, but one fundamental consideration is the distinction between allocentric (or environmental) and egocentric (observer) reference.
    • An egocentric frame of reference encodes an object location relative to the position (location and orientation) of the observer or "self" (e.g., relative to an intended listener's position).
    • An allocentric frame of reference encodes an object location using reference locations and directions relative to other objects in the environment.
  • Fig. 1 and Fig. 2 schematically illustrate examples of an egocentric frame of reference and an allocentric frame of reference, respectively. In the illustrated examples, the egocentric location is 56° azimuth and 2m from the listener. The allocentric location is 1/4 of the way from left to right wall, 1/3 of the way from front to back wall.
  • An egocentric reference is commonly used for the study and description of perception; the underlying physiological and neurological processes of acquisition and coding most directly relate to the egocentric reference. For audio scene description, an egocentric representation is appropriate in scenarios when the sound scene is captured from a single point (such as with an Ambisonics microphone array, or other "scene-based" models), or when the sound scene is intended for a single, isolated listener (such as listening to music over headphones). As suggested in Fig. 1A above, a spherical coordinate system is often well suited for specifying locations when using an egocentric frame of reference. Furthermore, most scene-based spatial audio descriptions are based on a decomposition that utilizes circular or spherical coordinates, as in the example of Fig. 3 , which illustrates a simplified single-band in-phase B-format decoder for a square loudspeaker layout. Notably, Fig. 3 illustrates a naive example which does not fulfil the psychoacoustic criteria for Ambisonics decoding. The ADM supports scene-based, egocentric representations and spherical coordinates.
  • An allocentric reference is well suited for audio scene descriptions that are independent of a single observer position, and when the relationship between elements in the playback environment is of interest. A rectangular or Cartesian coordinate system is often used for specifying locations when using an allocentric frame of reference. The ADM supports specifying location using an allocentric frame of reference, and Cartesian coordinates.
  • 1.2.2 Coordinate Systems
  • All direct speaker and dynamic object channels are accompanied by metadata (associated metadata) that specifies at least a location.
  • Spherical coordinates indicate the location of an object, as a direction of arrival, in terms of azimuth and elevation, relative to one listening position. In addition, a (relative) distance parameter (e.g., in the range 0...1) may be used to place an object at a point between the listener and the boundary of the speaker array.
  • Cartesian coordinates indicate the location of an object, as a position relative to a normalized listening space, in terms of X, Y and Z coordinates of a unit cube (the "Cartesian cube", defined by |X| < 1, |Y| < 1 and |Z| < 1). The X index corresponds to the left-right dimension; the Y index corresponds to the rear-front dimension; and the Z index corresponds to the down-up dimension. As we will see, the cornerstones for the allocentric model are the corners of the unit cube and the loudspeakers that define these corners.
  • Note that the use of spherical coordinates, as the means for specifying object locations, does not imply that the loudspeakers in the playback environment must also lie on a sphere. Similarly, the use of Cartesian coordinates, as the means for specifying object locations, does not imply that the loudspeakers in the playback environment must also lie on a rectangular surface. It is safer to assume that different listening environments will contain loudspeakers that are placed so as to satisfy a variety of acoustic, aesthetic and practical constraints.
  • The ADM supports both egocentric spherical coordinates and allocentric Cartesian coordinates. The panning function defined in section 3.2.1 "Rendering Point Objects" below may be based on Cartesian coordinates to specify the location of audio sources in space. Thus in order to render a scene described using egocentric spherical coordinates, a translation is required. A change of coordinate systems could be achieved using simple trigonometry. However, translation of the frame of reference is more complicated, and requires that the space be "warped" to preserve the artistic intent. In the following sections we provide more details on the allocentric frame of reference used, and the means to translate location metadata.
  • 1.2.3 Mapping from Egocentric Spherical to Allocentric Cartesian Coordinates
  • For each ITU channel configuration, an allocentric frame of reference is constructed based on key channel locations. That is, the object location is defined relative to landmark channels. This ensures that the relative location of channels and objects remains consistent, and that the most important spatial aspects of an audio program (from the mixer's perspective) are preserved. For example, an object that moves across the front sound stage from "full left" to "full right" will do so in every playback environment.
  • In defining the mapping function, from spherical to Cartesian, the following principles will generally be adhered to:
    1. 1. For any channel configuration with 2 or more speakers, there will always be a channel located at (X,Y,Z) = (-1,1,0) (the front-left corner of the cube) and there will always be a speaker located at (X,Y,Z) = (1,1,0) (the front-right corner of the cube).
    2. 2. For any channel configuration with 4 or more speakers in the middle layer, there will always be a speaker located at (X,Y,Z) = (-1, -1,0) (the back-left corner of the cube) and there will always be a channel located at (X,Y,Z) = (1, -1,0) (the back-right corner of the cube).
    3. 3. For any channel configuration with 2 or more elevated channels, there will always be a speaker located at (X,Y,Z) = (-1,1,1) (the top-front-left corner of the cube) and there will always be a speaker located at (X,Y,Z) = (1,1,1) (the top-front-right corner of the cube).
    4. 4. For any channel configuration with 4 or more elevated speakers, there will always be a speaker located at (X,Y,Z) = (-1,-1,1) (the top-back-left corner of the cube) and there will always be a speaker located at (X,Y,Z) = (1,-1,1) (the top-back-right corner of the cube).
    5. 5. For any channel configuration with 2 or more bottom speakers, there will always be a speaker located at (X,Y,Z) = (-1,1,-1) (the bottom-front-left corner of the cube) and there will always be a speaker located at (X,Y,Z) = (1,1,-1) (the bottom-front-right corner of the cube).
  • These rules ensure that, within each layer (middle, upper and bottom layers) channels are assigned to the extremes of each axis (the corners of the unit cube), with highest priority being given to the front corners of the cube.
  • 1.2.3.1 Reference Rendering Environment
  • When an audio scene is authored, the author will generally have a specific playback environment in mind. This will generally coincide with the playback environment used by the author during the content-creation process.
  • The playback environment that is deemed, by the author, to be preferred for playback of the audio file, will be referred to as the reference rendering environment. By inspection of the audioPackFormat in the file, the renderer will, if possible, determine the identity of the reference rendering environment, and in particular, it will determine Azmax, the largest azimuth angle of all speakers at elevation = 0 in the reference rending environment.
  • Most often, Azmαx will be equal to 110° or 135° (although it may also be 30°, if the reference rendering environment was Stereo, or 180°, if the reference rendering environment included a rear-center speaker). If the identity of the reference rendering environment can be determined by the renderer, and Azmαx = 110°, then we assign the attribute Flag 110 = true. Otherwise, we assign Flag 110 = false.
  • Flag 110 is therefore an attribute that, when true, tells us that the author created this audio content in an environment where the rear-most surround channel was located at Azmαx = 110° (and this will generally occur when there are 5 channels in the elevation = 0 plane).
  • 1.2.3.2 Rules for Mapping Spherical to Cartesian Coordinates
  • If a dynamic audio object (or direct speaker signal) has its location specified in terms of Spherical Coordinates, a mapping function, MapSC (), will be used to map egocentric spherical coordinates to allocentric Cartesian coordinates as follows: X Y Z = Map SC Az El R Flag 110
    Figure imgb0001
  • The following rules are used to define the behavior of this mapping function:
    1. 1. An object that is located in Spherical coordinates at (Az, El) = (30°, 0°) will be mapped to Cartesian coordinates at (X,Y,Z) = (-1,1,0).
    2. 2. If Flag 110 = true,
      An audio object located in Spherical coordinates at (Az, El) = (110°, 0°) will be mapped to Cartesian coordinates at (X,Y,Z) = (-1,-1,0). This rule ensures that any sounds that were intended, by the content creator, to be played from the left surround speaker, will play correctly from the rear-most left surround speaker in the playback environment.
      Otherwise (if Flag 110 = false),
      An audio object located in Spherical coordinates at (Az, El) = (135°, 0°) will be mapped to Cartesian coordinates at (X,Y,Z) = (-1,-1,0). This rule ensures that any sounds that were intended, by the content creator, to be played from the rear-most left surround speaker, will play correctly from the rear-most left surround speaker in the playback environment.
    3. 3. An object that is located in Spherical coordinates at El = 30° will be mapped to Cartesian coordinates at Z = 1.
    4. 4. An object that is located in Spherical coordinates at El = -30° will be mapped to Cartesian coordinates at Z = -1.
  • The definition of the MapSC () function can be found in section 3.3.2 "Object and Channel Location Transformations" below.
  • 2. SYSTEM OVERVIEW 2.1 Inputs
  • Primary inputs to the baseline renderer are:
    1. 1. Audio described in accordance to ADM (ITU-R BS.2076-0), contained in a BW64 file in accordance to ITU-R BS.2088-0, and
    2. 2. A speaker layout selected from one specified in Recommendation ITU-R BS.2051-0, Advanced sound systems for programme production (Annex 1, ITU-R BS.2051-0). Notably, ITU-R BS.2051-0 Systems A through H may be referred to simply as Systems A through H in the remainder of this document, occasionally omitting the qualifier "ITU-R BS.2051-0".
  • Additional secondary inputs can be incorporated in the rendering algorithm to modify its behavior:
    1. 1. Importance - The renderer importance is used as a threshold for selecting which elements are excluded from the rendering process. The importance is nominally specified as a pair of integer values from 0 to 10 one expressing the importance threshold for audioPacks (referred to simply as <importance>) the second expressed the threshold applied to individual Object elements (<obj_importance>). If only one input value is provided both <importance> and <obj_importance> are set to that value. See section 3.3.9 "Importance" below for details how these importance values are used in the renderer.
    2. 2. Screen position - The renderer accepts a screen position defined using the same elements that the audioProgrammeReferenceScreen is specified in ADM, referred to as <playback_screen>. When an audioProgrammeReferenceScreen is present in the content and <playback_screen> is defined the renderer will use these definitions when interpreting the screenEdgeLock and screenRef metadata features. See section 3.3.7 "Screen Scaling" for details of the valid range of screen positions in the baseline rendering algorithm, and how the screenRef metadata is applied. Section 3.3.8 "Screen Edge Lock" below describes the application of the screenEdgeLock flag.
    3. 3. Screen Speaker locations - The renderer accepts two speaker locations which are used to define the M+SC and M-SC speaker azimuths (for use in System G).
    2.1.1 Limitations and Exclusions on Inputs
  • The renderer (e.g., baseline renderer) supports a subset of the formats and features specified by ADM. In limiting the ADM input format the focus has been on defining new Object, DirectSpeaker and HOA behavior as these represent the core of the new experiences enabled by ADM. Matrix content and Binaural content are not addressed by the baseline renderer.
  • Additionally, structures in ADM aimed at supporting the cataloguing and compositing of multiple elements are also set aside in the baseline renderer, in favor of describing the rendering process for the programme elements themselves.
  • The ADM input content and format must conform to the reduced UML model illustrated in Fig. 4 , which an example of an input ADM format. This subset of the full model is sufficient to express all the features supported in the renderer (e.g., baseline renderer). If the input metadata contains objects and references between objects beyond those depicted in the UML diagram above, such metadata shall be ignored by the renderer.
  • For simplicity, the renderer will only attempt to parse the first audioPackFormatIDRef that it encounters inside an audioObject. Therefore, it is recommended that an audioObject only reference a single audioPackFormat. The renderer will also assume that audioObjects persist throughout the duration of the audioProgramme (i.e., audioObject start time will be assumed to be 0 and duration attributes shall be ignored). This implies that the list of Track Numbers in the BWF File .chna chunk must be non-repeating, as shown in Fig. 4 .
  • A common audioPackFormat reference in an audioObject instance shall be interpreted by the renderer to indicate the speaker layout that was used during content creation. Only one reference to an audioPackFormat from the common definitions is therefore allowed to exist in the file. However, multiple instances of non-common audioPackFormats may be present.
  • It is worth noting that, as specified in BS.2076, an audioStreamFormat instance may refer to either an audioPackFormat or audioChannelFormat instance, but not both. However, if an audioStreamFormat instance refers to audioPackFormat, but not audioTrackFormat, the renderer loses the ability to link an audio track to the specific audioChannelFormat instance containing its metadata. Therefore, while audioPackFormat instances may be present in the .xml chunk, they shall not be referenced from audioStreamFormat instances. The renderer shall associate audio tracks to their corresponding audioPackFormat (if any) through the audioPackFormat reference in the .chna chunk.
  • Finally all audio data is assumed to be presented as un-encoded PCM waveform data for the purpose of describing the rendering algorithms. It is recommended that encoded sources are decoded and aligned as a pre-step to the rendering stage in order to avoid timing complexities introduced when combining decoding and rendering into a single stage of processing.
  • 2.2 Outputs
  • The output from the renderer (e.g., baseline renderer) may be passed through a B-chain for reproduction in a studio environment. Alternatively, the output could be captured as new ADM content, however before writing to a file the signal overload protection (i.e., peak limiting) which the B-chain would provide in a studio environment may need to be simulated in software. If the output is captured as ADM, it is recommended that it should only contain common audioObjectIDs, matching the waveform information to the BS.2051-0 speaker configuration specified. Fig. 5 illustrates the reduced model which the output of the renderer may conform to as an example of the output ADM format. This output may be ready for presentation to a reproduction system which conforms to what is specified in Recommendation ITU-R BS.1116. It is recommended that reproduction systems used to evaluate rendered ADM content are calibrated to provide level and time alignment within 0.25 dB and 100 µs respectively at the listening position.
  • 2.3 Renderer Architecture
  • An example of the system architecture of the renderer (e.g., baseline renderer) 600 is schematically illustrated in Fig. 6 .
  • The renderer 600 is constructed in three major blocks:
    • ADM reader 300
    • Scene Renderer 200
    • Object and Channel Renderer 100
  • The ADM reader 300 parses ADM content 10 to extract the metadata 25 into an internal representation and aligns the metadata 25 with associated audio data 20 to feed, in blocks, to the rendering engines. The ADM reader 300 also validates the metadata 25 to ensure a consistent and complete set of metadata is present, for example the ADM reader 300 ensures all components of an HOA scene are present before attempting to render the scene.
  • The scene renderer 200 consumes scene-based channels and renders them to the desired speaker layout. Details of the scene formats supported by the renderer and the rendering methods are detailed in section 4 "Scene Renderer" below.
  • The object and channel renderer 100 consumes DirectSpeaker channels and Object channels and renders them to the desired speaker layout. Details of the metadata features supported by the baseline renderer and the rendering methods are detailed in section 3 "Channel and Object Renderer" below. The speaker renders created by the two render stages are mixed (summed) at mixing stage 400 and the resulting speaker feeds are passed to the reproduction system 500.
  • 2.4 System Characteristics 2.4.1 Latency
  • The renderer algorithm (e.g., baseline renderer algorithm) adds no latency to the audio signal path.
  • When integrated into an environment where metadata is being fed into the renderer through a console, or other control surface, the maximum delay between the time when the metadata is presented to the rendering algorithm, and when its effect is represented on the output may be 64 samples.
  • The delay incurred between the control surface and the renderer depends on the hardware/software integration encapsulating the baseline renderer, and the delay incurred after the output is updated before it is reproduced by the speakers depends on the latency of the B-chain processing and the software/hardware interfaces linking the system to the speakers. These delays should be minimized when integrating the renderer into a studio environment.
  • 2.4.2 Sampling Rates
  • The renderer algorithm (e.g., baseline renderer algorithm) described in this document supports ADM content with homogenous sampling rates. It is recommended that content with mixed sampling rates be converted to the highest common sampling rate and aligned as a pre-step to the rendering stage in order to avoid timing complexities introduced when combining sample rate conversion and rendering into a single stage of processing.
  • 2.4.3 Metadata Update Rate
  • In order to manage the computational and algorithm complexity which would otherwise come with arbitrary metadata update times, all changes to metadata may be applied at 32 sample-spaced boundaries. Updates to the mixing matrices are not limited to the 32 sample boundaries and may be updated on a per-sample basis-section 3.4 "Ramping Mixer" below details how the mixing matrices may be updated and applied in the channel and object renderer.
  • 3. CHANNEL AND OBJECT RENDERER 3.1 Architecture
  • An example of the system architecture of the object and channel renderer (embodying an example of an apparatus for rendering input audio for playback in a playback environment) 100 is schematically illustrated in Fig. 7 . The object and channel renderer 100 comprises a metadata pre-processor (embodying an example of a metadata processing unit) 110, a source panner 120, a ramping mixer 130, a diffuse ramping mixer 140, a speaker decorrelator 150, and a mixing stage 160. The object and channel renderer 100 may receive metadata (e.g., ADM metadata) 25, audio data (e.g., PCM audio data) 20, and optionally a speaker layout 30 of the reproduction environment as inputs. The object and channel renderer 100 may output one or more speaker feeds 50.
  • The metadata pre-processor 110 converts existing direct speaker and dynamic object metadata, implementing the channelLock, divergence and screenEdgeLock features. It also takes the speaker layout 30 and implements the zoneExclusion metadata features to create a virtual room.
  • The Source Panner 120 takes the new virtual source metadata, and virtual room metadata and pans the sources to create speaker gains, and diffuse speaker gains. The source panner 120 may implement the extent and diffuseness features respectively described in section 3.2.2 "Rendering Object Locations with Extents" and section 3.2.5 "Diffuse" below.
  • The Ramping Mixer 130 mixes the audio data 20 with the speaker gains to create the speaker feeds 50. The ramping mixer 130 may implement the jumpPosition feature. There are two ramping mixer paths. The first path implements the direct speaker feeds, while the second path implements the diffuse speaker feeds.
  • In the case of the Diffuse Ramping Mixer 140, the per-object gains are speaker independent, so the diffuse ramping mixer 140 produces a mono downmix. This downmix feeds the Speaker Decorrelator 150 where the diffuse speaker dependent gains are applied. Finally the two paths are mixed together at the mixing stage 160 to produce the final speaker feeds.
  • The source panner 120 and the ramping mixer(s) 130, 140, and optionally the speaker decorrelator 150 may be said to form a rendering unit.
  • 3.2 Source Panning
  • An example of the system architecture of the source panner 120 is schematically illustrated in Fig. 8 . The source panner 120 comprises a point panner 810, an extent panner (size panner) 820 and a diffusion block (diffusion unit) 830. The source panner 120 may receive the virtual sources 812 and virtual rooms 814 as inputs. Outputs 832, 834, 836 of the source panner 120 may be provided to the ramping mixer 130, the diffuse ramping mixer 140, and the speaker decorrelator 150, respectively.
  • In more detail, the source panner 120 receives the pre-processed objects, and virtual room metadata from the metadata pre-processor 110, and first pans them to speaker gains, assuming no extent or diffusion using the point panner 810. The resulting speaker gains are then processed by the extent panner 820, adding source extent and producing a new set of speaker gains. Finally these speaker gains pass to the diffusion block 830. The diffusion block 830 maps these gains to speaker gains for the ramping mixer 130, the diffuse ramping mixer 140 and the speaker decorrelator 150.
  • 3.2.1 Rendering Point Objects
  • The purpose of the point panner 810 is to calculate a gain coefficient for each speaker in the output speaker layout, given an object position. The point panning algorithm may consist of a 3D extension of the 'dual-balance' panner concept that is widely used in 5.1- and 7.1-channel surround sound production. One of the main requirements of the point panner 810 is that it is able to create the impression of an auditory event at any point inside the room. The advantage of using this approach is that it provides a logical extension to the current surround sound production tools used today.
  • The inputs to the point panner 810 comprise (e.g., consist of) an object's position [pox,poy,poz ] and the positions of the output speakers, all in Cartesian coordinates, for example. Let [psx (j),psy (j),psz (j)] denote the position of the j-th speaker. Let N denote the number of speakers in the layout.
  • With regards to speaker layout, the point panner 810 requires that the following conditions are satisfied in order to be able to accurately place a phantom image of the object anywhere in the room (i.e., in the playback environment):
    • The speakers must be grouped into one or more discrete planes in the z-dimension.
    • The speakers on each plane must be grouped into one or more discrete rows in the y-dimension.
    • There must be two or more speakers on every row and there must be speakers at x = 1 and x = -1.
    • Every speaker location must lie on the surface of the room cube, that is, either on the floor, ceiling, or walls.
  • The coordinate transformations described in section 3.3.2 "Object and Channel Location Transformations" below result in mapping all the ITU-R BS.2051 speaker layouts of interest to meet these requirements-the resulting speaker locations are set out in Appendix A.
  • The point panner 810 works with any number of speaker planes, but for simplicity and without loss of generality, the algorithm will be described using an output layout consisting of three speaker planes: the bottom or floor speaker plane at z = -1, the middle plane at z = 0, and the upper or ceiling plane at z = 1.
    Step 1: Determine the two planes that will be used to pan the object.
       /* assumptions: -1 <= p_oz <= 1 */
       if (p_oz < 0)
       {
           z(1) = -1;
           z(2) = 0;
       } else if (p_oz >= 0) {
           z(1) = 07
           z(2) = 1;
       }
Step 2: Group speakers by plane, applying the object's zone exclusion mask (see section 3.3.3 "Zone Exclusion" below).
    • Let j = {1,2,...,N} be the set of speaker indices.
    • Construct a set of speaker indices for each plane:
      For i = 1 to 2 k i = j : p sz j = z i mask o j = 1
      Figure imgb0002
    Step 3: For each plane i, find the speakers lying in rows just in front of the object and just behind the object.
    For i = 1 to 2 k i + = k i : p sy k i p oy 0
    Figure imgb0003
    k i = k i : p sy k i p oy < 0
    Figure imgb0004
    r i + = arg min k i + p sy k i + p oy
    Figure imgb0005
    r i = arg min k i p sy k i p oy
    Figure imgb0006

    Observe that for each plane i, r i + + r i
    Figure imgb0007
    is either 1 or 2. In other words, an object is either between two rows of speakers, exactly over a row of speakers, or between one row of speakers and a wall.
    Step 4: For each row found in step 3, find the closest speakers to the left and right of the object.
    For i = 1 to 2 idx i 1 = arg min r i + p sx r i + : p sx r i + p ox 0 p ox
    Figure imgb0008
    idx i 2 = arg min r i + p sx r i + : p sx r i + p ox < 0 p ox
    Figure imgb0009
    idx i 3 = arg min r i p sx r i : p sx r i p ox 0 p ox
    Figure imgb0010
    idx i 4 = arg min r i p sx r i : p sx r i p ox < 0 p ox
    Figure imgb0011

    Observe that 1 ≤ ∑ n |idx(i,n)| ≤ 4, meaning that for each speaker plane, at most four speakers will be selected for panning.
    Step 5: Compute the gains G(j) for each speaker j.
    Figure imgb0012
    Figure imgb0013
    It is worth noting that the sum of the squares of the speaker gains will always be 1, i.e., the panning operation is energy preserving. 3.2.2 Rendering Object Locations with Extents
  • The purpose of the extent panner 820 is to calculate a gain coefficient for each speaker in the output speaker layout, given an object position and object extent (e.g., object size). The intention of extent (e.g., size) is to make the object appear larger so that when the extent is at the maximum the object fills the room, while when it is set to zero the object is rendered as a point object.
  • To achieve this, the extent panner 820 considers a grid (e.g., a three-dimensional rectangular grid) of many virtual sources in the room. Each virtual source fires speakers exactly in the same way any object rendered with the point panner 810 would. The extent panner 820, when given an object position and object extent, determines which (and how many) of those virtual sources will contribute. That is, candidates for the contributing virtual sources may be arranged in a grid (e.g., a three-dimensional rectangular grid) across the playback environment (e.g., room).
  • 3.2.2.1 Algorithm Overview
  • Fig. 24 is a flowchart schematically illustrating an example of a method (e.g., algorithm) for rendering object locations with extents as an example for a method of rendering input audio for playback in a playback environment. The input audio includes at least one audio object and associated metadata. The associated metadata indicates (e.g., specifies) at least a location (e.g., position) of the at least one audio object and a three-dimensional extent (e.g., size) of the at least one audio object. The method comprises rendering the audio object to one or more speaker feeds in accordance with its three-dimensional extent. This may be achieved by the following steps:
    At step S2410, locations of a plurality of virtual audio objects (virtual sources) within a three-dimensional volume defined by the location of the audio object and its three-dimensional extent are determined. Determining said locations may involve imposing a respective minimum extent for the audio object in each of the three dimensions (e.g., {x,y,z} or {θ,ϕ,r}). Further, said determining may involve selecting a subset of locations of (active) virtual audio objects among a predetermined set of fixed potential locations of virtual audio objects in the reproduction environment. The fixed potential positions may be arranged in a three-dimensional grid, as explained below. At step S2420, a weight factor is determined for each virtual audio object that specifies the relative importance (e.g., relative weight) of the respective virtual audio object. Notably, the "relative importance" dealt with in this section is not to be confused with the metadata feature relating to <importance> and <obj_importance> described in section 3.3.9 "Importance" below. At step S2430, the audio object and the plurality of virtual audio objects are rendered to the one or more speaker feeds in accordance with the determined weight factors. Performing step S2430 results in a gain coefficient for each of the one or more speaker feeds that may be applied to (e.g., mixed with) the audio data for the audio object. The audio data for the audio object may be the audio data (e.g., audio signal) of the original audio object. Step S2430 may comprise the following further steps:
    • Step 1: Calculate point gains for all virtual sources
    • Step 2: Combine all the gains from virtual sources within the room to produce inside extent gains (e.g., inside size gains).
    • Step 3: Combine all the gains from virtual sources on the boundaries of the room to produce boundary extent gains (e.g., boundary size gains).
    • Step 4: Combine the inside and boundary extent gains to produce the final extent gains (e.g., final size gains).
    • Step 5: Combine the final extent gains with the gains (e.g., point gains) for the object (e.g., the gains for the object that would result when assuming zero extent for the object).
  • An apparatus (rendering apparatus, renderer) for rendering input audio for playback in a playback environment (e.g., for performing the method of Fig. 24 ) may comprise a rendering unit. The rendering unit may comprise a panning unit and a mixer (e.g., the source panner 120 and either or both of the ramping mixer(s) 130, 140). Step S2410, step S2420 and step S2430 may be performed by the rendering unit.
  • In general, the method may comprise steps S2510 and S2520 illustrated in the flowchart of Fig. 25 and steps S2610 to S2640 illustrated in the flowchart of Fig. 26 . Said steps may be said to be sub-steps of step S2430. Accordingly, steps S2510 and S2520 as well as steps S2610 to S2640 may be performed by the aforementioned rendering unit.
  • At step S2510, a gain is determined, for each virtual audio object and for each of the one or more speaker feeds, for mapping the respective virtual audio object to the respective speaker feed. These gains may be the point gains referred to above. At step S2520, respective gains determined at step S2510 are scaled, for each virtual object and for each of the one or more speaker feeds, with the weight factor of the respective virtual audio object.
  • At step S2610, a first combined gain is determined for each speaker feed depending on the gains of those virtual audio objects that lie within a boundary of the playback environment (e.g., room). The first combined gains determined at step S2610 may be the inside extent gains (one for each speaker feed) referred to above. At step S2620, a second combined gain is determined for each speaker feed depending on the gains of those virtual audio objects that lie on said boundary. The second combined gains determined at step S2620 may be the boundary extent gains (one for each speaker feed) referred to above. Then, at step S2630, a resulting gain for the plurality of virtual audio objects is determined for each speaker feed based on the first combined gain, the second combined gain, and a fade-out factor indicative of the relative importance of the first combined gain and the second combined gain. The resulting gains determined at step S2630 may be the final extent gains (one for each speaker feed) referred to above. The fade-out factor may depend on the three-dimensional extent of the audio object and the location of the audio object. For example, the fade-out factor may depend on a fraction of the overall extent of the audio object that is within the boundary of the playback environment (e.g., the fraction of the overall three-dimensional volume of the audio object that is that is within the boundary of the playback environment). The first and second combined gains may be normalized before performing step S2630. Finally, at step S2640, a final gain is determined for each speaker feed based on the resulting gain for the plurality of virtual audio objects, a respective gain for the audio object, and a cross-fade factor depending on the three-dimensional extent of the audio object. This may relate to combining the final extent gains with the point gains for the object.
  • 3.2.2.2 Algorithm Detail
  • Next, details of the algorithm described with reference to Fig. 24 , Fig. 25 , and Fig. 26 will be described.
  • As a first step, which is an optional step, the extent value (e.g., size value) may be scaled up to a larger range. That is, the first step may be to scale up the ADM extent value to a larger range. The user is exposed to extent values s ∈ [0,1], which may be mapped into the actual extent used by the algorithm to the range [0, 5.6]. The mapping may be done by a piecewise linear function, for example a piecewise linear function defined by the value pairs (0, 0), (0.2, 0.6), (0.5, 2.0), (0.75, 3.6), (1, 5.6), as shown in Fig. 9 . The maximum value of 5.6 ensures that when extent is set to maximum, it truly occupies the whole room. In what follows, the variables s x ^ ,
    Figure imgb0014
    s y ^ ,
    Figure imgb0015
    s z ^ ,
    Figure imgb0016
    refer to the extent values after conversion. Notably, each of the three dimensions of the extent may be independently controlled when employing the presently described method.
  • To maintain desired behavior, extent should only be applied if s x ^ 2 N x 1 s y ^ 2 N y 1
    Figure imgb0017
    s z ^ 2 N z 1 .
    Figure imgb0018
    Accordingly, the renderer may clip (i.e., increase) small, non-zero extent values to respective minimum values as needed. That is, determining said locations at step S2410 may involve imposing a respective minimum extent for the audio object in each of the three dimensions (e.g., {x,y,z} or {θ,ϕ,r}). For example, minimum values may be enforced on s x ^ ,
    Figure imgb0019
    s y ^ ,
    Figure imgb0020
    s z ^
    Figure imgb0021
    as follows: s x = max s x ^ 2 N x 1 , s y = max s y ^ 2 N y 1 , s z = max s z ^ 2 N z 1 .
    Figure imgb0022
    These restricted values sx,sy,sz may be used throughout the algorithm, except for the computation of effective size seff below, which uses the unrestricted values s x ^ ,
    Figure imgb0023
    s y ^ ,
    Figure imgb0024
    s z ^ .
    Figure imgb0025
  • The grid of virtual sources referred to in step S2410 may be defined as a static rectangular uniform grid of Nx × Ny × Nz points. The grid may span the range of positions [-1, 1] in each dimension. That is, the grid may span the entire reproduction environment (e.g., room). The density may be set in a manner that includes a few sources between loudspeakers in a typical layout. Empirical testing showed that Nx = Ny = 20, Nz = 8 or Nx = Ny = 20, Nz = 16 created an appropriate grid of virtual sources. For loudspeaker layouts where there are no bottom layer loudspeakers (all layouts except Systems E and H), the range of virtual sources in the z dimension may be limited to [0, 1], and the recommended value of Nz is 8. The notation (xs,ys,zs ) will be used to denote the possible coordinates of the virtual sources. Each virtual source creates a set of gains g j point x s y s z s
    Figure imgb0026
    to each speaker j = 1,...,NJ of the layout (i.e., each speaker in the reproduction environment).
  • The object position and extent (xo,yo,zo,sx,sy,sz ) may be used to calculate a set of weights that determine how much each virtual source will contribute to the final gains. Accordingly, the set of weights may be determined based on the object position (location) and extent. This calculation may be performed at step S2420. For loudspeaker layouts where there are no loudspeakers in the bottom layer (e.g., all loudspeaker layouts listed in ITU-R BS.2051-0, except for System E and System H), the extent algorithm may use zo = max(poz ,0) as the object's position in the z dimension. Otherwise, zo = poz . For all loudspeaker layouts, the extent algorithm may use the same x and y position as the point source panner (i.e., yo = poy, xo = pox ). The weights for each virtual source are denoted w(xs ,ys,zs ,xo ,yo ,zo,sx,sy,sz ) and may be used to scale the gains (e.g., point gains) for each virtual source at step S2520. The gains (e.g., point gains) may have been determined at step S2510. Virtual sources with zero weight may be considered as not having been selected at step S2410, i.e., their locations are not among the locations determined at step S2410.
  • After being weighted, all the virtual source gains are summed together at step S2610 which produces the inside extent gains (first combined gains): g j inside x o y o z o s x s y z z = x s , y s , z s w x s y s z s x o y o z o s x s y s z × g j point x s y s z s
    Figure imgb0027
    where index j indicates respective speaker feeds.
  • However, the extent algorithm may alternatively combine virtual source gains in a way that varies depending on the extent of the object. In general, this can be described as: g i inside x o y o z o s x s y s z = x x , y s , z s w x s y s z s x o y o z o s x s y s z × g j point x s y s z s p 1 p
    Figure imgb0028
    The extent-dependent exponent p controls the smoothness of the gains across loudspeakers. It ensures homogeneous growth of the object at small extent value s, and correct energy distribution across all directions at large extent value s. The extent-dependent exponent p may be determined (e.g., calculated) as follows: First sort s x ^ s y ^ s z ^
    Figure imgb0029
    in descending order, and label the resulting ordered triad as {s 1,s 2,s 3}. The triad can then be combined to give an effective extent (e.g., effective size), for example via: s eff = 6 9 s 1 + 2 9 s 2 + 1 9 s 3
    Figure imgb0030
    For layouts with a single plane of loudspeakers, such as ITU-R BS.2051-0 System B, first sort s x ^ s y ^
    Figure imgb0031
    in descending order, and label the resulting ordered pair as {s 1,s 2}. The effective extent in this case is for example given by: s eff = 3 4 s 1 + 1 4 s 2 .
    Figure imgb0032
    For loudspeaker layouts with only two loudspeakers, such as ITU-R BS.2051-0 System A, seff = s x ^ ,
    Figure imgb0033
    for example.
  • The effective extent may then be used to calculate a piecewise defined exponent, for example via:
    p = 6, if seff ≤ 1.0 p = 6 + s eff 1.0 s max 1.0 4 , if s eff > 1.0
    Figure imgb0034
    where smax = 5.6, such that when s is at its maximum, p = 2.
  • In the above, some simplifications can be made. The first is that gains (e.g., point gains) can be separated into gains in each axis (i.e., one for each of the x axis, y axis, and z axis), for example via: g j point x y z = g j point x × g j point y × g j point z
    Figure imgb0035
  • The weight function can also treat each axis separately and the whole extent computation simplifies. For example, the weight functions can be separated via: w x s y s z s x o y o z o s x s y s z = w x s x o s x w y s y o s y w z s z o s z
    Figure imgb0036
    The chosen weight functions may look like something between circles and squares (or spheres and cubes, in 3D). For example, the weight functions may be given by: w x s x o s x = 10 3 2 x s x o s x 4
    Figure imgb0037
    w y s y o s y = 10 3 2 y s y o s y 4
    Figure imgb0038
    w z s z o s z = 10 3 2 z s z o s z 4
    Figure imgb0039
  • Using the above simplifications, the inside extent gains g j inside
    Figure imgb0040
    (first combined gains) can be simplified to g j inside x o y o z o s x s y s z = f j x x o s x f j y y o s y f j z z o s z
    Figure imgb0041
    where f j x x o s x = x s g j point x s w x s x o s x p
    Figure imgb0042
    f j y y o s y = y s g j point y s w y s y o s y p
    Figure imgb0043
    f j z z o s z = z s g j point z s w z s z o s z p
    Figure imgb0044
    For layouts with a single plane of loudspeakers, such as ITU-R BS.2051-0 System B, f j z z o s z =
    Figure imgb0045
    1 may be used. For loudspeaker layouts with only two loudspeakers, such as ITU-R BS.2051-0 System A, f j y y o s y = f j z z o s z = 1
    Figure imgb0046
    may be used.
  • Further, a normalization step may be applied to g j inside ,
    Figure imgb0047
    i.e., the first combined gains may be normalized. For example, said normalization may be performed according to: g j inside = g j inside n g n inside 2 , if n g n inside 2 > tol
    Figure imgb0048
    g j inside = g j inside tol ,
    Figure imgb0049
    otherwise,
    where indices j and n indicate respective speaker feeds, and tol is a small number preventing division by zero, e.g., tol = 10-5
  • One further modification that may be made is that, for aesthetic reasons, it is important to have a mode where there is no opposite loudspeaker firing. This is accomplished by using virtual sources located only on the boundary. To handle certain loudspeaker layouts as special cases, we set dim = 1 for ITU-R BS.2051-0 System A, dim = 2 for System B, dim = 4 for Systems E and H, and dim = 3 otherwise in the calculations below.
  • Accordingly, at step S2620 boundary extent gains g j bound
    Figure imgb0050
    (second combined gains) may be determined depending on the gains of those virtual sources that lie on the boundary of the reproduction environment (e.g., room). For example, the boundary extent gains may be determined via: g j bound x o y o z o s x s y s z = b j floor z o s z f j x x o s x f j y y o s y + b j ceil z o s z f j x x o s x f j y y o s y + b j left x o s x f j y y o s y f j z z o s s + b j right x o s x f j y y o s y f j z z o s z + b i front y o s y f j x x o s x f j z z o s z + b j back y o s y f j x x o s x f j z z o s z
    Figure imgb0051
    where b j floor z o s z = { g j point z s = 1.0 w z s = 1.0 , z o , s z p , if dim = 4 0 otherwise
    Figure imgb0052
    b j ceil z o s z = { g j point z s = 1.0 w z s = 1.0 , z o , s z p , if dim 3 0 otherwise
    Figure imgb0053
    b j left x o s x = g j point x s = 1.0 w x s = 1.0 , x o , s x p
    Figure imgb0054
    b j right x o s x = g j point x s = 1.0 w x s = 1.0 , x o , s x p
    Figure imgb0055
    b j front y o s y = { g j point y s = 1.0 w y s = 1.0 , y o , s y p if dim > 1 0 , otherwise
    Figure imgb0056
    b j back y o s y = { g j point y s = 1.0 w y s = 1.0 , y o , s y p if dim > 1 0 , otherwise
    Figure imgb0057
  • Further, a normalization step may be applied to the boundary extent gains g j bound ,
    Figure imgb0058
    i.e., the second combined gains may be normalized. For example, said normalization may be performed according to: g j bound = g j bound n g n bound 2 , if n g n bound 2 > tol
    Figure imgb0059
    g j bound = g j bound tol ,
    Figure imgb0060
    otherwise.
  • The boundary extent gains (second combined gains) may now be combined with the inside extent gains (first combined gains). To do so, a fade-out factor may be introduced for all virtual sources inside the room, with fade-out amount = 'fraction of object outside the room'. In general, the fade-out factor may indicate a relative importance of the inside extent gains and boundary extent gains. The fade-out factor may depend on the location and extent of the audio object. Combination of the inside extent gains and boundary extent gains may be performed at step S2630. For example, the combination may be performed via: g j size = g j bound + µ × g j inside 1 p
    Figure imgb0061
    where g j size
    Figure imgb0062
    denotes the final extent gains (resulting gains), d bound = { min x o + 1 , 1 x o , if dim = 1 min x o + 1 , 1 x o , y o + 1 , 1 y o , if dim = 2 min x o + 1 , 1 x o , y o + 1 , 1 y o , z o + 1 , 1 z o , otherwise
    Figure imgb0063
    µ = { h x o s x 3 , if dim = 1 h x o s x h y o s y 3 2 , if dim = 2 h x o s x h y o s y h z o s z , otherwise
    Figure imgb0064
    and h(c,s) is a fade out function for a single dimension. For example, h(c,s) may be given by: h c s = max s 0.4 3 0.16 s 1 3 , if d bound s and d bound 0.4
    Figure imgb0065
    h c s = d bound d bound 0.4 2 1 3 , otherwise
    Figure imgb0066
  • In general, the fade-out factor may be determined such that, as part of the sized object starts moving outside the room, all virtual sources inside the object start fading out, except for those at the boundaries. When an object reaches a boundary only the boundary gains will be contributing to the extent gains. In the above, dbound may be the minimum distance to a boundary.
  • Further, a normalization step may be applied to the final extent gains g j size
    Figure imgb0067
    (resulting gains). For example, said normalization may be performed according to: g j size = g j size n g n size 2 , if n g n size 2 > tol
    Figure imgb0068
    g j size = g j size tol ,
    Figure imgb0069
    otherwise.
  • The extent contributions (i.e., final extent gains) may then be combined with the gains for the audio object (e.g., point gains of the audio object-assuming zero extent for the audio object), and a crossfade between them may be applied as a function of extent. Combination of the final extent gains and the gains of the audio object may be performed at step S2640 and may result in a set of final gains (total gains), one for each speaker feed. For example, the combination may be performed via: g j total = α × g j point x o y o z o + β × g j size
    Figure imgb0070
    where
    • for seff < sfade, α = cos s eff s fade × π 2 ,
      Figure imgb0071
      β = sin s eff s fade × π 2
      Figure imgb0072
    • for seff sfade, α = 0, β = 1
    and sfade = 0.4. In general, the cross-fade factor may depend on the extent (e.g., effective extent) of the audio object. This ensures smooth panning and smooth growth of the object, providing a nice transition all the way between the smallest and the largest possible extents.
  • Finally, a last normalization may be applied to the final gains. For example, said normalization may be performed according to: G j s = g j total n g n total 2 , if n g n total 2 > tol
    Figure imgb0073
    G j s = g j total tol ,
    Figure imgb0074
    otherwise.
  • The final gains G j s
    Figure imgb0075
    may be provided to the diffusion block 830 if present, or otherwise directly to the ramping mixer 130. The final gains may be the outcome of the rendering at step S2430.
  • 3.2.2.3 Spherical Coordinate System
  • For an object with position metadata specified in spherical coordinates, its location may be transformed to Cartesian coordinates using the mapping function MapSC (), described in section 3.3.2 "Object and Channel Location Transformations" below. Before transforming the location, any associated extent metadata given in spherical coordinates (i.e., width, height, and depth ADM parameters, in degrees) may be first converted into appropriate Cartesian extent metadata (i.e., X-width, Y-width, Z-width ADM parameters, e.g., in the range [0, 1]) that can be used by the extent panner described in section 3.2.2 "Rendering Object Locations with Extents".
  • Extent metadata may be converted from spherical to Cartesian coordinates by finding the size of a cuboid that encompasses the angular extents. The Cartesian cuboid can be found by determining the extremities in each dimension of the shape described by the spherical extent angles and depth. Two examples are shown in Fig. 10A and Fig. 10B , limited to the x and y plane, for simplicity. Fig. 10A illustrates the case of an extent defined by acute angles, and Fig. 10B illustrates the case of an extent defined by obtuse angles. The distance will be halved to match the range of extent given in the Cartesian coordinate system and these parameters can then be used by the extent panner to render an object.
  • In general terms, a method for converting the extent from spherical coordinates to Cartesian coordinates may comprise the steps illustrated in the flowchart of Fig. 27 . This method is applicable to any audio object whose associated metadata indicates a first three-dimensional extent (e.g., size) of the audio object in a spherical coordinate system by respective ranges of values for a radius, an azimuth angle, and an elevation angle. At step S2710, a second three-dimensional extent (e.g., size) in a Cartesian coordinate system is determined as dimensions (e.g., lengths along the X, Y, and Z coordinate axes, i.e., X-width, Y-width, and Z-width) of a cuboid that circumscribes the part of a sphere that is defined by said respective ranges of the values for the radius, the azimuth angle, and the elevation angle. At step S2720, the second three-dimensional extent is used as the three-dimensional extent of the audio object in the above method for rendering object locations with extents as an example for a method of rendering input audio for playback in a playback environment.
  • The aforementioned apparatus (rendering apparatus, renderer) for rendering input audio for playback in a playback environment (e.g., for performing the method of Fig. 24 ) may further comprise a metadata processing unit (e.g., metadata pre-processor 110). Step S2710 may be performed by the metadata processing unit. Step S2720 may be performed by the rendering unit.
  • The following pseudocode defines an example of an algorithm for calculating X-width, Y-width, and Z-width from spherical width, height, and depth:
  •        function (x_width, y_width, z_width)
               = extent_spher2cart(r, az, el, width, height, depth)
           {
                r_min = max(0, r - depth)
                r_max = min(1, r + depth)
                el_min = el - height / 2
                el_max = el + height / 2
                az_min = az - width / 2
                az_max = az + width / 2
           //z_width: find max width of spherical elevation arc
                el_min_z = el_min
                el_max_z = el_max
                if (el_min_z < -90 && el_max_z > -90)
                {
                  el_min z = -90
                }
                if (el_max_z > 90 && el_min_z < 90)
                {
                  el_max_z = 90
                }
                (∼, ∼, z1) = s_to_c(r_max, 0, el_min_z)
                (∼, ∼, z2) = s_to c(r_min, 0, el_min_z)
                (∼, ∼, z3) = s_to_c(r_max, 0, el_max_z)
                (∼, ∼, z4) = s_to_c(r_min, 0, el_max_z)
                z_width = absrange(z1, z2, z3, z4) / 2
           //x width: find maximum x-width of spherical width arcs
           //(consider one width arc at each elevation and depth extremity)
                (az_min_x, az_max_x) = clip_angles(az_min, az_max, -90)
                (az_min_x, az_max_x) = clip_angles(az_min_x, az_max_x, 90)
                (az_min_x, az_max_x) = clip_angles(az_min_x, az_max_x, 270)
                (az_min_x, az_max_x) = clip_angles(az_min_x, az_max_x, -270)
                x1 = s_to_c(r_max, az_min_x,el_max)
                x2 = s_to_c(r_max, az_max_x,el_max)
                x3 = s_to_c(r_min, az_min_x,el_max)
                x4 = s_to_c(r_min, az_max_x,el_max)
                x5 = s_to_c(r_max, az_min_x,el_min)
                x6 = s_to_c(r_max, az_max_x,el_min)
                x7 = s_to_c(r_min, az_min_x,el_min)
                x8 = s_to_c(r_min, az_max_x,el_min)
                x9 = s_to_c(r_max, az_min_x,el)
                x10 = s_to_c(r_max, az max_x,el)
                x11 = s_to_c(r_min, az_min_x,el)
                x12 = s_to_c(r_min, az_max_x,el)
                ×_width = absrange(x1, x2, x3, x4, x5, x6
                             , x7, x8, x9, x10, x11, x12)/2
           //y_width: find maximum y-width of spherical width arcs
                (az_min_y, az_max_y) = clip_angles(az_min, az_max, 0)
                (az_min_y, az_max_y) = clip_angles(az_min_y, az_max_y, 180)
                (az_min_y, az_max_y) = clip_angles(az_min_y, az_max_y, -180)
                (∼,y1) = s_to_c(r_max, az min_y,el_max)
                (∼,y2) = s_to_c(r_max, az_max_y,el_max)
                (∼,y3) = s_to_c(r_min, az_min_y,el_max)
                (∼,y4) = s_to_c(r_min, az_max_y,el_max)
                (∼,y5) = s_to_c(r_max, az_min_y,el_min)
                (∼,y6) = s_to_c(r_max, az_max_y,el_min)
                (∼,y7) = s_to_c(r_min, az_min_y,el_min)
                (∼,y8) = s_to_c(r_min, az_max_y,el_min)
                (∼,y9) = s_to_c(r_max, az_min_y,el)
                (∼,y10) = s_to_c(r_max, az_max_y,el)
                (∼,y11) = s_to_c(r_min, az_min_y,el)
                (∼,y12) = s_to_c(r_min, az_max_y,el)
                y_width = absrange(y1, y2, y3, y4, y5, y6
                             , y7, y8, y9, y10, y11, y12)/2
           }
           function (mintheta, maxtheta)
               = clip angles(mintheta, maxtheta, thresh)
           {
                if (mintheta <= thresh && maxtheta >= thresh)
                {
                  if(abs(mintheta-thresh) < abs(maxtheta-thresh))
                  {
                      mintheta = thresh
                  } else {
                      maxtheta = thresh
                  }
                }
           }
           function y = absrange(x)
           {
                y = max(x) - min(x)
           }
           function (x, y, z) = s_to_c(r, az, el)
           {
               x = r ∗ cos(el) ∗ cos(az+90)
               y = r ∗ cos(el) ∗ sin(az+90)
               z = r ∗ sin(el)
           }
  • 3.2.3 Rendering Direct Speakers
  • When processing channel-based content (i.e., audioChannelFormat instances of type 'DirectSpeakers'), a renderer must strive to achieve two potentially conflicting outcomes:
    • The audio is panned entirely to a single output speaker.
    • The audio is reproduced at a position that is similar to the position that was auditioned during content creation.
  • These outcomes are especially difficult to achieve because the renderer might be configured to use an output speaker layout that differs from the layout that was used to create the content.
  • To find a reasonable balance between the above two criteria over possibly mismatched speaker layouts, the renderer takes the following strategy to render channel-based content:
    • If the channel's ID matches one of the common audioChannelFormat definitions, the channel is assigned a position equal to the nominal position of that speaker channel as per the ITU-R BS.2051-0 specification.
    • If the channel's position is specified in Cartesian coordinates, the position is not modified, and passed directly to the renderer in Cartesian coordinates.
    • If the channel's ID does not match one of the common channel definitions, and its position inside the active audioBlockFormat sub-element is specified in spherical coordinates, the metadata pre-processor 110 (see section 3.1 "Architecture") will:
      • ∘ Inspect the channel conversion table (Table 1 through Table 4) corresponding to the current output speaker configuration. If the channel's azimuth and elevation falls within one of the ranges listed, change the channel's position to be the nominal position given on the table. Otherwise, leave the channel's position as is.
      • ∘ Convert the channel's position from spherical to Cartesian coordinates, using the conversion function MapSC () specified in section 3.3.2 "Object and Channel Location Transformations" below.
    • The channel is panned to its (possibly modified) position using the point panner 810.
  • The position ranges specified in the Tables 1 to 4 below were derived from the ranges specified in ITU-R BS.2051-0 for Sound Systems B, F, G, and H. Because the specification gives no ranges to the speakers in Systems A, C, D, and E, the ranges for the System B surround speakers are used for all these systems, but the upper-layer speakers in systems C, D, and E are given no ranges (i.e., they will always be panned to the position specified in the metadata). In the case of System F, the M+/-90 and M+/-135 speakers overlap in azimuth range, so a boundary between them was set at the midpoint of +/-112.5 degrees azimuth.
  • The position adjustment strategy defined herein ensures that channel-based content that was authored using a Sound System conformant to ITU-R BS.2051-0 will be sent entirely to the correct loudspeaker when rendered to the same system, even when there is not an exact match between the speaker positions used during content creation and during playback (because different positions were chosen within the ranges allowed by the BS.2051 specification).
  • In the case of mismatched output speaker configurations (i.e., System X was used in content creation, System Y is being used in the renderer), channel-based content will still be sent to a single loudspeaker if the position specified in metadata is within the allowed range for a speaker in the output layout. Otherwise, in order to preserve the approximate position of the sound during content creation, the channel-based content will be panned to the location specified in its metadata. Table 1- Channel Position Conversion for Systems A through E
    speakerLabel Azimuth range Elevation range Nominal azimuth Nominal elevation
    M+000 0 0 0 0
    M+030 30 0 30 0
    M-030 -30 0 -30 0
    M+110 [100, 120] [0, 15] 110 0
    M-110 [-120, -100] [0, 15] -110 0
    U+030 30 30 30 30
    U-030 -30 30 -30 30
    U+110 110 30 110 30
    U-110 -110 30 -110 30
    B+000 0 -30 0 -30
    Table 2 - Channel Position Conversion for System F
    speakerLabel Azimuth range Elevation range Nominal azimuth Nominal elevation
    M+000 0 0 0 0
    M+030 30 0 30 0
    M-030 30 0 -30 0
    M+090 [60, 112.5] 0 90 0
    M-090 [-112.5, -60] 0 -90 0
    M+135 (112.5, 150] 135 0
    M-135 [-150, -112.5) -135 0
    U+045 [30, 45] [30, 45] 45 30
    U-045 [-45, -30] [30, 45] -45 30
    UH+180 180 [45, 90] 180 45
    Table 3 - Channel Position Conversion for System G
    speakerLabel Azimuth range Elevation range Nominal azimuth Nominal elevation
    M+000 0 0 0 0
    M+030 [30, 45] 0 30 0
    M-030 [-45, -30] 0 -30 0
    M+090 [90, 110] 0 90 0
    M-090 [-110, -90] 0 -90 0
    M+135 [135, 150] 0 135 0
    M-135 [-150, -135] 0 -135 0
    M+SC N/A 0 Left screen edge (or 25 if unknown) 0
    M-SC N/A 0 Right screen edge (or -25 if unknown) 0
    U+045 [30, 45] [30, 45] 45 30
    U-045 [-45, -30] [30, 45] -45 30
    U+110 [110, 135] [30, 45] 110 30
    U-110 [-135, -110] [30, 45] -110 30
    Table 4 - Channel Position Conversion for System H
    spea kerLa bel Azimuth range Elevation range Nominal azimuth Nominal elevation
    M+000 0 [0, 5] 0 0
    M+030 [22.5, 30] [0, 5] 30 0
    M-030 [-30, -22.5] [0, 5] -30 0
    M+060 [45, 60] [0, 5] 60 0
    M-060 [-60, -45] [0, 5] -60 0
    M+090 90 [0, 15] 90 0
    M-090 -90 [0, 15] -90 0
    M+135 [110, 135] [0, 15] 135 0
    M-135 [-135, -110] [0, 15] -135 0
    M+180 180 [0, 15] 180 0
    M+SC N/A 0 Left screen edge (or 25 if unknown) 0
    M-SC N/A 0 Right screen edge (or -25 if unknown) 0
    U+000 0 [30, 45] 0 30
    U+045 [45, 60] [30, 45] 45 30
    U-045 [-60, -45] [30, 45] -45 30
    U+090 90 [30, 45] 90 30
    U-090 -90 [30, 45] -90 30
    U+135 [110, 135] [30, 45] 135 30
    U-135 [-135, -110] [30, 45] -135 30
    U+180 180 [30, 45] 180 30
    B+000 0 [-30, -15] 0 -30
    B+045 [45, 60] [-30, -15] 45 -30
    B-045 [-60, -45] [-30, -15] -45 -30
    T+000 N/A 90 N/A 90
  • 3.2.4 LFE Channels and Sub-Woofer Speakers
  • The distinction between Low Frequency Effects (LFE) channels and sub-woofer speaker feeds is subtle, and understanding this with respect to how the renderer (e.g., baseline renderer) treats LFE content requires some clarification. Recommendation ITU-R BS.775-3 has more detail and recommended use of the LFE channel.
  • Sub-woofer speakers are specialized speakers in a reproduction system with the purpose of reproducing low-frequency signals or content. They may require other signal processing (e.g. bass management, overload protection) in the B-chain of a reproduction system. As such the renderer (e.g., baseline renderer) does not include any effort to perform these functions.
  • ITU-R BS.2051-0 includes speakers labelled as LFE, which are intended to carry the audio expected to be output by sub-woofers. Similarly, ADM may contain DirectSpeaker content labelled as LFE. The baseline renderer ensures input LFE content is directed to the LFE output channels, with minimal processing. The following cases are described explicitly:
    • Speaker configuration A
      ∘ all LFE inputs are discarded, typical for stereo downmix.
    • Speaker configurations B through E and G (1 output LFE)
      ∘ all LFE inputs are mixed with unity gain to create the output LFE1.
    • Speaker configurations F and H (2 output LFEs)
      • ∘ all LFE inputs with (Azimuth < 0) or (X < 0) are mixed with unity gain to LFE1
      • ∘ all LFE inputs with (Azimuth > 0) or (X > 0) are mixed with unity gain to LFE2
      • ∘ all LFE inputs with (Azimuth = 0) or (X = 0) are mixed equally into LFE1 and LFE2 LFE 1 = 0.5 LEF in LEF 2 = 0.5 LEF in
        Figure imgb0076
  • The renderer shall consider LFE input content to be either any common audioChannelFormat with an ID equal to AC_00010004 (LFE), AC_00010020 (LFEL), or AC_00010021 (LFER), or any input audioChannelFormat of type DirectSpeakers with an active audioBlockFormat sub-element containing 'LFE' as the first three characters in its speakerLabel element.
  • 3.2.5 Diffuse
  • The associated metadata of the audio object may further or alternatively indicate (e.g., specify) a degree of diffuseness for the audio object. In other words, the associated metadata may indicate a measure of a fraction of the audio object that is to be rendered isotropically (i.e., with equal energies from all directions) with respect to the intended listener's position in the playback environment. The degree of diffuseness (or equivalently, said measure of a fraction) may be indicated by a diffuseness parameter p, for example ranging from 0 (no diffuseness, full directionality) to 1 (full diffuseness, no directionality). For example, the ADM audioChannelFormat.diffuse metadata field ranging from ρ = 0 to ρ = 1 may describe the diffuseness of a sound.
  • In the source panner 120, ρ may be used to determine the fraction of signal power sent to the direct path and to the decorrelated paths. When ρ = 1, an object is mixed completely to the diffuse path. When ρ = 0, an object is mixed completely to the direct path.
  • In the source panner 120, objects are processed by the extent panner 820 to produce the direct gains G ij s .
    Figure imgb0077
  • The gains sent to the ramping mixer 130 and diffuse ramping mixer 140 are, G ij M = G ij S . 1 ρ
    Figure imgb0078
    and g i = ρ
    Figure imgb0079
    respectively.
  • During initialization of a new room configuration, an object is panned to the center of the room and fed to the extent panner 820, with Cartesian extent width = depth = height = 1 (i.e., with an extent filling out the entire reproduction environment), to calculate the diffuse speaker gains G j ʹ
    Figure imgb0080
    necessary to produce as uniform a sound field as possible for the given room configuration. These are the gains passed to the speaker decorrelator 150.
  • In other words, the diffuse ramping mixer 140 pans a fraction of the audio object (the fraction being determined by the diffuseness of the audio object) to the center of the reproduction environment (e.g., room). This fraction may be considered as an additional audio object. Further, the ramping mixer assigns an extent (e.g., three-dimensional size) to the additional object such that the three-dimensional volume of the additional object (located at the center of the reproduction environment) fills the entire reproduction environment.
  • A summary of an example of a method for rendering an audio object with diffuseness is illustrated in the flowchart of Fig. 28 . The method may comprise the steps of Fig. 28 either as stand-alone or in combination with the method illustrated in Fig. 24 , Fig. 25 , and Fig. 26 .
  • At step S2810, an additional audio object is created at a center of the playback environment (e.g., room). Further, an extent (e.g., three-dimensional size) is assigned to the additional audio object such that a three-dimensional volume defined by the extent of the additional audio object fills out the entire playback environment. At step S2820, respective overall weight factors are determined for the audio object and the additional audio object based on a measure of a fraction of the audio object that is to be rendered isotropically with respect to the intended listener's position in the playback environment. That is, said two overall weight factors may be determined based on the diffuseness of the audio object, e.g., based on the diffuseness parameter ρ. For example, the overall weight factor for the direct fraction (direct part) of the audio object may be given by 1 ρ ,
    Figure imgb0081
    and the overall weight factor for the diffuse fraction (diffuse part) of the audio object (i.e., for the additional audio object) may be given by ρ .
    Figure imgb0082
    At step S2830, the audio object and the additional audio object, weighted by their respective overall weight factors, are rendered to the one or more speaker feeds in accordance with their respective three-dimensional extents. Rendering of an object in accordance with its extent may be performed as described above in section 3.2.2 "Rendering Object Locations with Extents", and may be performed by the size panner 820 in conjunction with the diffuse ramping mixer 140, for example. The direct fraction of the audio object is rendered at its actual location with its actual extent. The diffuse fraction of the audio object is rendered at the center of the room, with an extent chosen such that it fills the entire room. As indicated above, the resulting gains for the diffuse fraction of the audio object may be determined beforehand, when initializing a new room configuration (reproduction environment). Each speaker feed may be obtained by summing respective contributions from the direct and diffuse fractions of the audio object (i.e., from the audio object and the additional audio object). At step S2840, decorrelation is applied to the contribution from the additional audio object to the one or more speaker feeds. That is, the contributions to the speaker feeds stemming from the additional audio object are decorrelated from each other.
  • An apparatus (rendering apparatus, renderer) for rendering input audio for playback in a playback environment (e.g., for performing the method of Fig. 27 ) may comprise a metadata processing unit (e.g., metadata pre-processor 110) and a rendering unit. The rendering unit may comprise a panning unit and a mixer (e.g., the source panner 120 and either or both of the ramping mixer(s) 130, 140), and optionally, a decorrelation unit (e.g., the speaker decorrelator 150). Steps S2810 and S2820 may be performed by the metadata processing unit. Steps S2830 and S2840 may be performed by the rendering unit. The apparatus may be the further configured to perform the method of Fig. 24 (optionally, with the sub-steps illustrated in Fig. 25 and Fig. 26 ), and optionally, the method of Fig. 27 .
  • 3.3 Metadata Pre-Processing
  • Much of the metadata (e.g., ADM metadata) can be simplified once the playback system is known. The metadata pre-processor 110 is the component that achieves this for the renderer by either reducing the number of speakers available for render or modifying the positional metadata.
  • 3.3.1 Metadata Processing Order
  • An example for the processing order of metadata (metadata features) is schematically illustrated in Fig. 11 . To prevent undesirable interactions between features, metadata parameters are processed in a very specific order. Importance is processed first for efficiency reasons as it may result in fewer sources to process. screenEdgeLock and screenRef are mutually exclusive. zoneExclusion must happen prior to channelLock to prevent locking to speakers that will not be part of the panning layout. Finally divergence is placed after channelLock to allow the mixer to produce a phantom image that remains centered at the location of the locked channel.
  • 3.3.2 Object and Channel Location Transformations
  • The mapping function, MapSC () takes inputs (-180° ≤ Az ≤ 180°, -90° ≤ El ≤ 90°, 0 ≤ R ≤ 1) and the system attribute (Flag 110 = true|false) and may operate as follows:
    1. 1. Warp the elevation angles, so that ±30° maps to ±45°, as follows:
      • if |El| > 30 Elʹ = sgn El × 90 90 El × 45 60
        Figure imgb0083
      • else Elʹ = El × 45 30
        Figure imgb0084
      where we define sgn x = { 1 if x 0 1 if x < 0
      Figure imgb0085
    2. 2. Warp the azimuth angles, according to the Flag 110 attribute:
      1. a. If Flag 110 = true, Azʹ = sgn Az × 3 × Az 2 3 × max 0 , Az 30 8 27 × max 0 , Az 110 56
        Figure imgb0086
      2. b. Else (if Flag 110 = false) Azʹ = sgn Az × 3 × Az 2 3 × max 0 , Az 30 4 + max 0 , Az 90 4
        Figure imgb0087
    3. 3. Map the Az', El' pair to a point on the unit sphere (x',y',z'): = sin Azʹ × cos Elʹ
      Figure imgb0088
      = cos Azʹ × cos Elʹ
      Figure imgb0089
      = sin Elʹ
      Figure imgb0090
    4. 4. Now, distort the sphere into a cylinder: scale cyl = 1 max 2 + 2
      Figure imgb0091
      = × scale cyl
      Figure imgb0092
      = × scale cyl
      Figure imgb0093
      = × scale cyl
      Figure imgb0094
    5. 5. And finally, 'stretch' the cylinder into a cube, and then scale the coordinates according to R: scale cube = 1 max sin Azʹ cos Azʹ
      Figure imgb0095
      X = × R × scale cube
      Figure imgb0096
      Y = × R × scale cube
      Figure imgb0097
      Z = × R
      Figure imgb0098
  • Hence, the outputs of the MapSC () function will be the (X,Y,Z) values, as produced by the procedure above. The inverse function, MapCS (), converts an (X,Y,Z) position to (θ,ϕ,r) and may be achieved through a step-by-step inversion of MapSC ().
  • 3.3.3 Zone Exclusion
  • zoneExclusion is an ADM metadata parameter that allows an object to specify a spatial region of speakers that should not be used to pan the object. An audioChannelFormat of type "Objects" may include a set of "zoneExclusion" sub-elements to describe a set of cuboids. Speakers inside this set of cuboids shall not be used by the renderer to pan the object.
  • The metadata pre-processor 110 may handle zone exclusion by removing speakers from the virtual room layout that is generated for each object. Exclusion zones are applied to speakers before spherical speaker coordinates are transformed to Cartesian coordinates by the warping function described in section 3.3.2 "Object and Channel Location Transformations".
  • The algorithm that processes zone exclusion metadata to remove speakers from the object's virtual speaker layout is described below.
    Step 1: For each of the N speakers in the virtual speaker layout, check if the speaker lies inside any of the M exclusion zone rectangular cuboids. If so, remove it from the layout by setting its mask value to zero.
  •        for j = 1 to N
           {
               /*get cartesian position (without warping)*/
                x = distance(j) ∗ cos(elevation(j)) ∗ cos(azimuth(j));
               y = distance(j) ∗ cos(elevation(j)) ∗ sin(azimuth(j));
               z = distance(j) ∗ sin(elevation(j));
               mask(j) = 1;
               for k = 1 to M
               {
                      if(zone(k).minX ≤ x ≤ zone(k).maxX
                      & zone(k).minY ≤ y ≤ zone(k).maxY
                      & zone(k).minZ ≤ z ≤ zone(k).maxZ)
                      {
                         mask(j) = 0;
                      }
                 }
           }
    Step 2: Remove additional speakers to ensure that the resulting layout is valid for the triple-balance panner, as described in section 3.2.1 "Rendering Point Objects". The following speaker layout rule is enforced on the speaker rows: every speaker row, except for the front and back rows, must have a speaker at x = 1 and another speaker at x = -1. This rule is applied after the speaker coordinates have been transformed using the warping function described in section 3.3.2 "Object and Channel Location Transformations".
           for j = 1 to N
           {
           /*if a side wall speaker is disabled
               if (mask(j) = 0 && abs(p_sx(j)) == 1 && abs(p_sy(j)) != 1);
               for k = 1 to N
               {
           /* remove all row speakers */
                    if(p_sy(j) == p_sy(k))
                    {
                        mask(k) = 0;
                    }
           }
       }
  • The mask values will then be used by the point panner 810 to select which speakers are considered part of the output layout for the object, as described in section 3.2.1 "Rendering Point Objects".
  • The enforcement of the rule in Step 2 ensures that the resulting speaker layout does not lead to undesired panning behavior. For example, consider the System F layout from ITU-R BS.2051, where only the M-90 speaker has been removed. If we then pan an object from the front right to the back right of the room, the panner will pan the object entirely to the left (speaker M+90) as the object crosses the middle of the room. To correct this, we also remove the M+90 speaker, and now the object renders correctly from front to back on the right side, by panning between the M-30 and M-135 speakers.
  • 3.3.4 Gain
  • Support for the gain metadata in the audioBlockFormat is implemented by the source panner 120 and scales the gains of each object provided to the ramping mixers 130, 140. Gain metadata thus receives the same cross-fade defined by the objects jumpPosition metadata.
  • 3.3.5 Channel Lock
  • Support for channelLock metadata is implemented inside the metadata pre-processor 110 component described in section 3.1 "Architecture". If the channelLock flag is set to 1 in an audioBlockFormat element contained by an audioChannelFormat instance of type Objects, the virtual source renderer component will modify the position sub-elements of the audioBlockFormat to ensure that the object's audio is panned entirely to a single output channel.
  • The optional maxDistance attribute controls whether the channelLock effect is applied to the object, based on the unweighted Euclidean distance between an object's position and the output speaker closest to it. If maxDistance is undefined, the renderer assumes a default value of infinity, meaning that the object always "snaps" to the closest speaker.
  • For objects with position metadata specified in spherical coordinates, channelLock processing is performed after the object's position has been transformed into Cartesian coordinates, as described in section 3.3.2 "Object and Channel Location Transformations". Similarly, the distances between the object and the speakers are calculated using the speaker positions after they have been transformed from spherical to Cartesian coordinates, as described in section 3.3.2 "Object and Channel Location Transformations".
  • For determining which speaker to "lock" the object to, a weighted Euclidean distance measure has been designed to yield rectangular cuboid "lock" regions around each speaker in Cartesian space. Dividing the snap regions in this way improves the intuitiveness of the snap feature during content creation in a mixing studio, and is consistent with the allocentric rendering philosophy behind the point panner 810.
  • For example, Channel Lock may be applied as follows:
  •       min_dist_u = Inf;
          min_dist = Inf;
          wx = 1/16; wy = 4; wz = 32;
           /* find the closest speaker */
           for j = 1 to N /* for each speaker */
           {
               /* weighted Euclidean distance using Cartesian object
                * and speaker positions*/
               dist = wx∗(p_ox-p_sx(j))^2
                     + wy∗(p_ox-p_sy(j))^2
                     + wz∗(p_ox-p_sz(j))^2;
               dist_u = (p_ox-p_sx(j))^2
                       + (p_ox-p_sy(j))^2
                       + (p_ox_p_sz(j))^2;
               if (dist < min_dist)
               {
                    min_dist = dist;
                    min_dist_u = dist_u;
                    idx_min = j;
                }
           }
           /* apply maxDistance attribute using unweighted distance */
           if (min_dist_u <= maxDistance)
           {
               p_ox = p_sx(idx_min);
               p_oy = p_sy(idx_min);
               p_oz = p_sy(idx_min);
           }
  • It should be noted that in the above pseudocode, the speakers 1 to N are pre-sorted as follows: center is always placed at the head of the list if it is present. The remaining speakers are then ordered first by decreasing z-value, then by increasing y-value and finally by increasing x-value, such that when there are multiple speakers with exactly the same weighted distance to the object, the object is locked to the speaker that is closest to the top-front-left of the room.
  • 3.3.6 Divergence
  • This section relates to a method for controlling constraints when rendering audio objects with divergence.
  • Within traditional mixing, the idea of creating phantom sources by panning a coherent source to adjacent speakers has been used for some time-most commonly in the context of creating a phantom center source in a stereo system where only a left and right speaker exist. To do this, a power preserving pan is used to distribute a source to the left and right channels, based on the expectation that this power preserving pan will cause an acoustic summing in the room to create a source of the correct level at the correct location.
  • This assumption is reasonable when the left and right speakers are spaced relatively sparsely, as is the case in cinemas, but if speakers are too close together, the apparent level of the phantom source may increase noticeably.
  • When considering contemporary immersive audio, the idea of creating a phantom source using adjacent audio objects persists with content creators. In the new idiom of object based audio, the efficient way of expressing this intent in the content is to use metadata to note that a source is intended to be rendered as a phantom source. This metadata feature is labeled 'Divergence' in the ITU-R BS.2076 ADM standard.
  • Section 9.6 of the ADM standard specifies a way to express the concept of divergence in metadata and provides what could be considered an obvious approach to phantom source panning in an effort to provide the same functionality as legacy mixing through objects. One detail provided within the ADM specification is that in order to create a phantom image, a power preserving pan should be created between two virtual objects (additional audio objects) and an original audio object-as would be expected when using left and right speakers to create a phantom center channel. Needless to say, the phantom image to be created is located at the position of the original audio object.
  • Fig. 12 illustrates an example of two virtual objects (additional audio objects) 1220, 1230 that are provided for an (original) audio object 1210 for purposes of phantom source panning. In this example, each virtual object 1220, 1230 is spaced from the audio object 1210 by an angular distance 1240. Evidently, the two virtual objects 1220, 1230 are spaced from each other by twice the angular distance 1240. This angular distance 1240 may be referred to as an angle of divergence.
  • As has been realized, there are two direct problems in this naïve adaption of the legacy approach to object based audio content. The first problem comes from the ability to specify the angle of divergence, and the second problem from how objects are rendered to speakers in an object audio renderer.
  • The freedom (e.g., in ADM) for object based divergence to specify an angle that dictates where the new pair of virtual objects are created relative to the desired phantom image location means that the new virtual objects can be located very close to the phantom location. The location of these virtual objects close to the phantom location is analogous to placing speakers close together when rendering a phantom center-if this is realized in practice, a power preserving pan would result in inappropriate level of the phantom image (e.g., increased loudness), due to the coherent summation of the new sources.
  • To playback object audio content, it must first be rendered to speaker feeds that map to the reproduction system's speaker locations, and this is when the second issue present in the naïve formulation of divergence is exposed. For sparse speaker arrangements (as are common, e.g., in home theatre playback scenarios) multiple audio objects in the content space are mapped (rendered) to the same speaker-in fact each individual object will typically play back through multiple speakers with a variety of gains designed to create phantom images in the playback environment. In the context of the divergence feature this means that the virtual objects created to simulate the phantom source will themselves be subject to the rendering process, and may be mapped to the same speakers in such a way that the power preserving gains intended to create a phantom image when summed acoustically will instead be summed in the renderer, coherently-which again will cause level differences.
  • Ultimately the naïve formulation of divergence (e.g., in ADM) that relies on simple power preserving panning will suffer notable level issues given (i) the added flexibility of virtual source locations, and (ii) the potential for the rendering process to cause the virtual sources to be summed electrically (coherently) instead of acoustically. Embodiments of the present disclosure address both these issues.
  • Section 9.6 of the ADM standard (ITR-R BS.2076) provides a definition of the divergence metadata's behavior in terms of two parameters: objectDivergence (0, 1) and azimuthRange. While this is not the only way such a behavior could be described, it will be used to help explain the context and formulation of this invention. In general, the metadata may be said to indicate (e.g., specify), apart from a location of the audio object, a distance measure (e.g., the azimuthRange) indicative of a distance between the virtual sources. The distance measure may be expressed by a distance parameter D. The distance measure may indicate an angular distance or a Euclidean distance. In the examples below, the distance measure indicates an angular distance. Further, the distance measure may directly indicate a distance between the virtual sources themselves, or a distance between each of the virtual sources and the original audio object. As will be appreciated by the person of skill in the art, such distance measures can be easily converted into each other. Further, the metadata may indicate (e.g., specify) a measure of relative importance of the virtual sources and the original audio object (e.g., the objectDivergence). This measure of relative importance may be referred to as divergence and may be expressed by a divergence parameter (divergence value) d. The divergence parameter d may range from 0 to 1, with 0 indicating zero divergence (i.e., no power is provided to the virtual sources-zero relative importance of the virtual sources), and 1 indicating full divergence (i.e., no power is provided to the original audio object-full relative importance of the virtual sources).
  • For each object Oi with divergence (e.g., objectDivergence) d, the renderer (e.g., virtual object renderer) creates two additional audio objects O i+, O i- at the locations controlled by the distance measure D (e.g., by the azimuthRange element) and calculates three gains gdi, g di+, g di- to ensure the power across the three new objects is equivalent to the original object.
  • If the location of Oi is specified in spherical coordinates (θi ,ϕi,ri ), locations for the virtual objects (additional audio objects) may be defined as: θ i ± = θ i ± 0.5 × azimuthRange
    Figure imgb0099
    φ i ± = φ i
    Figure imgb0100
    r i ± = r i
    Figure imgb0101
    That is, the additional audio objects may be located in the same horizontal plane (i.e., at the same elevation, or at the same z coordinate) as the original audio object, at equal (angular) distances from the original audio object, on opposite sides of the original audio object when seen from the intended listener's position, and at the same (radial) distance from the intended listener's position as the original audio object. In general, the locations for the virtual objects (additional audio objects) are determined by the location of the original audio object and the distance measure D.
  • If one or both of the resulting virtual objects fall outside the rendering region, the distance measure (e.g., azimuthRange) value may be reduced to ensure both virtual objects are within the rendering region (e.g., within the reproduction environment). The need to recalculate the position of both virtual objects is to ensure the phantom image created remains at the correct location.
  • For objects with locations specified in Cartesian coordinates (xi,yi,zi ), locations for the virtual objects may be determined first by transforming the Cartesian location to spherical coordinates using the mapping function MapSC (), described in section 3.3.2 "Object and Channel Location Transformations". Then the spherical locations of Oi + and O i- are determined, e.g., in accordance with the above formula, and finally the locations may be transformed to Cartesian coordinates with the inverse transformation function MapSC ( ).
  • The content played at the virtual locations may have a simple gain relationship with the original object audio. If x[n] is the original object audio (the audio signal of the original object), the divergence metadata allows for three new audio objects: y[n] (the signal from the original location), and y v1[n] and y v2[n] (the signals from the two virtual object locations). Then, y n = g d x n
    Figure imgb0102
    y V 1 n = y V 2 n = g v x n
    Figure imgb0103
    where gd and gv are weight factors (e.g., mixing gains) to be applied to the (original) audio object and the virtual (additional) audio objects.
  • The power preserving dictate of ADM implies that g d 2 + 2 g v 2 = 1
    Figure imgb0104
  • The ADM specification also provides a specification for how these gains vary as the objectDivergence changes.
  • Example: With an LCR loudspeaker configuration and the object positioned directly at the C position, and the LR virtual objects specified by using an azimuthRange of 30 degrees. An objectDivergence value of 0 indicating no divergence, only the center speaker would be firing. A value of 0.5 would have all three (LCR) loudspeakers firing equally, and a value of 1 would have the L and R loudspeakers firing equally.
  • In more detail, according to the ADM specification, the gains to be applied to the original object and the two new virtual objects provide a power preserving spread across the three sources with the divergence (e.g., objectDivergence value) d controlling the distribution of the power between the sources. As indicated above, the divergence (e.g., objectDivergence value) d varies between 0 and 1, where a value of 1 represents all the power coming from the virtual objects, and the original object made silent. The following equations specify the weight factors (e.g., mixing gains) for the objects as functions of d in the ADM specification: g di = { 1 4 d + 1 0 < d 0.5 1 d 2 d 0.5 < d 1 g di ± = { 2 d 4 d + 1 0 < d 0.5 1 4 2 d 0.5 < d 1
    Figure imgb0105
  • While panning according to the above equations works for the simple case of phantom center channels in legacy systems, it has been realized to fail for more general applications. Namely, it has been realized that for phantom source panning for audio objects, the following general rules should be applied:
    1. 1. If signals will be summed coherently, use amplitude preserving panning functions
    2. 2. If signals will sum incoherently, use power preserving panning functions.
  • In view thereof, the present disclosure describes divergence processing that accounts for the following guiding principles:
    1. 1. The perceived effect created by playing back coherent signals from spatially separated speakers varies as a function of distance between the speakers, and varies across frequencies.
    2. 2. All frequencies tend towards adding incoherently when the distance between speakers is large.
    3. 3. Low frequency components tend to add coherently over greater distances than high frequency components.
    4. 4. As the distance between speakers decreases the transition between which frequencies add coherently versus incoherently begins at higher frequencies.
  • These guiding principles are accounted for by the frequency and angle dependent aspects of the present disclosure.
  • The second issue which compounds the loudness issues described above is the effect that the rendering algorithm has on the combination of the virtual objects when rendering them to speaker feeds. Fig. 13 schematically illustrates a speaker layout comprising plural speakers 1342, 1344, 1346, 1348, among them a Left-surround speaker (Ls) 1342 and a front-left speaker (L) 1344. The figure further illustrates an audio object 1310 and two virtual objects 1320, 1330 for phantom source rendering. The virtual objects 1320, 1330 are created based on divergence metadata. The rendering algorithm is to determine how to mix these objects in order to create the speaker feeds. Intuitively, any rendering algorithm will mix these two objects into the speakers 1342, 1344 labelled L and Ls, essentially calculating gains in accordance with: L n = g V 1 L x V 1 n + g V 2 L x V 2 n
    Figure imgb0106
    Ls n = g V 1 Ls x V 1 n + g V 2 Ls x V 2 n
    Figure imgb0107
  • As both virtual objects 1320, 1330 in the example of Fig. 13 are closer to the L speaker 1342 than to the Ls speaker 1344 it is expected that the gains for creating the speaker feed L[n] for the L speaker 1342 would direct the majority of each of their power to the L speaker 1342. Since the mixing is done in the renderer, the virtual objects 1320, 1330 will be summed coherently-hence the power preserving gains generated as part of creating the virtual objects will be summed inappropriately.
  • This phenomenon is again dependent on the distance measure (e.g., azimuthRange) of the divergence, and it is possible to have the situation where the virtual objects are both panned to the same set of speakers, or to entirely distinct sets of speakers, depending on how their locations sit within the renderer's speaker layout. Fig. 14A, Fig. 14B, and Fig. 14C illustrate examples of relative arrangements of object locations 1410x, virtual object locations 1420x, 1430x and speaker locations 1441x, 1442x, 1443x, 1445x (x = A, B, C) for a given speaker layout. As can be seen from these examples, which speakers the virtual objects get mixed to depends on the distance measure (e.g., azimuthRange) and the speaker layout.
  • In view of the issues described above, the present disclosure describes methods for controlling the constraints applied to render objects with divergence in order to tune their signal power or perceived loudness. In particular, the present disclosure describes two methods for rendering audio objects with divergence metadata that address the aforementioned issues and that could be applied independently or in combination with each other.
  • Fig. 15 illustrates, as a general overview, a block diagram of an example of a renderer (rendering apparatus) 1500 according to embodiments of the disclosure that is capable of rendering audio objects with divergence metadata. Some or all of the functional blocks illustrated in Fig. 15 may correspond to functional blocks illustrated in Fig. 6 , Fig. 7 , or Fig. 8 . The renderer 1500 comprises a divergence metadata processing block (metadata processing unit) 1510, a point panner 1520, and a mixer block (mixer unit) 1530. The divergence metadata processing block 1510 may correspond to, or be included in, the metadata pre-processor 110 in Fig. 7 . The point panner 1520 may correspond to the point panner 810 in Fig. 8 . The mixer block 1530 may correspond to the ramping mixer 130 in Fig. 7 . The renderer 1500 receives an object (x[n]) 1512 and associated (divergence) metadata 1514 as input. The metadata 1514 may include an indication of divergence d and the distance measure D. Further, the renderer 1500 may receive the speaker layout 1524 as an input. If the object 1512 has divergence metadata 1514 (e.g., divergence d and distance measure D) associated with it, first the divergence metadata preprocessing block 1510 will interpret that metadata 1514 to create three audio objects 1522, namely virtual object sources (yV1[n] and yV2[n]) and the modified original object (y[n]). The point panner 1520 then will calculate the gain matrix G ij M
    Figure imgb0108
    1534 which contains the gain applied to object i to create the signal for speaker j. The point panner 1520 may further modify the signals associated with the three audio objects to thereby create three modified audio objects 1532, namely y'[n], y'V1[n], and y'V2[n]. The final stage of rendering is to apply the gain matrix created in the point panner 1520 to object signals in order create the speaker feeds 1542-this is the function of the mixer block 1530.
  • Both the aforementioned methods for rendering audio objects with divergence metadata can be performed by the renderer 1500, for example. The first method describes a control function which can be added during the creation of the virtual objects, which compensates for the variation in how these virtual sources would be summed acoustically if rendered to speakers at their virtual locations. This could be integrated within the divergence metadata processing block 1510 of the renderer 1500. The second method describes how the rendering gains can be normalized (for example in the point panner 1520) to ensure that a desired signal level is produced from the speakers in a specific layout. Both methods will now be described in detail.
  • 3.3.6.1 Controlled Method for Creation of Virtual Sources (First Method)
  • The naïve method for creating a set of power preserving divergence gains follows g d 2 +
    Figure imgb0109
    2 g v 2 = 1 ,
    Figure imgb0110
    regardless of the distance (e.g., angle) separating the virtual sources. The first element of the present method is to incorporate a distance (e.g., an angle of separation) into the calculation of the gains to allow for the effective panning to vary between an amplitude preserving pan and a power preserving pan. For example, an angle of separation (θ) may be defined as the angle between the two virtual sources (more generally, as the distance, or distance measure). Typically, the virtual sources will be located symmetrically about the original source, and in such cases, the angle of separation may easily be derived from the angle between the original source and either of the virtual sources (for example, the angle of separation of the virtual sources may be equal to twice the angle between the original source and either of the virtual sources). By introducing a control function p(θ), the naïve prescription for creating the set of power preserving divergence gains can be revised to: g d p θ + 2 g v p θ = 1
    Figure imgb0111
    In general, the control function p is a function of the distance measure D, p(D). Without intended limitation, reference will be made to the control function p being a function of the angle of separation θ, p(θ).
  • The range of p(θ) may vary from 1, where the above equation represents the constraints of an amplitude preserving pan, to 2 where the above equation is equivalent to enforcing constraints of a power preserving pan.
  • Fig. 29 is a flowchart illustrating an overview of the first method of rendering audio objects with divergence as an example of method of rendering input audio for playback in a playback environment. Input audio received by the method includes at least one audio object and associated metadata. The associated metadata indicates at least a location of the audio object. The metadata further indicates that the audio object is to be rendered with divergence, and may also indicate a degree of divergence (divergence parameter, divergence value) d and a distance measure D. The degree of divergence may be said to be a measure of relative importance of virtual objects (additional audio objects) compared to the audio object.
  • The method comprises steps S2910 to S2930 described below. Optionally, the method may comprise, as an initial step, referring to the metadata for the audio object and determining whether a phantom object at the location of the audio object is to be created. If so, steps S2910 to S2930 may be executed. Otherwise, the method may end.
  • At step S2910, two additional audio objects associated with the audio object are created such that respective locations of the two additional audio objects are evenly spaced from the location of the audio object, on opposite sides of the location of the audio object when seen from an intended listener's position in the playback environment. The additional audio objects may be referred to as virtual audio objects.
  • At step S2920, respective weight factors for application to the audio object and the two additional audio objects are determined. The weight factors may be the mixing gains gd and gv described above. The weight factors gains may impose a desired relative importance across the three objects. The two additional audio objects may have equal weight factors. In general, the weight factors (e.g., mixing gains gd and gv ; without intended limitation, reference may be made to the mixing gains gd and gv in the following) may depend on the measure of relative importance (e.g., divergence parameter d; without intended limitation, reference may be made to the divergence parameter d in the following) indicated by the metadata. For small values of the divergence parameter, the majority of energy may be provided by the original object, while for high values of the divergence parameter, the majority of energy may be provided by the virtual objects. In one example, the values of the divergence parameter may vary between 0 and 1. A divergence value of 0 indicates that all energy will be provided by the original object, so that gd will be equal to 1. Conversely, a divergence value of 1 indicates that all energy will be provided by the virtual objects. In this case, gd will be 0. Further, the weight factors may depend on the distance measure D. Examples of this dependence will be provided below.
  • At step S2930, the audio object and the two additional audio objects are rendered to one or more speaker feeds in accordance with the determined weight factors. For example, application of the weight factors to the audio object and the additional audio objects may yield the three new audio objects y[n], y V1[n], and y V2[n] described above, which may be rendered to the speaker feeds, for example by the point panner 1520 and the mixer block 1530 of the renderer 1500. The rendering of the audio object and the two additional audio objects to the one or more speaker feeds may result in a gain coefficient for each of the one or more speaker feeds (e.g., for an audio object signal x[n] of the original audio object).
  • An apparatus (rendering apparatus, renderer) for rendering input audio for playback in a playback environment (e.g., for performing the method of Fig. 29 ) may comprise a metadata processing unit (e.g., metadata pre-processor 110) and a rendering unit. The rendering unit may comprise a panning unit and a mixer (e.g., the source panner 120 and either or both of the ramping mixer(s) 130, 140). Step S2910 and step S2920 may be performed by the aforementioned metadata processing unit (e.g., metadata pre-processor 110). Step S2930 may be performed by the rendering unit.
  • The method may further comprise normalizing the weight factors based on the distance measure D. That is, initial weight factors may be determined, for example in accordance with the divergence parameter d, and the initial weight factors may subsequently be normalized based on the distance measure D. An example of such a method is illustrated in the flowchart of Fig. 30 .
  • Step S3010, step S3020, and step S3040 in Fig. 30 may correspond to steps S2910, S2920, and S2930, respectively, in Fig. 29 , wherein the weight factors determined at step S3020 may be referred to as initial weight factors. At step S3030, the (initial) weight factors determined at step S3020 are normalized based on the distance measure. In general, the weight factors may be normalized such that a function f(g 1, g 2, D) of the weight factors g 1, g 2 and the distance measure D attains a predetermined value, such as 1, for example. In this case, f(g1, g 2, D) = 1 would need to hold. Step S3030 may be performed by the metadata processing unit.
  • For example, the weight factors may be normalized such that a sum of equal powers of the normalized weight factors is equal to a predetermined value (e.g., 1). Here, an exponent of the normalized weight factors in said sum may be determined based on the distance measure. As indicated above, this normalization may be performed in accordance with the control function p(θ). The control function p(θ) may be used as said exponent. The weight factors may be the mixing gains, as indicated above, so that g 1 = gd and g 2 = gv. In other words, the mixing gains may be normalized to satisfy equation [6]. Here and in the remainder of this disclosure, normalizing a set of quantities is understood to relate to uniformly scaling an initial set of quantities (i.e., using the same scaling factor for each quantity of the set) so that the set of scaled quantities satisfies a normalization condition, such as equation [6].
  • The control function p(θ) may be a smooth monotonic function of the distance measure (e.g., angle of separation θ; without intended limitation, reference may be made to the angle of separation θ in the following). The function p(θ) may yield 1 for the distance measure below a first threshold value and may yield 2 for the distance measure above a second threshold value. Thus, the image range of p(θ) extends from 1, where equation [6] represents the constraints of an amplitude preserving pan, to 2 where equation [6] is equivalent to enforcing constraints of a power preserving pan, as in equation [3]. For values of the distance measure between the first and second threshold values, p(θ) varies between 1 and 2 (i.e., takes on intermediate values) as the distance measure (e.g., the angle of separation θ) increases. p(θ) may have zero slope at the first and second threshold values. Further, p(θ) may have an inflection point at an intermediate value between the first and second threshold values. Fig. 16A illustrates an example of the general characteristic expected of p(θ). Notably, the control function p(θ) follows the guiding principles that the panning function should tend to favor amplitude preservation if the virtual sources are close to the phantom image location, and should provide for power preservation once the sources become sufficiently separated.
  • In addition to the distance measure (e.g., angle of separation), the values of the weight factors (e.g., gd and gv ) may also depend on the divergence parameter. For small values of the divergence parameter, the majority of energy will be provided by the original object, while for high values of the divergence parameter, the majority of energy will be provided by the virtual objects. In one example, the values of the divergence parameter may vary between 0 and 1. A divergence value of 0 indicates that all energy will be provided by the original object. In this case, gv will be equal to 0 and gd will be equal to 1, regardless of the value of p(θ). Conversely, a divergence value of 1 indicates that all energy will be provided by the virtual objects. In this case, gd will be 0, the value 2 g v p θ
    Figure imgb0112
    will be equal to 1, and the value of gv will vary between 1 2
    Figure imgb0113
    and 2 2
    Figure imgb0114
    as p(θ) varies between 1 and 2.
  • The introduction of the control function p(θ) as a pure function of the distance measure (e.g., angle of separation) still constrains the weight factors (e.g., mixing gains) generated to be wideband-i.e. they apply the same gain to all frequencies. This may not fully agree with the guiding principle that the perception of phantom images varies across frequencies. To address this frequency dependency, the control function can be extended to include frequency as a control parameter. That is, the control function p can be extended to be a function of the distance measure (e.g., the angle of separation) and frequency, p(θ,f). Modifying equation [6], this yields: g d p θ f + 2 g v p θ f = 1
    Figure imgb0115
  • The extended control function, p(θ,f) still conforms to the same range as p(θ), however the inclusion of frequency, f, allows for the recognition that low frequency signals will continue to sum coherently over a larger angle of separation than higher frequency signals. Fig. 16B illustrates an example of the general characteristic expected of p(θ,f), i.e., how the control function p(θ,f) varies across frequencies. As can be seen from Fig. 16B , for low frequencies the amplitude panning constraint is preserved for larger distances (e.g., larger angles of separation) than for high frequencies. That is, for lower frequencies, the aforementioned first and second thresholds may be higher than for higher frequencies. That is, the first threshold may be a monotonically decreasing function of frequency, and the second threshold may be a monotonically decreasing function of frequency. In general, regardless of frequency, it may be assumed that for values of θ greater than or equal to 120 degrees, two sources are sufficiently far apart that they should be reproduced using power preserving panning (i.e., p(θ,f) = 2).
  • In accordance with the above, normalization of the weight factors (e.g., mixing gains) may be performed on a sub-band basis, depending on frequency. That is, normalization of the weight factors may be performed for each of a plurality of sub-bands. Then, said exponent of the normalized weight factors in said sum mentioned above may be determined on the basis of a frequency of the frequency sub-band, so that the exponent is a function of the distance measure (e.g., angle of separation) and the frequency. The frequency that is used for determining said exponent may be the center frequency of a respective sub-band or may be any other frequency suitably chosen within the respective sub-band. The exponent may be the control function p(θ,f).
  • 3.3.6.2 Method for Constraining Speaker Rendering of Virtual Sources(Second Method)
  • By employing a control function in the method for creating virtual sources, the method described in the foregoing section addresses the issues that would arise through blindly applying a power preserving set of gains (weight factors) prior to rendering. However it does not address the issues which may arise within an object renderer where divergence is allowed to be applied to an object located anywhere in the immersive space. These issues arise primarily because rendering of the final speaker feeds occurs in the playback environment, rather than in the controlled environment of the content creator, and are intrinsic to the object renderer paradigm of immersive audio. Thus, under certain conditions, using the second method that will now be described in more detail may be of advantage. As noted above, the second method may be employed either as a stand alone or in combination with the first method that has been described in the foregoing section.
  • Fig. 31 is a flowchart illustrating an overview of the second method of rendering audio objects with divergence as an example of method of rendering input audio for playback in a playback environment. Input audio received by the method includes at least one audio object and associated metadata. The associated metadata indicates at least a location of the audio object. The metadata further indicates that the audio object is to be rendered with divergence, and may also indicate a degree of divergence (divergence parameter, divergence value) d and a distance measure D. The degree of divergence may be said to be a measure of relative importance of virtual objects (additional audio objects) compared to the audio object.
  • The method comprises steps S3110 to S3150 described below. Optionally, the method may comprise, as an initial step, referring to the metadata for the audio object and determining whether a phantom object at the location of the audio object is to be created. If so, steps S3110 to S3150 may be executed. Otherwise, the method may end. Step S3110 and step S3120 in Fig. 31 may correspond to step S2910 and step S2920, respectively, in Fig. 29 .
  • At step S3130, a set of rendering gains for mapping (e.g., panning) the audio object and the two additional audio objects to the one or more speaker feeds is determined. This step may be performed by the point panner 1520, for example. Setting aside the details of the internal algorithms used by the point panner 1520, its purpose is to determine how to steer an audio object, given the audio object's location, to the set of speakers it is currently rendering for. So for a set of {i} object locations, and knowing the locations of the set of {j} speakers, step S3130 (for example performed by the point panner 1520) determines a rendering matrix G ij M
    Figure imgb0116
    (i.e., a set of rendering gains) which dictates the gains (rendering gains) applied to each object's content when mixing it into each speaker signal.
  • At step S3140, the rendering gains are normalized based on the distance measure (e.g., angle of separation). Step S3140 may be performed by the point panner 1520, for example. In general, the rendering gains may be normalized so that, when inspecting the gains for a single object (i = I) over all speakers, the normalization condition is given by: i j = 1 J G ij M p = 1
    Figure imgb0117
  • If equation [8] is enforced for p = 1, the panning would be categorized as an amplitude preserving panning. If equation [8] is enforced for p = 2, the panning would be power preserving panning. Generally, there is no inherent need for an object panner to meet either of these criteria, and it is possible to build a panner where equation [8] is satisfied for no value of p.
  • This method of inspection is useful when evaluating the panner's behavior when rendering objects (and virtual objects) created through divergence. If equation [8] is evaluated over a limited set of objects Ψ, which includes only the audio object and the additional audio objects (virtual objects) created from a single original object through the application of divergence metadata, a rendering constraint of the following form can be constructed: i Ψ j = 1 J i = 1 3 G ij M p = 1
    Figure imgb0118
  • Equation [9], if true, would imply panning of all objects and virtual objects associated with an object with divergence so that the objects are actually reproduced in the speaker feeds in accordance with either an amplitude preserving pan (p = 1) or a power preserving pan (p = 2). Further, if it was found that this constraint did not hold naturally, it could be enforced by rescaling the gains (rendering gains) associated with the set Ψ of divergence objects.
  • Additionally, when the normalization condition is formulated in this manner, the control functions p(θ) and p(θ, f) can be introduced, for example to replace p in equation [9]. Yet further, if we extend the concept of a wideband point panner to a panner which may also create frequency dependent panning functions G ij M f ,
    Figure imgb0119
    then the speaker panning constraint (normalization condition) can be expressed as: i Ψ j = 1 J i = 1 3 G ij M f p θ f = 1
    Figure imgb0120
  • In general, the rendering gains may be normalized (e.g., re-scaled) such that a sum of equal powers of the normalized rendering gains for all of the one or more speaker feeds and for all of the audio objects and the two additional audio objects is equal to a predetermined value (such as 1, for example). An exponent of the normalized rendering gains in said sum may be determined based on said distance measure. Said exponent may be the control function p(θ) described above. In analogy to the normalization of weight factors described in the foregoing section, the normalization of the rendering gains may be performed on a sub-band basis and in dependence on frequency.
  • At step S3150, the audio object and the two additional audio objects are rendered to the one or more speaker feeds in accordance with the determined weight factors and the (normalized) rendering gains.
  • In this way, a method of enforcing separation angle and frequency dependent panning constraints on the speaker outputs created when applying the divergence metadata is obtained.
  • It should be noted that the method of Fig. 31 may additionally include a step of normalizing the weight factors, in analogy to step S3030 in Fig. 30 .
  • Finally, it should be noted that both equations [7] and [10] recite a function p(θ,f). While these functions may typically be the same, in some cases they may be defined independently of one another, such that p(θ,f) in equation [7] may not necessarily be equivalent to p(θ,f) in equation [10].
  • An apparatus (rendering apparatus, renderer) for rendering input audio for playback in a playback environment (e.g., for performing the method of Fig. 31 ) may comprise a metadata processing unit (e.g., metadata pre-processor 110) and a rendering unit. The rendering unit may comprise a panning unit and a mixer (e.g., the source panner 120 and either or both of the ramping mixer(s) 130, 140). Step S3110 and step S3120 may be performed by the aforementioned metadata processing unit (e.g., metadata pre-processor 110). Step S3130, step S3140, and step S3150 may be performed by the rendering unit.
  • 3.3.7 Screen Scaling
  • The screenScaling feature allows objects in the front half of the room (e.g., the playback environment) to be panned relative to the screen. The screenRef flag in the object's metadata is used to indicate whether the object is screen related. If the flag is set to 1, the renderer will use metadata about the reference screen that was used during authoring (e.g., contained in the audioProgramme element) and the playback screen (e.g., given to the renderer as configuration parameters) to warp the azimuth and elevation of the objects in order to account for differences in the location and size of the screens. ITU-R BS.2076-0 provides default screen specification for the reference screen for use when such information is not contained in the input file. The renderer shall use default values for the playback screen, e.g., these same default values, when no configuration data is provided.
  • To maintain sensible behavior in the screen scaling feature, the following conditions should be satisfied by the attributes of the audioProgrammeReferenceScreen sub-element of the audioProgramme element. The same conditions apply to the corresponding renderer configuration parameters that specify the properties of the playback screen.
    • It is assumed that the normal vector facing outward from the center of the screen intersects the center of the room (i.e., the screen is facing the center of the room).
    • The distance from the center of the room to the screen must be greater than 0.01.
    • The azimuth angle of the center of the screen must be between -40 to +40 degrees.
    • The elevation angle of the center of the screen must be between -40 to +40 degrees.
    • When the center of the screen is projected to the front wall, the entire screen surface must lie entirely on the front wall.
    • The azimuth and elevation at every corner of the screen must be between -45 and 45 degrees.
  • These limitations may be enforced in the metadata and in the renderer configuration by the following procedure:
    Step 1. If the screen position and size values are given in Cartesian coordinates, convert to spherical coordinates using the warping function described in section 3.3.2 "Object and Channel Location Transformations".
    Step 2. Apply limits to the screen position and size metadata, as follows:
  •      /*limit screen position*/
         screenCentrePosition.distance = ...
             max(screenCentrePosition.distance, 0.01);
         screenCentrePosition.azimuth = ...
             min(max(screenCentrePosition.azimuth, -40), 40);
         screenCentrePosition.elevation = ...
             min(max(screenCentrePosition.elevation, -40), 40);
         /* screen width and height at distance = 1*/
         width = 2 ∗ tan(screenWidth.azimuth/2);
         height = width / aspectRatio;
         height_elevation = 2 ∗ arctan(height/2);
         /* limit screen size azimuth */
         max az = 90 - abs(screenCentrePosition.azimuth);
         if (screenWidth.azimuth > max_az)
         {
           screenWidth.azimuth = max_az;
           width = 2 ∗ tan(screenWidth.azimuth/2);
           aspectRatio = width/height;
         }
         /* limit aspect ratio */
         max_el = 90 - abs(screenCentrePosition.elevation);
         if (height_elevation > max-el)
         { height = 2 ∗ tan(max_el/2);
              aspectRatio = width/height;
         }
  • Once appropriate limits have been applied to the screens, screen scaling is applied to objects with screenRef = 1 as follows:
    • Step 1. If the object's position is given in Cartesian coordinates, it is converted to spherical coordinates using the MapSC ( ) function (section 3.3.2 "Object and Channel Location Transformations").
    • Step 2. Apply a warping function to the object's direction az and el that maps the azimuth and elevation range of the reference screen to the range of the playback screen.
      Figure imgb0121
  • It is worth noting that the warp function begins to warp angles at +/- 50 degrees. This is because the screen edges are allowed to be at +/- 45 degrees, and there needs to be a bit of "slack" space to prevent the warping function from producing line segments with zero slope, which would result in panning "dead zones".
  • The angle-warping strategy naturally causes the displacement of objects due to screen scaling to be greater near the front of the room than in the center of the room. The screen distance is purposely not considered in this strategy, as this allows a small screen near the center of the room to be treated the same as a larger screen near the front wall-i.e., the algorithm always considers the projection of the screen to the front wall of the room. This is schematically illustrated in Fig. 17 in which the screen is projected to the front wall of the room in accordance with its width azimuth angle 1710 (screenWidth.azimuth).
  • Fig. 18A and Fig. 18B schematically show the resulting warping functions for azimuth and elevation for the following screen configurations:
  •      ref.screenCentrePosition.azimuth = -5;
         ref.screenWidth.azimuth = 20;
         ref.screenCentrePosition.elevation = -10;
         ref.aspectRatio = 1.33;
         play.screenCentrePosition.azimuth = 5;
         play.screenWidth.azimuth = 30;
         play.screenCentrePosition.elevation = 30;
         play.aspectRatio = 2.11;
    3.3.8 Screen Edge Lock
  • ADM specifies screenEdgeLock for both channels and objects. screenEdgeLock ensures that an audioObject is rendered at the edge of a playback screen. The playback screen size will be an input to the command line of the renderer and will be in the audioProgrammeReferenceScreen format.
    • Step 1. Check if the playback screen information is available. If it is not available then screenEdgeLock will be ignored and no further processing will be done with this parameter.
    • Step 2. Ensure that screenEdgeLock has been specified for a valid dimension. Left/Right is only valid for azimuth and x, Top/Bottom is only valid for elevation and z. If it is not specified for a valid dimension, screenEdgeLock will be ignored and no further processing will be done with this parameter.
    • Step 3. If the audioBlockFormat has been specified in Cartesian coordinates these will be converted to spherical coordinates using the function described in section 3.3.2 "Object and Channel Location Transformations".
    • Step 4. The audioObject must be in the front half of the room. Elevation must be in the range [-90, 90] and azimuth must be in the range [-90, 90]. If the coordinates are outside of this range then screenEdgeLock will be ignored and no further processing will be done with this parameter
    • Step 5. The playback screen information will be used to determine the spherical coordinates of the four corners of the screen. The method to calculate this information is described in section 3.3.2 "Object and Channel Location Transformations.
    • Step 6. Clip the azimuth and elevation coordinates so that they fall within the range of the screen edges and set the distance to be 1.0.
      For example if the playback screen 1910 of Fig 19A and Fig. 19B has four spherical coordinates (-30,-20,0.9), (30,-20,0.9), (30,20,0.9) and (-30,20,0.9) and an object is specified at (-45,0,0.8) with screenEdgeLock set to "Left", its coordinates will be modified so that it sits at (-30,0,1.0). If an object is specified at (45,-45,0.5) with screenEdgeLock set to "Right", its coordinates will be modified so that it sits at (30,-20,1.0). Here, coordinates are given as (azimuth, elevation, distance). Fig. 19A and Fig. 19B show examples of this behavior in two dimensions. Fig. 19A is an example of a top view of the room illustrating the clipping of the coordinates of an audio object 1920 at -45 azimuth and 0.8 distance with screenEdgeLock set to "Left". In this example, the left screen edge of the playback screen 1910 is located at -30 azimuth and 0.9 distance, and the right screen edge is located at 30 azimuth and 0.9 distance. The coordinates of the screen-edge-locked object 1930 after clipping are -30 azimuth and 1.0 distance. In Fig. 19A , the coordinates are given as (azimuth, distance). Fig. 19B is an example of a side view of the room illustrating the clipping of the coordinates of an audio object 1920 at -45 elevation and 0.5 distance with screenEdgeLock set to "Bottom". In this example, the bottom screen edge of the playback screen 1910 is located at -20 elevation and 0.9 distance, and the top screen edge is located at 20 elevation and 0.9 distance. The coordinates of the screen-edge-locked object 1930 after clipping are -20 elevation and 1.0 distance. In Fig. 19B , the coordinates are given as (elevation, distance).
    • Step 7. Convert spherical coordinates to Cartesian coordinates and modify the audioBlockFormat to these new coordinates. The audioObject can now be rendered.
    3.3.9 Importance
  • The ADM metadata provides for the specification of importance both of an audioPackFormat and an audioObject. The ADM baseline renderer takes inputs related to importance called <importance> and <obj_importance>, both ranging from 0 to 10. audioPackFormats with an importance value less than the <importance> parameter will be ignored by the metadata pre-processor 110. Within audio packs that will be rendered, objects with audioObject.importance less than <obj_importance> will be ignored by the metadata pre-processor 110.
  • 3.3.10 Frequency
  • ADM allows audioChannelFormat elements to contain optional frequency parameters specifying frequency ranges of audio data. The baseline renderer treats this element of ADM as purely informational as has no direct influence on the renderer output. Explicitly no frequency information is required for LFE channels and no low pass characteristic is enforced on sub-woofer speaker outputs. However, because future processing stages in the playback system may choose to do something with this information, frequency metadata shall be passed through to the output LFE channels. See section Error! Reference source not found.3.2.4 "LFE Channels and Sub-Woofer Speakers" for more details regarding LFE channels and sub-woofer speaker rendering.
  • 3.4 Ramping Mixer
  • The ramping mixer combines the input object audio PCM samples to create speaker feeds using the gains calculated in the source panner 120. The gains are crossfaded from their previous values over a length of time determined by the object's metadata.
  • For efficiency, the ramping mixer operates on time slot intervals of SL = 32 samples. For each slot sn, the metadata update for object i is represented by a new vector of speaker gains, G ij M ,
    Figure imgb0122
    and the number of slots remaining before the metadata update should be completed, Ωi, whose calculation is described in the next section.
  • If Ωi = 0, the speaker gains are updated immediately via G ij R = G ij M
    Figure imgb0123
    and the ramp delta is zeroed R ij Δ = 0 .
    Figure imgb0124
    Otherwise a new ramp delta for each object is calculated via R ij Δ = G ij M G ij R / Ω i .
    Figure imgb0125
  • For each slot sn, each active object's PCM data is mixed into the speaker feeds yj . y j sn SL + n = i x i sn SL + n G ij R + R ij Δ n SL , n = 0.. SL 1
    Figure imgb0126
  • The slots remaining and current gains are also updated: G ij R = G ij R + R ij Δ
    Figure imgb0127
    Ω i = max 0 , Ω i 1
    Figure imgb0128
  • These are stored in state for the next slot.
  • 3.4.1 JumpPosition
  • This metadata feature controls the cross-fade of an object's position from its previous position. The crossfade length is determined by the object's metadata. For efficiency reasons, the crossfade length is rounded to a whole number of SL = 32 sample slots, denoted Ωi. The cross-fade is implemented directly by the ramping mixers 130, 140. This section details the calculation of Ωi.
  • To simplify notation, the following symbols are used to refer to ADM metadata fields:
    • t1 audioObject.start,
    • t2 audioBlockFormat.rtime,
    • tB, audioBlockFormat.duration,
    • tI audioBlockFormat.interpolationLength.
    • jp audioBlockFormat.jumpPosition.
  • Let Fs denote the sample rate. For each time slot sn, updates due to audioBlockFormat metadata are applied in time sequential order - i.e., for the last audioBlockFormat for which (t1 + t2). Fs < (sn + 1). SL, the new gains G ij M
    Figure imgb0129
    are calculated using the audioBlockFormat metadata by the source panner 120.
  • The cross-fade duration is Ωi = round t B . F s SL
    Figure imgb0130
    when jp = 0 or Ωi = round t I . F s SL ,
    Figure imgb0131
    otherwise. In either case Ωi is forced to be at least 1, to ensure no audio glitches occur.
  • The new gains calculated from an audioBlockFormat metadata item will not be reached until time t1 + t2 plus the cross-fade duration.
  • The newly calculated gains G ij M
    Figure imgb0132
    and slots-remaining Ωi will be used by the ramping mixers 130, 140.
  • 3.5 Diffuse Ramping Mixer
  • The diffuse ramping mixer 140 combines the input object audio PCM samples using the gains calculated in the source panner 120 to feed the speaker decorrelator 150. The gains may be crossfaded from their previous values over a length of time determined by the object's metadata.
  • On the diffuse path, all objects are panned to the center of the room, so the speaker gains have the property G ij = g i G j ʹ .
    Figure imgb0133
    The speaker-dependent part of the gain G j ʹ
    Figure imgb0134
    is fixed by the speaker layout and so is applied directly in the decorrelator block. The diffuse ramping mixer 140 thus down-mixes all the objects to a single mono channel yD using the gains g i .
    Figure imgb0135
  • The equations for the diffuse ramping mixer 140 are identical to the ramping mixer 130 except there is no-longer any speaker dependence.
  • 3.6 Speaker Decorrelator
  • The Speaker Decorrelator 150 takes the down-mixed channel yD from the diffuse ramping mixer 140, and the diffuse speaker gains G j ʹ
    Figure imgb0136
    and creates the diffuse speaker feeds y j ʹ .
    Figure imgb0137
  • To create the effect of diffuseness, and prevent collapse, it is necessary to introduce decorrelation. The core decorrelation will first be described, followed by improvements to the transient response, and finally distribution to speakers.
  • 3.6.1 Core Decorrelator
  • The design makes use of one decorrelation filter per speaker pair. A large number of orthogonal decorrelation filters may lead to audible decorrelation artefacts. Therefore, a maximum of four unique decorrelation filters are implemented. For larger numbers of speakers the decorrelation filter outputs are re-used.
  • Each decorrelation filter consists of four all-pass filter sections APns in series, where n indexes over the decorrelation filters, and s indexes over the all-pass sections within a decorrelation filter. Fig. 20 illustrates an example of the four decorrelation filters and their respective all-pass filter sections. Each all-pass filter section consists of a single parameter CDS and a delay line with delay ds . An example of the all-pass section is illustrated in Fig. 21 and implements the difference equation y n = C Ds x n + x n d s C Ds y n d s .
    Figure imgb0138
    The delay for the all-pass section is calculated via R s = 3 s 1 / 4
    Figure imgb0139
    d s = ceil τ . F s . R s / s = 0 3 R s ,
    Figure imgb0140
    where Fs is the sample rate, and τ is chosen to be 20ms and does not vary across decorrelation filters n. The coefficient CDS is given by CDS = 0.4 ∗ Hadamard4(n, s).
  • 3.6.2 Improving the Transient Response
  • The transient response of the decorrelators is improved by ducking the input upon detecting a quick rise in the signal envelope, and ducking the output upon detecting a quick fall in envelope. An example of the full decorrelator structure is shown in Fig. 22 .
  • The decorrelator blocks are fed by a look-ahead delay to compensate for the ducking calculation latency. The look-ahead delay is 2ms.
  • The ducking calculation first works by creating fast and slow smoothed envelope estimates. The input yD is high-pass filtered with a single-pole filter having cut-off frequency of 3kHz, then the absolute value is taken and an offset of ε = 1 × 10-5 is added. The result is then smoothed with a single-pole smoother with slow time constant of 80ms, and a fast time constant of 5ms to produce eslow and efast , respectively.
  • The rise transient ducking gain is smoothed towards 1 using dg r n = dg r n 1 1 c dr + 1 ,
    Figure imgb0141
    where cdr is chosen to give a time constant of 50ms and follows the transient during a rise via dg r n = 1.1 e slow e fast , if 1.1 e slow < dg r n e fast .
    Figure imgb0142
    Similarly the fall transient ducking gain is also smoothed towards 1 using dg f n = dg f n 1 1 c df + 1 ,
    Figure imgb0143
    where cdf is also chosen to give a time constant of 50ms and follows the transient during a fall via dg f n = 1.1 e fast e slow , if 1.1 e fast < dg f n e slow .
    Figure imgb0144
  • In the yD mix block, the original downmix signal yD is mixed with the ducked decorrelation filter signal, with yD receiving a mix coefficient of 0.9 and the ducked decorrelation filter signal receiving a mix coefficient of 0.3.
  • The negation of each yD mix block gives another decorrelated output. These decorrelated outputs are then multiplied by the appropriate speaker gain G j ʹ
    Figure imgb0145
    and distributed to the speakers.
  • 3.6.3 Speaker Distribution
  • The section describes how the decorrelated outputs will map to speakers for specific speaker layouts. Symbol 'D1' will denote the output of the decorrelator 1 block and '-D1' the negated output of the decorrelator 1 block. Since there are only up to 8 outputs from the decorrelator blocks, some outputs are re-used on the larger speaker layouts. On the smaller speaker layouts some decorrelator blocks will not be required.
  • Layouts are described in the notation U+M+L. Where U is the number of speakers on the upper ring, M is the number of speakers on the middle ring, and L is the number of speakers on the lower ring. The particular speaker on a ring is represented in the format by its azimuth angle measured counter clockwise from center. Table 5 - Decorrelator speaker distribution for Layout A (0+2+0)
    Speaker Decorrelation
    M-030 D1
    M+030 -D1
    Table 6 - Decorrelator speaker distribution for Layout B (0+5+0)
    Speaker Decorrelation
    M+000 none
    M-O30 D1
    M+030 -D1
    M-110 D2
    M+110 -D2
    Table 7 - Decorrelator speaker distribution for Layout C (2+5+0)
    Speaker Decorrelation
    M+000 none
    M-030 D1
    M+030 -D1
    M-110 D2
    M+110 -D2
    U-030 D3
    U+030 -D3
    Table 8 - Decorrelator speaker distribution for Layout D (4+5+0)
    Speaker Decorrelation
    M+000 none
    M-030 D1
    M+030 -D1
    M-110 D2
    M+110 -D2
    U-030 D3
    U+030 -D3
    U-110 D4
    U+110 -D4
    Table 9 - Decorrelator speaker distribution for Layout E (4+5+1)
    Speaker Decorrelation
    M+000 none
    M-030 D1
    M+030 -D1
    M-110 D2
    M+110 -D2
    U-030 D3
    U+030 -D3
    U-110 D4
    U+110 -D4
    B+000 none
    Table 10 - Decorrelator speaker distribution for Layout F (3+7)
    Speaker Decorrelation
    M+000 none
    M-030 D1
    M+030 -D1
    M-90 D2
    M+90 -D2
    M-135 D3
    M+135 -D3
    U-045 D4
    U+045 -D4
    U+180 none
    Table 11- Decorrelator speaker distribution for Layout G (4+9)
    Speaker Decorrelation
    M+000 none
    M-SC D1
    M+SC -D1
    M-030 D1
    M+030 -D1
    M-90 D2
    M+90 -D2
    M-135 D3
    M+135 -D3
    U+045 D4
    U-045 -D4
    U+110 -D4
    U+110 D4
    Table 12 - Decorrelator speaker distribution for Layout H (9+10+3)
    Speaker Decorrelation
    M+000 none
    M-030 D1
    M+030 -D1
    M-060 D1
    M+060 -D1
    M-090 D2
    M+090 -D2
    M-135 -D2
    M+135 +D2
    M-180 none
    U+000 none
    U-045 D3
    U+045 -D3
    U-090 D4
    U+090 -D4
    U-135 -D4
    U+135 +D4
    U+180 none
    T+000 none
    B+000 none
    B-045 -D3
    B+045 +D3
  • 4. SCENE RENDERER
  • An example of the architecture of the scene renderer 200 is illustrated in Fig. 23. The scene renderer 200 comprises a HOA panner 2310 and a mixer (e.g., HOA mixer) 2320. The scene renderer 200 is presented with input audio objects, i.e., with metadata (e.g., ADM metadata) 25 and audio data (e.g., PCM audio data) 20, and with the speaker layout 30. The scene renderer 200 outputs speaker feeds 2350 that can be combined (e.g., by addition) with the speaker feeds output by the object and channel renderer 100 and provided to the reproduction system 500.
  • In more detail, the scene renderer 200 is presented with (N + 1)2 channels of HOA input audio, with the channels sorted in the standard ACN channel ordering, such that channel number c contains the HOA component of Order l and Degree m (where -lml), such that c = 1 + l(l + 1) + m. Any LFE inputs are passed through or mixed to output LFE channels following the same rules as the channel and object renderer uses as set out in section 3.2.4 "LFE Channels and Sub-Woofer Speakers".
  • 4.1 HOA Panner
  • The scene renderer 200 may contain a Higher Order Ambisonics (HOA) Panner, which is supplied with the following metadata: N = HOA Order 1 2 3 4 5
    Figure imgb0146
    Scale = ScalingMode N 3 D , SN 3 D , FuMa
    Figure imgb0147
    SprkConfig = SpeakerConfig 1..8
    Figure imgb0148
  • The HOA Panner is responsible for generating a (N + 1)2 × NS matrix of gain coefficients, in the matrix G i , j M ,
    Figure imgb0149
    where NS is the number of speakers in the playback system (excluding LFE channels): G i , j M : 1 i N + 1 2 1 j N S
    Figure imgb0150
    This panner matrix is computed by first selecting the Reference HOA Matrix from the set of predefined matrices described in Appendix B. For example, for N = 3 (3rd order HOA) and SprkConfig = 4 (4+5+0 configuration), array H0A_Ref_HOA3_Cfg4 is chosen: RefMatrix = HOA_Ref_HOA 3 _Cfg 4
    Figure imgb0151
  • Each row of this matrix is scaled by a scale factor that depends on the HOA Scaling Mode. This scaling is performed by the following procedure:
    1. 1. Define the HOAScale[] array, of length (N + 1)2.
    2. 2. For c= 1..(N+ 1)2 {
      • define l = floor c 1
        Figure imgb0152
      • if ScalingMode == N3D,
        HOAScale[c] = 1.0
      • elseif ScalingMode == SN3D, HOAScale c = 2 l + 1
        Figure imgb0153
      • else
        HOAScale[c] = FuMaScale[c]
        }
  • In this procedure the FuMaScale[c] is derived from the Furse-Malham scaling table, as provided in Appendix B
  • The G i , j M
    Figure imgb0154
    coefficients are then created by the following process:
    1. 1. GM is created as a (N + 1)2 × NS matrix (where NS is the number of speakers)
    2. 2. The coefficients are then defined by scaling the coefficients in the RefMatrix array: G i , j M = RefMatrix i , j × HOAScale i 1 i N + 1 2 1 j N S
      Figure imgb0155
    4.2 HOA Mixer
  • The HOA mixer processes the (N + 1)2 input channels to produce NS output channels, by a linear mixing operation: Out j n = i = 1 N + 1 2 G i , j M × HOA i n
    Figure imgb0156
  • It should be noted that the description and drawings merely illustrate the principles of the proposed methods and apparatus. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope. Furthermore, all examples recited herein are principally intended expressly to be only for pedagogical purposes to aid the reader in understanding the principles of the proposed methods and apparatus and the concepts contributed by the inventors to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to encompass equivalents thereof.
  • The methods and apparatus described in the present document may be implemented as software, firmware and/or hardware. Certain components may e.g. be implemented as software running on a digital signal processor or microprocessor. Other components may e.g. be implemented as hardware and or as application specific integrated circuits. The signals encountered in the described methods and apparatus may be stored on media such as random access memory or optical storage media. They may be transferred via networks, such as radio networks, satellite networks, wireless networks or wireline networks, e.g. the Internet.
  • Various aspects of the present invention may be appreciated from the following enumerated example embodiments (EEEs):
    1. 1. A method of rendering input audio for playback in a playback environment, wherein the input audio includes at least one audio object and associated metadata, wherein the associated metadata indicates at least a location of the audio object, the method comprising:
      • creating two additional audio objects associated with the audio object such that respective locations of the two additional audio objects are evenly spaced from the location of the audio object, on opposite sides of the location of the audio object when seen from an intended listener's position in the playback environment;
      • determining respective weight factors for application to the audio object and the two additional audio objects; and
      • rendering the audio object and the two additional audio objects to one or more speaker feeds in accordance with the determined weight factors.
    2. 2. The method according to EEE 1, wherein the associated metadata further indicates a distance measure indicative of a distance between the two additional audio objects.
    3. 3. The method according to any one of the preceding EEEs, wherein the associated metadata further indicates a measure of relative importance of the two additional audio objects compared to the audio object; and
      the weight factors are determined based on said measure of relative importance.
    4. 4. The method according to EEE 2 or EEE 3 depending on EEE 2, further comprising:
      normalizing the weight factors based on said distance measure.
    5. 5. The method according to EEE 4, wherein the weight factors are normalized such that a sum of equal powers of the normalized weight factors is equal to a predetermined value; and
      an exponent of the normalized weight factors in said sum is determined based on the distance measure.
    6. 6. The method according to EEE 4 or 5, wherein normalization of the weight factors is performed on a sub-band basis, in dependence on frequency.
    7. 7. The method according to EEE 2 or any one of EEEs 3 to 6 depending on EEE 2, further comprising:
      • determining a set of rendering gains for mapping the audio object and the two additional audio objects to the one or more speaker feeds; and
      • normalizing the rendering gains based on said distance measure.
    8. 8. The method according to EEE 7, wherein the rendering gains are normalized such that a sum of equal powers of the normalized rendering gains for all of the one or more speaker feeds and for all of the audio objects and the two additional audio objects is equal to a predetermined value; and
      an exponent of the normalized rendering gains in said sum is determined based on said distance measure.
    9. 9. The method according to EEE 7 or 8, wherein normalization of the rendering gains is performed on a sub-band basis and in dependence on frequency.
    10. 10. A method of rendering input audio for playback in a playback environment, wherein the input audio includes at least one audio object and associated metadata, wherein the associated metadata indicates at least a location of the at least one audio object and a three-dimensional extent of the at least one audio object, the method comprising rendering the audio object to one or more speaker feeds in accordance with its three-dimensional extent, by:
      • determining locations of a plurality of virtual audio objects within a three-dimensional volume defined by the location of the audio object and its three-dimensional extent;
      • for each virtual audio object, determining a weight factor that specifies the relative importance of the respective virtual audio object; and
      • rendering the audio object and the plurality of virtual audio objects to the one or more speaker feeds in accordance with the determined weight factors.
    11. 11. The method according to EEE 10, further comprising:
      • for each virtual audio object and for each of the one or more speaker feeds, determining a gain for mapping the respective virtual audio object to the respective speaker feed; and
      • for each virtual object and for each of the one or more speaker feeds, scaling the respective gain with the weight factor of the respective virtual audio object.
    12. 12. The method according to EEE 11, further comprising:
      • for each speaker feed, determining a first combined gain depending on the gains of those virtual audio objects that lie within a boundary of the playback environment;
      • for each speaker feed, determining a second combined gain depending on the gains of those virtual audio objects that lie on said boundary; and
      • for each speaker feed, determining a resulting gain for the plurality of virtual audio objects based on the first combined gain, the second combined gain, and a fade-out factor indicative of the relative importance of the first combined gain and the second combined gain.
    13. 13. The method according to EEE 12, further comprising:
      for each speaker feed, determining a final gain based on the resulting gain for the plurality of virtual audio objects, a respective gain for the audio object, and a cross-fade factor depending on the three-dimensional extent of the audio object.
    14. 14. The method according to any one of EEEs 10 to 13, wherein the associated metadata indicates a first three-dimensional extent of the audio object in a spherical coordinate system by respective ranges of values for a radius, an azimuth angle, and an elevation angle; and
      the method further comprises:
      • determining a second three-dimensional extent in a Cartesian coordinate system as dimensions of a cuboid that circumscribes the part of a sphere that is defined by said respective ranges of the values for the radius, the azimuth angle, and the elevation angle; and
      • using the second three-dimensional extent as the three-dimensional extent of the audio object.
    15. 15. The method according to any one of EEEs 10 to 14, wherein the associated metadata further indicates a measure of a fraction of the audio object that is to be rendered isotropically with respect to an intended listener's position in the playback environment; and
      the method further comprises:
      • creating an additional audio object at a center of the playback environment and assigning a three-dimensional extent to the additional audio object such that a three-dimensional volume defined by the three-dimensional extent of the additional audio object fills out the entire playback environment;
      • determining respective overall weight factors for the audio object and the additional audio object based on the measure of said fraction; and
      • rendering the audio object and the additional audio object, weighted by their respective overall weight factors, to the one or more speaker feeds in accordance with their respective three-dimensional extents, wherein each speaker feed is obtained by summing respective contributions from the audio object and the additional audio object.
    16. 16. The method according to EEE 15, further comprising:
      applying decorrelation to the contribution from the additional audio object to the one or more speaker feeds.
    17. 17. An apparatus for rendering input audio for playback in a playback environment, wherein the input audio includes at least one audio object and associated metadata, wherein the associated metadata indicates at least a location of the audio object, the apparatus comprising:
      • a metadata processing unit configured to:
        • create two additional audio objects associated with the audio object such that respective locations of the two additional audio objects are evenly spaced from the location of the audio object, on opposite sides of the location of the audio object when seen from an intended listener's position in the playback environment; and
        • determine respective weight factors for application to the audio object and the two additional audio objects; and
      • a rendering unit configured to render the audio object and the two additional audio objects to one or more speaker feeds in accordance with the determined weight factors.
    18. 18. The apparatus according to EEE 17, wherein the associated metadata further indicates a distance measure indicative of a distance between the two additional audio objects.
    19. 19. The apparatus according to EEE 17 or 18, wherein the associated metadata further indicates a measure of relative importance of the two additional audio objects compared to the audio object; and
      the weight factors are determined based on said measure of relative importance.
    20. 20. The apparatus according to EEE 18 or EEE 19 depending on EEE 18, wherein the metadata processing unit is further configured to normalize the weight factors based on said distance measure.
    21. 21. The apparatus according to EEE 20, wherein the weight factors are normalized such that a sum of equal powers of the normalized weight factors is equal to a predetermined value; and
      an exponent of the normalized weight factors in said sum is determined based on the distance measure.
    22. 22. The apparatus according to EEE 20 or 21, wherein normalization of the weight factors is performed on a sub-band basis, in dependence on frequency.
    23. 23. The apparatus according to EEE 18 or any one of EEEs 19 to 22 depending on EEE 18, wherein the rendering unit is further configured to:
      • determine a set of rendering gains for mapping the audio object and the two additional audio objects to the one or more speaker feeds; and
      • normalize the rendering gains based on said distance measure.
    24. 24. The apparatus according to EEE 23, wherein the rendering gains are normalized such that a sum of equal powers of the normalized rendering gains for all of the one or more speaker feeds and for all of the audio objects and the two additional audio objects is equal to a predetermined value; and
      an exponent of the normalized rendering gains in said sum is determined based on said distance measure.
    25. 25. The apparatus according to EEE 23 or 24, wherein normalization of the rendering gains is performed on a sub-band basis, in dependence on frequency.
    26. 26. An apparatus for rendering input audio for playback in a playback environment, wherein the input audio includes at least one audio object and associated metadata, wherein the associated metadata indicates at least a location of the at least one audio object and a three-dimensional extent of the at least one audio object, the apparatus comprising a rendering unit for rendering the audio object to one or more speaker feeds in accordance with its three-dimensional extent,
      wherein the rendering unit is configured to:
      • determine locations of a plurality of virtual audio objects within a three-dimensional volume defined by the location of the audio object and its three-dimensional extent;
      • for each virtual audio object, determine a weight factor that specifies the relative importance of the respective virtual audio object; and
      • render the audio object and the plurality of virtual audio objects to the one or more speaker feeds in accordance with the determined weight factors.
    27. 27. The apparatus according to EEE 26, wherein the rendering unit is further configured to:
      • for each virtual audio object and for each of the one or more speaker feeds, determine a gain for mapping the respective virtual audio object to the respective speaker feed; and
      • for each virtual object and for each of the one or more speaker feeds, scale the respective gain with the weight factor of the respective virtual audio object.
    28. 28. The apparatus according to EEE 27, wherein the rendering unit is further configured to:
      • for each speaker feed, determine a first combined gain depending on the gains of those virtual audio objects that lie within a boundary of the playback environment;
      • for each speaker feed, determine a second combined gain depending on the gains of those virtual audio objects that lie on said boundary; and
      • for each speaker feed, determine a resulting gain for the plurality of virtual audio objects based on the first combined gain, the second combined gain, and a fade-out factor indicative of the relative importance of the first combined gain and the second combined gain.
    29. 29. The apparatus according to EEE 28, wherein the rendering unit is further configured to:
      for each speaker feed, determine a final gain based on the resulting gain for the plurality of virtual audio objects, a respective gain for the audio object, and a cross-fade factor depending on the three-dimensional extent of the audio object.
    30. 30. The apparatus according to any one of EEEs 26 to 29, wherein the associated metadata indicates a first three-dimensional extent of the audio object in a spherical coordinate system by respective ranges of values for a radius, an azimuth angle, and an elevation angle; and
      the apparatus further comprises a metadata processing unit configured to:
      • determine a second three-dimensional extent in a Cartesian coordinate system as dimensions of a cuboid that circumscribes the part of a sphere that is defined by said respective ranges of the values for the radius, the azimuth angle, and the elevation angle; and
      • the rendering unit is configured to use the second three-dimensional extent as the three-dimensional extent of the audio object.
    31. 31. The apparatus according to any one of EEEs 26 to 30, wherein the associated metadata further indicates a measure of a fraction of the audio object that is to be rendered isotropically with respect to an intended listener's position in the playback environment;
      the apparatus further comprises a metadata processing unit configured to:
      • create an additional audio object at a center of the playback environment and assigning a three-dimensional extent to the additional audio object such that a three-dimensional volume defined by the three-dimensional extent of the additional audio object fills out the entire playback environment;
      • determine respective overall weight factors for the audio object and the additional audio object based on the measure of said fraction; and
      • output the audio object and the additional audio object, weighted by their respective overall weight factors, to the rendering unit for rendering the audio object and the additional audio object to the one or more speaker feeds in accordance with their respective three-dimensional extents; and
      the rendering unit is configured to obtain each speaker feed by summing respective contributions from the audio object and the additional audio object.
    32. 32. The apparatus according to EEE 31, wherein the rendering unit is further configured to:
      apply decorrelation to the contribution from the additional audio object to the one or more speaker feeds.
    33. 33. A software program adapted for execution on a processor and for performing the method steps of the method according to any one of EEEs 1 to 15 when carried out on a computing device.
    34. 34. A storage medium comprising a software program adapted for execution on a processor and for performing the method steps of the method according to any one of EEEs 1 to 15 when carried out on a computing device.
    Appendix A - Cartesian coordinates for speaker layouts
  • Table 13 - Cartesian coordinates for Speaker Layout A: 0+2+0
    SP Label X Y Z isLFE
    M+030 -1.000000 1.000000 0.000000 0
    M-030 1.000000 1.000000 0.000000 0
    Table 14 - Cartesian coordinates for Speaker Layout B : 0+5+0
    SP Label X Y Z isLFE
    M+000 0.000000 1.000000 0.000000 0
    M+030 -1.000000 1.000000 0.000000 0
    M-030 1.000000 1.000000 0.000000 0
    M+110 -1.000000 -1.000000 0.000000 0
    M-110 1.000000 -1.000000 0.000000 0
    LFE1 1.000000 1.000000 -1.000000 1
    Table 15 - Cartesian coordinates for Speaker Layout C: 2+5+0
    SP Label X Y Z isLFE
    M+000 0.000000 1.000000 0.000000 0
    M+030 -1.000000 1.000000 0.000000 0
    M-030 1.000000 1.000000 0.000000 0
    M+110 -1.000000 -1.000000 0.000000 0
    M-110 1.000000 -1.000000 0.000000 0
    U+030 -1.000000 1.000000 1.000000 0
    U-030 1.000000 1.000000 1.000000 0
    LFE1 1.000000 1.000000 -1.000000 1
    Table 16 - Cartesian coordinates for Speaker Layout D: 4+5+0
    SP Label X Y Z isLFE
    M+000 0.000000 1.000000 0.000000 0
    M+030 -1.000000 1.000000 0.000000 0
    M-030 1.000000 1.000000 0.000000 0
    M+110 -1.000000 -1.000000 0.000000 0
    M-110 1.000000 -1.000000 0.000000 0
    U+030 -1.000000 1.000000 1.000000 0
    U-030 1.000000 1.000000 1.000000 0
    U+110 -1.000000 -1.000000 1.000000 0
    U-110 1.000000 -1.000000 1.000000 0
    LFE1 1.000000 1.000000 -1.000000 1
    Table 17 - Cartesian coordinates for Speaker Layout E : 4+5+1
    SP Label X Y Z isLFE
    M+000 0.000000 1.000000 0.000000 0
    M+030 -1.000000 1.000000 0.000000 0
    M-030 1.000000 1.000000 0.000000 0
    M+110 -1.000000 -1.000000 0.000000 0
    M-110 1.000000 -1.000000 0.000000 0
    U+030 -1.000000 1.000000 1.000000 0
    U-030 1.000000 1.000000 1.000000 0
    U+110 -1.000000 -1.000000 1.000000 0
    U-110 1.000000 -1.000000 1.000000 0
    B+000 0.000000 1.000000 -1.000000 0
    LFE1 1.000000 1.000000 -1.000000 1
    Table 18 - Cartesian coordinates for Speaker Layout F : 3+7+0
    SP Label X Y Z isLFE
    M+000 0.000000 1.000000 0.000000 0
    M+030 -1.000000 1.000000 0.000000 0
    M-030 1.000000 1.000000 0.000000 0
    M+090 -1.000000 0.000000 0.000000 0
    M-090 1.000000 0.000000 0.000000 0
    M+135 -1.000000 -1.000000 0.000000 0
    M-135 1.000000 -1.000000 0.000000 0
    U+045 -1.000000 1.000000 1.000000 0
    U-045 1.000000 1.000000 1.000000 0
    U+180 0.000000 -1.000000 1.000000 0
    LFE1 1.000000 1.000000 -1.000000 1
    Table 19 - Cartesian coordinates for Speaker Layout G : 4+9+0
    SP Label X Y Z isLFE
    M+000 0.000000 1.000000 0.000000 0
    M+SC -0.414214 1.000000 0.000000 0
    M-SC 0.414214 1.000000 0.000000 0
    M+030 -1.000000 1.000000 0.000000 0
    M-030 1.000000 1.000000 0.000000 0
    M+090 -1.000000 0.000000 0.000000 0
    M-090 1.000000 0.000000 0.000000 0
    M+135 -1.000000 -1.000000 0.000000 0
    M-135 1.000000 -1.000000 0.000000 0
    U+045 -1.000000 1.000000 1.000000 0
    U-045 1.000000 1.000000 1.000000 0
    U+110 -1.000000 -1.000000 1.000000 0
    U-110 1.000000 -1.000000 1.000000 0
    LFE2 1.000000 1.000000 -1.000000 1
    LFE1 -1.000000 1.000000 -1.000000 1
    Table 20 - Cartesian coordinates for Speaker Layout H : 9+10+3
    SP Label X Y Z isLFE
    M+000 0.000000 1.000000 0.000000 0
    M+030 -1.000000 1.000000 0.000000 0
    M-030 1.000000 1.000000 0.000000 0
    M+060 -1.000000 0.414214 0.000000 0
    M-060 1.000000 0.414214 0.000000 0
    M+090 -1.000000 0.000000 0.000000 0
    M-090 1.000000 0.000000 0.000000 0
    M+135 -1.000000 -1.000000 0.000000 0
    M-135 1.000000 -1.000000 0.000000 0
    M+180 0.000000 -1.000000 0.000000 0
    U+000 0.000000 1.000000 1.000000 0
    U+045 -1.000000 1.000000 1.000000 0
    U-045 1.000000 1.000000 1.000000 0
    U+090 -1.000000 0.000000 1.000000 0
    U-090 1.000000 0.000000 1.000000 0
    U+135 -1.000000 -1.000000 1.000000 0
    U-135 1.000000 -1.000000 1.000000 0
    U+180 0.000000 -1.000000 1.000000 0
    T+000 0.000000 0.000000 1.000000 0
    B+000 0.000000 1.000000 -1.000000 0
    B+045 -1.000000 1.000000 -1.000000 0
    B-045 1.000000 1.000000 -1.000000 0
    LFE2 1.000000 1.000000 -1.000000 1
    LFE1 -1.000000 1.000000 -1.000000 1
    Figure imgb0157
    Figure imgb0158
    Figure imgb0159
    Figure imgb0160
    Figure imgb0161
    Figure imgb0162
    Figure imgb0163
    Figure imgb0164
    Figure imgb0165
    Figure imgb0166
    Figure imgb0167
    Figure imgb0168
    Figure imgb0169
    Figure imgb0170
    Figure imgb0171
    Figure imgb0172
    Figure imgb0173
    Figure imgb0174
    Figure imgb0175
    Figure imgb0176
    Figure imgb0177
    Figure imgb0178
    Figure imgb0179
    Figure imgb0180
    Figure imgb0181
    Figure imgb0182
    Figure imgb0183
    Figure imgb0184
    Figure imgb0185
    Figure imgb0186
    Figure imgb0187
    Figure imgb0188
    Figure imgb0189
    Figure imgb0190
    Figure imgb0191
    Figure imgb0192
    Figure imgb0193
    Figure imgb0194
    Figure imgb0195
    Figure imgb0196
    Figure imgb0197
    Figure imgb0198
    Figure imgb0199
    Figure imgb0200
    Figure imgb0201
    Figure imgb0202
    Figure imgb0203
    Figure imgb0204
    Figure imgb0205
    Figure imgb0206
    Figure imgb0207
    Figure imgb0208
    Figure imgb0209
    Figure imgb0210
    Figure imgb0211
    Figure imgb0212
    Figure imgb0213
    Figure imgb0214
    Figure imgb0215
    Figure imgb0216
    Figure imgb0217
    Figure imgb0218
    Figure imgb0219
    Figure imgb0220
    Figure imgb0221
    Figure imgb0222
    Figure imgb0223
    Figure imgb0224
    Figure imgb0225
    Figure imgb0226
    Figure imgb0227
    Figure imgb0228
    Figure imgb0229
    Figure imgb0230
    Figure imgb0231
    Figure imgb0232
  • Claims (11)

    1. A method of rendering input audio for playback in a playback environment, wherein the input audio includes at least one audio object and associated metadata, wherein the associated metadata indicates at least a location of the audio object, the method comprising:
      creating (S2910, S3010, S3110) two additional audio objects associated with the audio object such that respective locations of the two additional audio objects are evenly spaced from the location of the audio object, on opposite sides of the location of the audio object when seen from an intended listener's position in the playback environment;
      determining (S2920, S3020, S3120) respective weight factors for application to the audio object and the two additional audio objects, wherein the associated metadata includes a parameter indicating a divergence value ranging between 0 and 1, and the respective weight factors are determined based on the divergence value; and
      rendering (S2930, S3040, S3150) the audio object and the two additional audio objects to two or more speaker feeds in accordance with the determined weight factors.
    2. The method according to claim 1, wherein the associated metadata further indicates a distance measure indicative of a distance between the two additional audio objects, and further comprising:
      normalizing (S3030) the weight factors based on said distance measure.
    3. The method according to claim 2, wherein the weight factors are normalized such that a sum of equal powers of the normalized weight factors is equal to a predetermined value; and
      an exponent of the normalized weight factors in said sum is determined based on the distance measure.
    4. The method according to claim 2 or 3, wherein normalization of the weight factors is performed on a sub-band basis, in dependence on frequency.
    5. The method according to any one of claims 2 to 4, wherein the step of rendering the audio object and the two additional audio objects to the two or more speaker feeds includes:
      determining (S3130) a set of rendering gains for mapping the audio object and the two additional audio objects to the two or more speaker feeds; and
      normalizing (S3140) the rendering gains based on said distance measure.
    6. The method according to claim 5, wherein the rendering gains are normalized such that a sum of equal powers of the normalized rendering gains for all of the two or more speaker feeds and for all of the audio objects and the two additional audio objects is equal to a predetermined value; and
      an exponent of the normalized rendering gains in said sum is determined based on said distance measure.
    7. The method according to claim 5 or 6, wherein normalization of the rendering gains is performed on a sub-band basis and in dependence on frequency.
    8. An apparatus for rendering input audio for playback in a playback environment, wherein the input audio includes at least one audio object and associated metadata, wherein the associated metadata indicates at least a location of the audio object, the apparatus comprising:
      a metadata processing unit (110) configured to:
      create two additional audio objects associated with the audio object such that respective locations of the two additional audio objects are evenly spaced from the location of the audio object, on opposite sides of the location of the audio object when seen from an intended listener's position in the playback environment; and
      determine respective weight factors for application to the audio object and the two additional audio objects, wherein the associated metadata includes a parameter indicating a divergence value ranging between 0 and 1, and the respective weight factors are determined based on the divergence value; and
      a rendering unit (120, 130, 140) configured to render the audio object and the two additional audio objects to two or more speaker feeds in accordance with the determined weight factors.
    9. The apparatus according to claim 8, wherein the associated metadata further indicates a distance measure indicative of a distance between the two additional audio objects, and wherein the metadata processing unit is further configured to normalize the weight factors based on said distance measure, wherein optionally:
      the weight factors are normalized such that a sum of equal powers of the normalized weight factors is equal to a predetermined value; and
      an exponent of the normalized weight factors in said sum is determined based on the distance measure,
      wherein further optionally normalization of the weight factors is performed on a sub-band basis, in dependence on frequency.
    10. The apparatus according to claim 9, wherein the rendering unit is further configured to render the audio object and the two additional audio objects to the two or more speaker feeds at least by:
      determining a set of rendering gains for mapping the audio object and the two additional audio objects to the two or more speaker feeds; and
      normalizing the rendering gains based on said distance measure,
      wherein optionally:
      the rendering gains are normalized such that a sum of equal powers of the normalized rendering gains for all of the two or more speaker feeds and for all of the audio objects and the two additional audio objects is equal to a predetermined value; and
      an exponent of the normalized rendering gains in said sum is determined based on said distance measure,
      wherein further optionally normalization of the rendering gains is performed on a sub-band basis, in dependence on frequency.
    11. A software program adapted for execution on a processor and for performing the method steps of the method according to any one of claims 1 to 7 when carried out on a computing device.
    EP20167910.7A 2015-11-20 2016-11-18 Improved rendering of immersive audio content Active EP3706444B1 (en)

    Priority Applications (1)

    Application Number Priority Date Filing Date Title
    EP23219882.0A EP4333461A3 (en) 2015-11-20 2016-11-18 Improved rendering of immersive audio content

    Applications Claiming Priority (4)

    Application Number Priority Date Filing Date Title
    US201562257994P 2015-11-20 2015-11-20
    US201562267832P 2015-12-15 2015-12-15
    PCT/IB2016/001831 WO2017085562A2 (en) 2015-11-20 2016-11-18 Improved rendering of immersive audio content
    EP16834241.8A EP3378241B1 (en) 2015-11-20 2016-11-18 Improved rendering of immersive audio content

    Related Parent Applications (2)

    Application Number Title Priority Date Filing Date
    EP16834241.8A Division-Into EP3378241B1 (en) 2015-11-20 2016-11-18 Improved rendering of immersive audio content
    EP16834241.8A Division EP3378241B1 (en) 2015-11-20 2016-11-18 Improved rendering of immersive audio content

    Related Child Applications (2)

    Application Number Title Priority Date Filing Date
    EP23219882.0A Division EP4333461A3 (en) 2015-11-20 2016-11-18 Improved rendering of immersive audio content
    EP23219882.0A Previously-Filed-Application EP4333461A3 (en) 2015-11-20 2016-11-18 Improved rendering of immersive audio content

    Publications (2)

    Publication Number Publication Date
    EP3706444A1 true EP3706444A1 (en) 2020-09-09
    EP3706444B1 EP3706444B1 (en) 2023-12-27

    Family

    ID=57984972

    Family Applications (3)

    Application Number Title Priority Date Filing Date
    EP20167910.7A Active EP3706444B1 (en) 2015-11-20 2016-11-18 Improved rendering of immersive audio content
    EP23219882.0A Pending EP4333461A3 (en) 2015-11-20 2016-11-18 Improved rendering of immersive audio content
    EP16834241.8A Active EP3378241B1 (en) 2015-11-20 2016-11-18 Improved rendering of immersive audio content

    Family Applications After (2)

    Application Number Title Priority Date Filing Date
    EP23219882.0A Pending EP4333461A3 (en) 2015-11-20 2016-11-18 Improved rendering of immersive audio content
    EP16834241.8A Active EP3378241B1 (en) 2015-11-20 2016-11-18 Improved rendering of immersive audio content

    Country Status (4)

    Country Link
    US (3) US11128978B2 (en)
    EP (3) EP3706444B1 (en)
    ES (2) ES2971421T3 (en)
    WO (1) WO2017085562A2 (en)

    Families Citing this family (19)

    * Cited by examiner, † Cited by third party
    Publication number Priority date Publication date Assignee Title
    BR112019020887A2 (en) * 2017-04-13 2020-04-28 Sony Corp apparatus and method of signal processing, and, program.
    KR20190141669A (en) * 2017-04-26 2019-12-24 소니 주식회사 Signal processing apparatus and method, and program
    GB201710093D0 (en) 2017-06-23 2017-08-09 Nokia Technologies Oy Audio distance estimation for spatial audio processing
    GB201710085D0 (en) 2017-06-23 2017-08-09 Nokia Technologies Oy Determination of targeted spatial audio parameters and associated spatial audio playback
    WO2019149337A1 (en) * 2018-01-30 2019-08-08 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Apparatuses for converting an object position of an audio object, audio stream provider, audio content production system, audio playback apparatus, methods and computer programs
    CN108683796B (en) * 2018-04-09 2020-12-15 惠州Tcl移动通信有限公司 Audio output power control method, mobile terminal and storage medium
    GB2577885A (en) * 2018-10-08 2020-04-15 Nokia Technologies Oy Spatial audio augmentation and reproduction
    EP3761672B1 (en) * 2019-07-02 2023-04-05 Dolby International AB Using metadata to aggregate signal processing operations
    EP4179738A1 (en) * 2020-07-09 2023-05-17 Telefonaktiebolaget LM Ericsson (publ) Seamless rendering of audio elements with both interior and exterior representations
    US11388537B2 (en) * 2020-10-21 2022-07-12 Sony Corporation Configuration of audio reproduction system
    US11750745B2 (en) 2020-11-18 2023-09-05 Kelly Properties, Llc Processing and distribution of audio signals in a multi-party conferencing environment
    CN115190412A (en) * 2022-05-27 2022-10-14 赛因芯微(北京)电子科技有限公司 Method, device and equipment for generating internal data structure of renderer and storage medium
    CN115038030A (en) * 2022-05-30 2022-09-09 赛因芯微(北京)电子科技有限公司 Method, device and equipment for determining scene output rendering item and storage medium
    CN115038029A (en) * 2022-05-30 2022-09-09 赛因芯微(北京)电子科技有限公司 Rendering item processing method, device and equipment of audio renderer and storage medium
    CN115226002A (en) * 2022-05-31 2022-10-21 赛因芯微(北京)电子科技有限公司 Scene rendering item data mapping method, device, equipment and storage medium
    CN115209310A (en) * 2022-06-07 2022-10-18 赛因芯微(北京)电子科技有限公司 Method and device for rendering sound bed-based audio by using metadata
    CN115348528A (en) * 2022-06-30 2022-11-15 赛因芯微(北京)电子科技有限公司 Sound bed rendering item data mapping method, device, equipment and storage medium
    CN115426611A (en) * 2022-07-29 2022-12-02 赛因芯微(北京)电子科技有限公司 Method and apparatus for rendering object-based audio using metadata
    CN115426613A (en) * 2022-07-29 2022-12-02 赛因芯微(北京)电子科技有限公司 Method and device for rendering scene-based audio by using metadata

    Citations (6)

    * Cited by examiner, † Cited by third party
    Publication number Priority date Publication date Assignee Title
    US20060120534A1 (en) * 2002-10-15 2006-06-08 Jeong-Il Seo Method for generating and consuming 3d audio scene with extended spatiality of sound source
    WO2008113427A1 (en) * 2007-03-21 2008-09-25 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Method and apparatus for enhancement of audio reconstruction
    WO2010122441A1 (en) * 2009-04-21 2010-10-28 Koninklijke Philips Electronics N.V. Driving of multi-channel speakers
    WO2013006330A2 (en) * 2011-07-01 2013-01-10 Dolby Laboratories Licensing Corporation System and tools for enhanced 3d audio authoring and rendering
    WO2015017235A1 (en) * 2013-07-31 2015-02-05 Dolby Laboratories Licensing Corporation Processing spatially diffuse or large audio objects
    WO2015062649A1 (en) * 2013-10-30 2015-05-07 Huawei Technologies Co., Ltd. Method and mobile device for processing an audio signal

    Family Cites Families (13)

    * Cited by examiner, † Cited by third party
    Publication number Priority date Publication date Assignee Title
    GB2372923B (en) * 2001-01-29 2005-05-25 Hewlett Packard Co Audio user interface with selective audio field expansion
    KR101215868B1 (en) * 2004-11-30 2012-12-31 에이저 시스템즈 엘엘시 A method for encoding and decoding audio channels, and an apparatus for encoding and decoding audio channels
    US8073160B1 (en) * 2008-07-18 2011-12-06 Adobe Systems Incorporated Adjusting audio properties and controls of an audio mixer
    US9883310B2 (en) * 2013-02-08 2018-01-30 Qualcomm Incorporated Obtaining symmetry information for higher order ambisonic audio renderers
    ES2650541T3 (en) * 2013-03-28 2018-01-19 Dolby Laboratories Licensing Corporation Rendering of audio objects of apparent size in arbitrary speaker distributions
    SG11201507726XA (en) * 2013-03-29 2015-10-29 Samsung Electronics Co Ltd Audio apparatus and audio providing method thereof
    US20160050508A1 (en) * 2013-04-05 2016-02-18 William Gebbens REDMANN Method for managing reverberant field for immersive audio
    KR20140128564A (en) * 2013-04-27 2014-11-06 인텔렉추얼디스커버리 주식회사 Audio system and method for sound localization
    US9666198B2 (en) * 2013-05-24 2017-05-30 Dolby International Ab Reconstruction of audio scenes from a downmix
    EP2809088B1 (en) * 2013-05-30 2017-12-13 Barco N.V. Audio reproduction system and method for reproducing audio data of at least one audio object
    EP2830335A3 (en) * 2013-07-22 2015-02-25 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Apparatus, method, and computer program for mapping first and second input channels to at least one output channel
    PL3074969T3 (en) * 2013-11-27 2019-05-31 Dts Inc Multiplet-based matrix mixing for high-channel count multichannel audio
    US10349197B2 (en) * 2014-08-13 2019-07-09 Samsung Electronics Co., Ltd. Method and device for generating and playing back audio signal

    Patent Citations (6)

    * Cited by examiner, † Cited by third party
    Publication number Priority date Publication date Assignee Title
    US20060120534A1 (en) * 2002-10-15 2006-06-08 Jeong-Il Seo Method for generating and consuming 3d audio scene with extended spatiality of sound source
    WO2008113427A1 (en) * 2007-03-21 2008-09-25 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Method and apparatus for enhancement of audio reconstruction
    WO2010122441A1 (en) * 2009-04-21 2010-10-28 Koninklijke Philips Electronics N.V. Driving of multi-channel speakers
    WO2013006330A2 (en) * 2011-07-01 2013-01-10 Dolby Laboratories Licensing Corporation System and tools for enhanced 3d audio authoring and rendering
    WO2015017235A1 (en) * 2013-07-31 2015-02-05 Dolby Laboratories Licensing Corporation Processing spatially diffuse or large audio objects
    WO2015062649A1 (en) * 2013-10-30 2015-05-07 Huawei Technologies Co., Ltd. Method and mobile device for processing an audio signal

    Also Published As

    Publication number Publication date
    EP3378241A2 (en) 2018-09-26
    US11128978B2 (en) 2021-09-21
    WO2017085562A3 (en) 2017-08-24
    ES2797224T3 (en) 2020-12-01
    EP4333461A2 (en) 2024-03-06
    WO2017085562A2 (en) 2017-05-26
    ES2971421T3 (en) 2024-06-05
    US11937074B2 (en) 2024-03-19
    EP3378241B1 (en) 2020-05-13
    EP3706444B1 (en) 2023-12-27
    EP4333461A3 (en) 2024-04-17
    US20210235215A1 (en) 2021-07-29
    US20240305952A1 (en) 2024-09-12
    US20200275233A1 (en) 2020-08-27

    Similar Documents

    Publication Publication Date Title
    US11937074B2 (en) Rendering of immersive audio content
    JP7493559B2 (en) Processing spatially diffuse or large audio objects
    US11785407B2 (en) Method and apparatus for rendering sound signal, and computer-readable recording medium
    EP3028476B1 (en) Panning of audio objects to arbitrary speaker layouts
    EP3335436B1 (en) Bass management for object-based audio
    US9905231B2 (en) Audio signal processing method
    Tsingos Object-based audio
    US20230274747A1 (en) Stereo-based immersive coding
    RU2803638C2 (en) Processing of spatially diffuse or large sound objects
    Noisternig et al. D3. 2: Implementation and documentation of reverberation for object-based audio broadcasting
    KR20140128181A (en) Rendering for exception channel signal

    Legal Events

    Date Code Title Description
    PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

    Free format text: ORIGINAL CODE: 0009012

    STAA Information on the status of an ep patent application or granted ep patent

    Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

    AC Divisional application: reference to earlier application

    Ref document number: 3378241

    Country of ref document: EP

    Kind code of ref document: P

    AK Designated contracting states

    Kind code of ref document: A1

    Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

    STAA Information on the status of an ep patent application or granted ep patent

    Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

    17P Request for examination filed

    Effective date: 20210309

    RBV Designated contracting states (corrected)

    Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

    REG Reference to a national code

    Ref country code: HK

    Ref legal event code: DE

    Ref document number: 40036459

    Country of ref document: HK

    STAA Information on the status of an ep patent application or granted ep patent

    Free format text: STATUS: EXAMINATION IS IN PROGRESS

    17Q First examination report despatched

    Effective date: 20210621

    RAP3 Party data changed (applicant data changed or rights of an application transferred)

    Owner name: DOLBY INTERNATIONAL AB

    Owner name: DOLBY LABORATORIES LICENSING CORP.

    RAP3 Party data changed (applicant data changed or rights of an application transferred)

    Owner name: DOLBY INTERNATIONAL AB

    Owner name: DOLBY LABORATORIES LICENSING CORP.

    GRAP Despatch of communication of intention to grant a patent

    Free format text: ORIGINAL CODE: EPIDOSNIGR1

    STAA Information on the status of an ep patent application or granted ep patent

    Free format text: STATUS: GRANT OF PATENT IS INTENDED

    RAP3 Party data changed (applicant data changed or rights of an application transferred)

    Owner name: DOLBY INTERNATIONAL AB

    Owner name: DOLBY LABORATORIES LICENSING CORPORATION

    RIC1 Information provided on ipc code assigned before grant

    Ipc: H04R 29/00 20060101ALN20230215BHEP

    Ipc: H04R 27/00 20060101ALN20230215BHEP

    Ipc: H04R 3/00 20060101ALI20230215BHEP

    Ipc: H04S 7/00 20060101AFI20230215BHEP

    INTG Intention to grant announced

    Effective date: 20230301

    P01 Opt-out of the competence of the unified patent court (upc) registered

    Effective date: 20230428

    GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

    Free format text: ORIGINAL CODE: EPIDOSDIGR1

    STAA Information on the status of an ep patent application or granted ep patent

    Free format text: STATUS: EXAMINATION IS IN PROGRESS

    GRAP Despatch of communication of intention to grant a patent

    Free format text: ORIGINAL CODE: EPIDOSNIGR1

    STAA Information on the status of an ep patent application or granted ep patent

    Free format text: STATUS: GRANT OF PATENT IS INTENDED

    INTC Intention to grant announced (deleted)
    RIC1 Information provided on ipc code assigned before grant

    Ipc: H04R 29/00 20060101ALN20230704BHEP

    Ipc: H04R 27/00 20060101ALN20230704BHEP

    Ipc: H04R 3/00 20060101ALI20230704BHEP

    Ipc: H04S 7/00 20060101AFI20230704BHEP

    INTG Intention to grant announced

    Effective date: 20230718

    GRAS Grant fee paid

    Free format text: ORIGINAL CODE: EPIDOSNIGR3

    GRAA (expected) grant

    Free format text: ORIGINAL CODE: 0009210

    STAA Information on the status of an ep patent application or granted ep patent

    Free format text: STATUS: THE PATENT HAS BEEN GRANTED

    AC Divisional application: reference to earlier application

    Ref document number: 3378241

    Country of ref document: EP

    Kind code of ref document: P

    AK Designated contracting states

    Kind code of ref document: B1

    Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

    REG Reference to a national code

    Ref country code: GB

    Ref legal event code: FG4D

    REG Reference to a national code

    Ref country code: CH

    Ref legal event code: EP

    REG Reference to a national code

    Ref country code: DE

    Ref legal event code: R096

    Ref document number: 602016085083

    Country of ref document: DE

    REG Reference to a national code

    Ref country code: IE

    Ref legal event code: FG4D

    REG Reference to a national code

    Ref country code: NL

    Ref legal event code: FP

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: GR

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20240328

    REG Reference to a national code

    Ref country code: LT

    Ref legal event code: MG9D

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: LT

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: LT

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    Ref country code: GR

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20240328

    Ref country code: FI

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    Ref country code: BG

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20240327

    REG Reference to a national code

    Ref country code: AT

    Ref legal event code: MK05

    Ref document number: 1645658

    Country of ref document: AT

    Kind code of ref document: T

    Effective date: 20231227

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: SE

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    Ref country code: RS

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    Ref country code: NO

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20240327

    Ref country code: LV

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    Ref country code: HR

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    REG Reference to a national code

    Ref country code: ES

    Ref legal event code: FG2A

    Ref document number: 2971421

    Country of ref document: ES

    Kind code of ref document: T3

    Effective date: 20240605

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: IS

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20240427

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: CZ

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    Ref country code: AT

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: SK

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: SM

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    Ref country code: SK

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    Ref country code: RO

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    Ref country code: IS

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20240427

    Ref country code: EE

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    Ref country code: CZ

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    Ref country code: AT

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: PT

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20240429

    Ref country code: PL

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: PT

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20240429

    Ref country code: PL

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: DK

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: DK

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20231227

    PLBE No opposition filed within time limit

    Free format text: ORIGINAL CODE: 0009261

    STAA Information on the status of an ep patent application or granted ep patent

    Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT