BR122024013703A2 - METHOD AND EQUIPMENT FOR EFFECTIVE USE AND DELIVERY OF AUDIO MESSAGES FOR HIGH QUALITY EXPERIENCE - Google Patents

Abstract

This is a method and a system for a virtual reality, VR, augmented reality, AR, mixed reality, MR or 360 degree Video environment. The system can be configured to: receive at least one video stream (106) associated with an audio and video scene to be reproduced; and receive at least a first audio stream (116, 316) associated with the audio and video scene to be reproduced, wherein the system comprises: at least one media video decoder (102) configured to decode at least one video signal from the at least one Video Stream (106) for the representation of the audio and video scene to a user; and at least one media audio decoder (112) configured to decode at least one audio signal from the at least one first audio stream (116, 316) for the representation of the audio and video scene to the user; a region of interest, ROI, processor (120), configured to: decide, based at least on the user's current viewport and/or head orientation and/or motion data (122) and/or viewport metadata (131) and/or audio information message metadata (141), whether an audio information message associated with the at least one ROI should be played, wherein the audio information message is independent of the at least one video signal and the at least one audio signal; and cause, upon deciding that the information message should be played, the audio information message to be played.

Description

Description 1. Introduction

[0001] In many applications, the delivery of audio messages can enhance the user experience during media consumption. One of the most relevant applications of such messages is provided by virtual reality (VR) content. In a VR environment, or similarly in Augmented Reality (AR) or Mixed Reality (MR) or 360-degree Video environments, the user can usually view the full 360-degree content using, for example, a Head Mounted Display (HMD) and hear it through headphones (or similarly through speakers, including correct rendering depending on their position). The user can usually move around in the VR/AR space, or at least change the viewing direction - the so-called "viewport". In 360-degree video environments, which use classic (wide-screen) playback systems instead of HMDs, remote control devices can be used to emulate the user's movement in the scene, and similar principles apply. It should be noted that 360-degree content can refer to any type of content that includes more than one viewing angle at the same time that the user can choose (e.g. by head orientation or using a remote control device).

[0002] Compared to classic content consumption, for VR, content creators can no longer control what the user sees at various times - the current viewport. The user has the freedom to choose different viewports at each instance of time, outside of the allowed or available viewports.

[0003] A common problem in VR content consumption is the risk of the user missing important events in the video scene due to selecting the wrong viewport. To solve this problem, the notion of Region of Interest (ROI) was introduced and several concepts for signaling the ROI are considered. Although the ROI is commonly used to indicate to the user the region that contains the recommended viewport, it can also be used for other purposes, such as: indicating the presence of a new character/object in the scene, indicating accessibility features associated with objects in the scene, basically any feature that can be associated with an element that makes up the video scene. For example, visual messages (e.g. "Turn your head to the left") can be used and overlaid on the current viewport. Alternatively, audible sounds can be used, natural or synthetic sounds, playing them at the position of the ROI. These audio messages are known as "Earcons".

[0004] In the context of this application, the notion of Earcon will be used to characterize audio messages transmitted to signal ROIs, but the proposed signaling and processing can be used also for generic audio messages with a purpose other than signaling ROIs. An example of such audio messages is provided by audio messages to convey information/indication of various options that the user has in an interactive AR/VR/MR environment (e.g., "skip the box on your left to enter room X"). Furthermore, the VR example will be used, but the mechanisms described in this document apply to any media consumption environment.

2. Terminology and Definitions

[0005] The following terminology is used in the technical field: Audio Elements: Audio signals that can be represented for example as Audio objects, Audio channels, scene based Audio (Higher Order Ambisonics - HOA), or a combination of all. Region-of-Interest (ROI): A region of the video content (or the displayed or simulated environment) that is of interest to the user at a given moment in time. This can usually be a region on a sphere, for example, or a polygonal selection of a 2D map. The ROI identifies a specific region for a specific purpose by defining the edges of an object under consideration. User position information: location information (e.g. x, y, z coordinates), orientation information (yaw, pitch, roll), direction and speed of movement, etc. Viewport: The spherical portion of the Video that is currently displayed and viewed by the user. Viewpoint: The center point of the Viewport. 360-degree video (also known as immersive video or spherical video): In the context of this document, this represents video content that contains more than one view (i.e. viewport) in one direction at the same point in time. This content can be created, for example, using an omnidirectional camera or a collection of cameras. During playback, the viewer controls the direction of the view. Adaptation Sets contain either a media stream or a set of media streams. In the simplest case, an Adaptation Set contains all the audio and video for the content, but to reduce bandwidth, each stream can be split into a separate Adaptation Set. A common case is to have one video Adaptation Set, and multiple audio Adaptation Sets (one for each supported language). Adaptation Sets can also contain captions or arbitrary metadata. Representations allow an Adaptation Set to contain the same content encoded in multiple ways. In most cases, Representations will be provided at multiple bitrates. This allows clients to request the highest quality content they can play without waiting for buffering. Representations can also be encoded with different codecs, allowing support for clients with different supported codecs. Media Presentation Description (MPD) is an XML syntax that contains information about media segments, their relationships, and information needed to choose between them.

[0006] In the context of this application, the notions of Adaptation Sets are used more generically, sometimes actually referring to Representations. Furthermore, media streams (audio/video streams) are usually first encapsulated into Media segments which are the actual media files played by the client (e.g. DASH client). Various formats can be used for Media segments, such as ISOBMFF (ISO Base Media File Format), which is similar to the MPEG-4 and MPEG-TS container formats. Encapsulation into Media segments and different Representations/Adaptation Sets is independent of the methods described here, the methods apply to all of the various options.

[0007] Furthermore, the description of the methods in this document may be centered around a DASH Server-Client communication, but the methods are generic enough to work with other delivery environments such as MMT, MPEG-2 Transport Stream, DASH-ROUTE, File Format for file playback, etc.

3. Current solutions

[0008] Current solutions are: [1] . ISO/IEC 23008-3:2015, Information technology -- High efficiency coding and delivery of media in heterogeneous environments -- Part 3: 3D audio [2] . N16950, ISO/IEC DIS 23000-20 Omnidirectional Media Format Study [3] . M41184, Use of Earcons for ROI Identification in 360-degree Video.

[0009] A delivery mechanism for 360-degree content is provided by the Omnidirectional Media Format, ISO/IEC 23000 20 [2]. This standard specifies the media format for encoding, storing, delivering, and rendering omnidirectional images, video, and associated audio. It provides information on the media codecs to be used for audio and video compression and additional metadata information for the correct consumption of 360-degree A/V content. It also specifies restrictions and requirements on delivery channels, such as Streaming over DASH/MMT or file-based playback.

[0010] The Earcon concept was first introduced in M41184, "Use of Earcons for ROI Identification in 360-Degree Video" [3], which provides a mechanism for signaling Earcon audio data to the user.

[0011] However, some users have reported disappointing feedback about these systems. Often, a large number of Earcons resulted in angry feedback. When designers reduced the number of Earcons, some users lost important information. Notably, each user has his or her own knowledge and experience level and prefers a system that suits him or her. Just to give one example, each user prefers the Earcons to be played at a preferred volume (independent, for example, of the volume used for other audio signals). It has proven difficult for the system designer to achieve a system that provides a good level of satisfaction for all potential users. Therefore, a solution was sought that would allow to increase the satisfaction of almost all users.

[0012] Furthermore, it has proven difficult to reconfigure systems, even for designers. For example, they had difficulty preparing new releases of Audio Streams and updating Earcons.

[0013] Furthermore, a restricted system imposes certain limitations on functionality, such as the Earcons not being able to be accurately identified in an Audio Stream. Additionally, the Earcons must always be active and may become annoying to the user if played when not needed.

[0014] Additionally, the spatial information from Earcons cannot be flagged or modified by a DASH client, for example. Easy access to this information at the systems level can enable additional features for a better user experience.

[0015] Additionally, there is no flexibility in handling various types of Earcons (e.g. natural sound, synthetic sound, DASH Client generated sound, etc.).

[0016] All these problems lead to poor user Quality of Experience. Therefore, a more flexible architecture would be preferable.

4. The current invention

[0017] In accordance with the examples, a system is provided for a virtual reality, VR, augmented reality, AR, mixed reality, MR, or 360-degree Video environment configured to: receive at least one Video Stream associated with an audio and video scene to be played; and receive at least a first Audio Stream associated with the audio and video scene to be played, wherein the system includes: at least one media video decoder configured to decode at least one video signal from the at least one Video Stream into the representation of the audio and video scene to a user; and at least one media audio decoder configured to decode at least one audio signal from the at least one first Audio Stream into the representation of the audio and video scene to the user; a region of interest, ROI, processor, coded to: decide, based on at least the user's current viewport and/or head orientation and/or motion data and/or viewport metadata and/or audio information message metadata, whether an audio information message associated with a ROI should be played, where the audio information message is independent of at least one video signal and at least one audio signal; and make, upon deciding that the information message should be played, the playback of the audio information message.

[0018] According to the examples, a system for a virtual reality, VR, augmented reality, AR, mixed reality, MR, or 360-degree Video environment configured to: receive at least one Video Stream; and receive at least a first Audio Stream, wherein the system includes: at least one media video decoder configured to decode at least one video signal from at least one Video Stream into the representation of a VR, AR, MR, or 360-degree Video environment scene to a user; and at least one media audio decoder configured to decode at least one audio signal from at least one first Audio Stream into the representation of the audio scene to the user; a region of interest, ROI, processor, encoded to: decide, based on the user's current viewport and/or head orientation and/or motion data and/or viewport metadata and/or audio information message metadata, whether to play an audio information message associated with a ROI, wherein the audio information message is an earcon; and make, upon deciding that the information message should be played, the audio information message is played.

[0019] The system may include: a metadata processor configured to receive and/or process and/or manipulate metadata of the audio information message so as to cause, upon deciding that the information message should be played, the playing of the audio information message in accordance with the metadata of the audio information message.

[0020] The ROI processor may be configured to: receive a user's current viewport and/or head position and/or orientation and/or motion data and/or other user-related data; and receive viewport metadata associated with a video signal from at least one video stream, the viewport metadata defining at least one ROI; and decide, based on at least one of the user's current viewport and/or head position and/or orientation data and/or motion data and the viewport metadata and/or other criteria, whether an audio information message associated with a ROI should be played.

[0021] The system may include: a metadata processor configured to receive and/or process and/or manipulate audio information message metadata describing the audio information message and/or audio metadata describing at least one audio signal encoded in at least one Audio Stream and/or viewport metadata, so as to cause playback of the audio information message in accordance with the audio information message metadata and/or audio metadata describing at least one audio signal encoded in at least one Audio Stream and/or viewport metadata.

[0022] The ROI processor may be configured to: if at least one ROI is outside of the user's current viewport and/or head position and/or orientation and/or motion data, cause the playback of an audio information message associated with a ROI in addition to the playback of at least one audio signal; and if at least one ROI is within the user's current viewport and/or head position and/or orientation and/or motion data, prohibit and/or disable the playback of the audio information message associated with a ROI.

[0023] The system may be configured to: receive at least one additional Audio Stream in which at least one audio information message is quantized, wherein the system further includes: at least one muxer or multiplexer for merging, under the control of the metadata processor and/or the ROI and/or another processor, packets from the at least one additional Audio Stream with packets from the at least one first Audio Stream into one Stream, based on the decision provided by the ROI processor that the at least one audio information message should be played, to cause the audio information message to be played beyond the audio scene.

[0024] The system may be configured to: receive at least one Audio Stream metadata describing at least one audio signal encoded in at least one Audio Stream; receive audio information message metadata with at least one audio information message from an Audio Stream; upon deciding that the information message should be played, modify the audio information message metadata to allow for the playback of the audio information message in addition to the playback of the at least one audio signal.

[0025] The system may be configured to: receive at least one Audio metadata describing at least one audio signal encoded in an Audio Stream; receive audio information message metadata associated with an audio information message from an Audio Stream; upon deciding that the audio information message should be played, modify the audio information message metadata to enable playback of the audio information message in addition to playback of the at least one audio signal; modify the audio metadata describing the at least one audio signal to enable merging of a first Audio stream and an additional Audio Stream.

[0026] The system may be configured to: receive at least one Audio metadata describing an audio signal encoded in an Audio Stream; receive metadata from the Audio Information Message with at least one Audio Information Message from an Audio Stream; upon deciding that the Audio Information Message should be played, providing the metadata from the Audio Information Message to a Synthetic Audio Generator to create a Synthetic Audio Stream so as to associate the metadata from the Audio Information Message with the Synthetic Audio Stream; and providing the metadata from the Synthetic Audio Stream and the Audio Information Message to a multiplexer or muxer to enable merging of the at least one Audio Stream and the Synthetic Audio Stream.

[0027] The system may be configured to: obtain the audio information message metadata from at least one additional Audio Stream in which the audio information message is encoded.

[0028] The system may include: an audio information message metadata generator configured to generate audio information message metadata based on the decision that the audio information message associated with a ROI should be played.

[0029] The system may be configured to: store, for future use, the audio information message metadata and/or the audio information message Stream.

[0030] The system may include: a synthetic audio generator configured to synthesize an audio information message based on the audio information message metadata associated with a ROI.

[0031] The metadata processor may be configured to control a muxer or multiplexer to merge, based on the audio metadata and/or audio information message metadata, packets of the audio information message Stream with packets of at least a first Audio Stream in a Stream to obtain an addition of the audio information message to an Audio Stream.

[0032] The audio information message metadata may be encoded in a configuration structure and/or in a data structure including at least one of the following: an identification tag, an integer that identifies the playback of the audio information message metadata, a message type, a status, an indication of scene dependency/non-dependency, positional data, gain data, an indication of the presence of an associated text label, number of available languages, language of the audio information message, data text length, data text of the associated text label, and/or description of the audio information message.

[0033] The metadata processor and/or the ROI processor may be configured to perform at least one of the following operations: extracting the metadata of the data information message from a Stream; modifying the metadata of the audio information message to activate the audio information message and/or set/change its position; embedding metadata back into a Stream; feeding the Stream to an additional media decoder; extracting audio metadata from at least a first Audio Stream; extracting metadata from audio information messages from an additional Stream; modifying the metadata of the audio information message to activate the audio information message and/or set/change its position; modifying audio metadata from at least a first Audio Stream taking into account the existence of the audio information message and enabling merging; feeding a Stream to the multiplexer or muxer to multiplex it based on the information received from the ROI processor.

[0034] The ROI processor may be configured to perform a local search for an additional Audio Stream in which the audio information message is encoded and/or audio information message metadata and, in the event of a non-retrieval, request the additional Audio Stream and/or audio information message metadata from a remote entity.

[0035] The ROI processor may be configured to perform a local search for an additional Audio Stream and/or audio information message metadata and, in the event of a non-retrieval, cause a synthetic audio generator to generate the Audio Information Message Stream and/or the Audio Information Message metadata.

[0036] The system may be configured to: receive at least one additional Audio Stream in which at least one audio information message associated with a ROI is included; and decode at least one additional Audio Stream if the ROI processor decides that an audio information message associated with at least one ROI should be played.

[0037] The system may include: at least a first audio decoder for decoding at least one audio signal from at least one first Audio Stream; at least one audio decoder for decoding at least one audio information message from an additional Audio Stream; and at least one mixer and/or renderer for mixing and/or overlaying the audio information message from an additional Audio Stream with the at least one audio signal from a first Audio Stream.

[0038] The system may be configured to track metrics associated with historical and/or statistical data associated with playback of the audio information message, so as to disable playback of the audio information message if the metrics are above a predetermined threshold.

[0039] The ROI processor's decision may be based on a prediction of the user's current viewport and/or head position and/or orientation and/or motion data relative to the ROI position. The system may be configured to receive at least a first Audio Stream and, upon deciding that the information message should be played, to request an Audio Message Information Stream from a remote entity.

[0040] The system may be configured to determine whether to play two audio information messages at the same time or whether a higher priority audio information message should be played with priority over a lower priority audio information message.

[0041] The system may be configured to identify an audio information message among a plurality of audio information messages encoded in an additional Audio Stream based on the address and/or position of the audio information messages in the Audio Stream.

[0042] Audio Streams can be formatted in MPEG-H 3D Audio Stream format.

[0043] The system may be configured to: receive data about the availability of a plurality of adaptation sets, the available adaptation sets including at least one audio scene adaptation set and one audio message adaptation set for an Audio Stream containing an audio information message; create, based on the decision of the ROI processor, selection data identifying which of the adaptation sets should be retrieved, the available adaptation sets including at least one audio scene adaptation set and/or at least one audio message adaptation set; and request and/or retrieve the data for the adaptation sets identified by the selection data, where each adaptation set groups different encodings for different bit rates.

[0044] The system may be such that at least one if its elements include a Dynamic Adaptive Streaming over HTTP, DASH, client and/or is configured to retrieve the data for each adaptation set using the ISO Base Media File Format, ISO BMFF, MPEG-2 Transport Stream, MPEG-2 TS.

[0045] The ROI processor may be configured to check for correspondences between the ROI and the current viewport and/or head position and/or orientation and/or motion data, to verify that the ROI is represented in the current viewport, and if the ROI is outside of the current viewport and/or head position and/or orientation and/or motion data, to audibly signal the presence of the ROI to the user.

[0046] The ROI processor may be configured to check for correspondences between the ROI and the current viewport and/or position and/or orientation and/or head motion data in order to verify that the ROI is represented in the current viewport and, if the ROI is within the current viewport and/or position and/or orientation and/or head motion data, to avoid audibly signaling the presence of the ROI to the user.

[0047] The system may be configured to receive, from a remote entity, at least one video stream associated with the video environment scene and at least one audio stream associated with the audio scene, where the audio scene is associated with the video environment scene.

[0048] The ROI processor may be configured to select, among a plurality of audio information messages to be played, the playing of a first audio information message before a second audio information message.

[0049] The system may include a cache memory for storing an audio information message received from a remote entity or synthetically generated, to reuse the audio information message at different time instances.

[0050] The audio information message may have an earcon.

[0051] The at least one video stream and/or at least one first audio stream may be part of the current video environment scene and/or audio video scene, respectively, and independently of the user's current viewport and/or the orientation and/or head movement data in the current video environment scene and/or audio video scene.

[0052] The system may be configured to request at least a first audio stream and/or at least one video stream from a remote entity in association with the audio stream and/or video environment stream, respectively, and to play an audio information message based on the user's current viewport and/or head orientation and/or motion data.

[0053] The system may be configured to request at least a first audio stream and/or at least one video stream from a remote entity in association with the audio stream and/or video environment stream, respectively, and to play an audio information message based on the user's current viewport and/or head orientation and/or motion data.

[0054] The system may be configured to request at least a first audio stream and/or at least one video stream from a remote entity in association with the audio stream and/or video environment stream, respectively, and to play an audio information message based on the user's current viewport and/or head orientation and/or motion data.

[0055] The system may be configured to check at least one of the additional criteria for playing the audio information message, further including selecting a user and/or configuring a user.

[0056] The system may be configured to check at least one of the additional criteria for playing the audio information message, further including the state of the system.

[0057] The system may be configured to check at least one of the additional criteria for playing the audio information message, including the number of plays of the audio information message that have already been performed.

[0058] The system may be configured to check at least one of the additional criteria for playing the audio information message, further including a marker in a datastream obtained from a remote entity.

[0059] According to one aspect, a system is provided that includes a client configured as the system of one of the examples above and/or below, and a remote entity configured as a server to provide at least one video stream and at least one audio stream.

[0060] The remote entity may be configured to search, in a database, intranet, internet and/or geographic network, for at least one additional Audio Stream and/or audio information message metadata, and upon retrieval, deliver at least one additional Audio Stream and/or the audio information message metadata.

[0061] The remote entity may be configured to synthesize at least one additional Audio Stream and/or generate the audio information message metadata.

[0062] According to the examples, a system is provided for a virtual reality, VR, augmented reality, AR, mixed reality, MR, or 360-degree video environment including: decoding at least one video signal of at least one video and audio scene to be played to a user; decoding at least one audio signal of at least one video and audio scene to be played; deciding, based on the user's current viewport and/or orientation and/or head motion data and/or metadata whether an audio information message associated with a ROI should be played, where the audio information message is independent of the at least one video signal and the at least one audio signal; and making, in the decision of whether the information message should be played, the playback of the audio information message.

[0063] According to the examples, a system is provided for a virtual reality, VR, augmented reality, AR, mixed reality, MR, or 360-degree Video environment including: decoding at least one video signal from at least one Video Stream into a representation of a VR, AR, MR, or 360-degree Video environment scene to a user; decoding at least one audio signal from at least a first Audio Stream into a representation of an audio scene to the user; deciding, based on the user's current viewport and/or orientation and/or head movement data and/or viewport metadata and/or audio information message metadata, whether an audio information message associated with a ROI should be played, where the audio information message is an earcon; and making, upon deciding whether the information message should be played, the playback of the audio information message.

[0064] The above and/or below methods may include: receiving and/or processing and/or manipulating metadata so as to, in deciding whether the information message should be played, play the audio information message in accordance with the metadata, such that the audio information message is part of the audio scene.

[0065] The above and/or below methods may include: playing the audio and video scene; and further deciding to play the audio information message based on the user's current viewport and/or orientation and/or head movement data and/or metadata.

[0066] The above and/or below methods may include: playing the audio and video scene; and if at least one ROI is outside of the user's current viewport and/or head position and/or orientation and/or motion data, causing the playback of an audio information message associated with a ROI in addition to the playback of at least one audio signal; and if at least one ROI is within the user's current viewport and/or head position and/or orientation and/or motion data, prohibiting and/or disabling the playback of the audio information message associated with a ROI.

[0067] In accordance with the examples, a system is provided for a virtual reality, VR, augmented reality, AR, mixed reality, MR, or 360-degree video environment configured to:

[0068] receiving at least one Video Stream; and receiving at least one first Audio Stream, wherein the system includes: at least one media video decoder configured to decode at least one video signal from at least one Video Stream into the representation of a VR, AR, MR, or 360-degree Video environment scene to a user; and at least one media audio decoder configured to decode at least one audio signal from at least one first Audio Stream into the representation of the audio scene to the user; a region of interest, ROI, processor, encoded to: decide, based on the user's current viewport and/or orientation and/or head motion data and/or metadata, whether an audio information message associated with a ROI should be played, and make, upon deciding that the information message should be played, the playback of the audio information message.

[0069] In the examples, a system for virtual reality, VR, augmented reality, AR, mixed reality, MR, or 360-degree Video is provided configured to: receive at least one Video Stream; and receive at least a first Audio Stream, wherein the system includes: at least one media video decoder configured to decode at least one video signal from at least one Video Stream into representing a VR, AR, MR, or 360-degree Video environment scene to a user; and at least one media audio decoder configured to decode at least one audio signal from a first Audio Stream into representing the audio and video scene to the user; a region of interest, ROI, processor, configured to decide, based on the user's current viewport and/or head position and/or orientation and/or motion data and/or metadata and/or other criteria, whether an audio information message associated with the at least one ROI should be played; and a metadata processor configured to receive and/or process and/or manipulate metadata so as to, upon deciding whether the information message should be played, play the audio information message in accordance with the metadata, such that the audio information message is part of the audio scene.

[0070] According to one aspect, a non-transitable storage unit is provided including instructions that, when executed by a processor, cause the processor to perform a method as above and/or below.

5. Description of the drawings

[0071] Fig. 1-5, 5a and 6 show examples of implementations;

[0072] Fig. 7 shows a method according to an example;

[0073] Fig. 8 shows an example of an implementation.

6. Examples 6.1 General examples

[0074] Fig. 1 shows an example of a system 100 for a virtual reality, VR, augmented reality, AR, mixed reality, MR or 360-degree video environment. The system 100 may be associated, for example, with a content consumption device (such as a head-mounted display or the like), which reproduces visual data on a spherical or hemispherical display closely associated with the user's head.

[0075] The system 100 may include at least one media video decoder 102 and at least one media audio decoder 112. The system 100 may receive at least one Video Stream 106 in which a video signal is encoded for representing a scene of the VR, AR, MR, or 360 degree video environment 118a to a user. The system 100 may receive at least one Audio Stream 116 in which a video signal is encoded for representing a scene of the audio environment 118b to a user.

[0076] The system 100 may also include a region of interest, ROI, processor 120. The ROI processor 120 may process data associated with a ROI. In general terms, the presence of the ROI may be signaled in the metadata of the viewport 131. The metadata of the viewport 131 may be encoded in the Video Stream 106 (in other examples, the metadata of the viewport 131 may be encoded in other Streams). The metadata of the viewport 131 may include, for example, positional information (such as coordinate information) associated with the ROI. For example, the ROI may, in the examples, be understood as a rectangle (identified by coordinates such as the position of one of the four vertices of the rectangles in the spherical video and the length of the sides of the rectangle). The ROI is typically projected onto the spherical video. The ROI is typically associated with a visible element that is believed (according to a particular configuration) to be of interest to the user. For example, the ROI can be associated with a rectangular area displayed by the content consumption device (or otherwise visible to the user).

[0077] The ROI processor 120 may, among other things, control operations of the media audio decoder 112.

[0078] The ROI processor 120 may obtain data 122 associated with the user's current viewport and/or head position and/or orientation and/or movement (in addition, virtual data associated with the virtual position may comprise, in some examples, as part of the data 122). Such data 122 may be provided at least in part, for example, by the content consumption device or by positioning/sensing units.

[0079] The ROI processor 120 may check for correspondences between the ROI and the viewport and/or the user's current position (real or virtual) and/or the orientation and/or the head movement data 122 (in examples, other criteria may be used). For example, the ROI processor may check whether the ROI is represented in the current viewport. If a ROI is only partially represented in the viewport (e.g., based on the user's head movements), it may determine, for example, whether a minimum percentage of the ROI is displayed on the screen. In any case, the ROI processor 120 is capable of recognizing whether the ROI is not represented or visible to the user.

[0080] If the ROI is considered to be outside of the user's current viewport and/or the position and/or orientation and/or movement data of head 122, ROI processor 120 may audibly signal the presence of the ROI to the user. For example, ROI processor 120 may request the playback of an audio information message (Earcon) in addition to the decoded audio signal from at least a first Audio Stream 116.

[0081] If the ROI is considered to be within the user's current viewport and/or the position and/or orientation and/or movement data of head 122, the processor may decide to avoid playing the audio information message.

[0082] The audio information message may be encoded in an Audio Stream 140, which may be the same as the Audio Stream 116 or a different Stream. The Audio Stream 140 may be generated by the system 100 or may be obtained from an external entity (e.g., server). Audio metadata, such as the Audio Information Message Metadata 141, may be defined to describe properties of the Audio Information Stream 140.

[0083] The audio information message may be superimposed (or mixed or multiplexed or merged or combined or composited) onto the encoded signal in the Audio Stream 116 or may not be selected, for example, simply based on a decision by the ROI processor 120. The ROI processor 120 may base its decision on viewport and/or position and/or orientation and/or movement data of head 122, metadata (such as viewport metadata 131 or other metadata), and/or other criteria (e.g., selections, system state, number of audio information message playbacks that have already been performed, specific functions and/or operations, preferred user settings that may disable use of the Earcons, and so on).

[0084] Metadata processor 132 may be implemented. Metadata processor 132 may be interposed, for example, between ROI processor 120 (by which it may be controlled) and media audio decoder 112 (which may be controlled from the metadata processor). In the examples, metadata processor is a section of ROI processor 120. Metadata processor 132 may receive, generate, process, and/or manipulate metadata of audio information message 141. Metadata processor 132 may also process and/or manipulate metadata of Audio Stream 116, for example, to multiplex Audio Stream 116 with the Stream of audio information message 140. Additionally or alternatively, metadata processor 132 may receive metadata of Audio Stream 116, for example, from a server (such as a remote entity).

[0085] The metadata processor 132 can therefore alter the playback of the audio scene and adapt the audio information message to particular situations and/or selections and/or states.

[0086] Some of the advantages of some implementations are discussed here.

[0087] Audio information messages may be precisely identified, for example, using audio information message metadata 141.

[0088] Audio information messages can be easily enabled/disabled, for example, by modifying metadata (by metadata processor 132). Audio information messages can be enabled/disabled based on the current viewport and ROI information (and also functions or special effects to be achieved).

[0089] Audio information messages (containing, for example, status, type, spatial information, and so on) can be easily signaled and modified by common equipment, such as a DASH (Dynamic Adaptive Streaming over HTTP) client, for example.

[0090] Easy access to the audio information message (containing, e.g., status, type, spatial information, and so on) at the system level can therefore enable additional features for an improved user experience. Therefore, the system 100 can be easily customized and allow for additional implementations (e.g., specific applications) that can be performed by personnel independent of the designers of the system 100.

[0091] Furthermore, flexibility is achieved in handling various types of audio information messages (e.g., natural sound, synthetic sound, DASH client-generated sound, etc.).

[0092] Other advantages (which will also be evident in the following examples): • Use of text labels in the metadata (as a basis for displaying something or generating the Earcon) • Adaptation of the Earcon position based on the device (if it is an HMD I want an exact location, if the speaker the best way is to use a different location - straight to a speaker). • Different classes of devices: o The Earcon metadata can be created in such a way that the Earcon is signaled to be active o Some devices will only know how to parse the metadata and play the Earcon o Some newer devices that additionally have a better ROI processor may decide to disable it if it is not needed • More information and an additional figure about adaptation sets.

[0093] Therefore, in a VR/AR environment, the user can usually view the full content in 360 degrees using, for example, a head-mounted display (HMD) and listen to it through headphones. The user can usually move around in the VR/AR space, or at least change the viewing direction - the so-called "viewport" for video. Compared to classical content consumption, for VR, content creators can no longer control what the user sees at various times - the current viewport. The user has the freedom to choose different viewports at each time instance, outside of the allowed or available viewports. To indicate to the user the region of interest (ROI), audible sounds can be used, natural or synthetic sounds, by playing them at the position of the ROI. These audio messages are known as "Earcons". This invention proposes a solution for efficient delivery of such messages and proposes an optimized receiver behavior to utilize the Earcons without affecting the user experience and content consumption. This leads to an increase in Quality of Experience. This can be achieved by using dedicated metadata and system-level metadata manipulation mechanisms to enable or disable Earcons in the final scene.

[0094] The metadata processor 132 may be configured to receive and/or process and/or manipulate the metadata 141 so as to cause, upon decision of whether the information message should be played, the playing of the Audio information message in accordance with the metadata 141. Audio signals (e.g., those to represent the scene) may be understood as part of the audio scene (e.g., an audio scene downloaded from a remote server). In general, the audio signals may be semantically meaningful to the audio scene, and all audio signals present together construct the audio scene. The audio signals may be encoded together into an audio bitstream. The audio signals may be created by the content creator and/or may be associated with the specific scene and/or may be independent of the ROI.

[0095] The audio information message (e.g., earcon) may be understood as not semantically significant to the audio scene. It may be understood as an independent sound that may be artificially generated, such as recorded sound, a recorded human voice, etc. It may also be device-dependent (a system sound generated by pressing a button on the remote control, for example). The audio information message (e.g., earcon) may be understood as being intended to guide the user in the scene, without being part of the scene.

[0096] The audio information message may be independent of the above-mentioned audio signals. According to different examples, it may be included in the same bitstream or transmitted in a separate bitstream or generated by the system 100.

[0097] An example of an audio scene composed of several audio signals could be: -- A concert hall with an Audio Scene that contains 5 audio signals: - -- Audio Signal 1: The sound of a piano - -- Audio Signal 2: The voice of a singer - -- Audio Signal 3: The Voice of Person 1 part of the audience - -- Audio Signal 4: The Voice of Person 2 part of the audience - -- Audio Signal 5: The sound created by the clock on the wall

[0098] The audio information message may be, for example, a recorded sound such as "look at the pianist" (the piano is the ROI). If the user is already looking at the pianist, the audio message will not be played.

[0099] Another example: a door (a virtual door) opens behind the user and a new person enters the room; the user is not looking there. The Earcon can be triggered, based on this (information related to the VR environment, such as virtual position), to announce to the user that something is happening behind him.

[0100] In the examples, each scene (with related audio and video streams) is transmitted from the server to the client when the user changes the environment.

[0101] The audio information message may be flexible. In particular: - the audio information message may be located in the same audio stream associated with the scene to be played; - the audio information message may be located in an additional audio stream; - the audio information message may be completely absent, but only the metadata describing the earcon may be present in the stream and the audio information message may be generated in the system; - the audio information message may be completely absent as well as the metadata describing the audio information message. In this case, the system generates both (the earcon and the metadata) based on other information about the ROI in the stream.

[0102] The audio information message is generally independent of any part of the audio signal of the audio scene and is not used for the representation of the audio scene.

[0103] Examples of systems that incorporate or include parts that incorporate system 100 are provided below.

6.2 The example in Fig. 2

[0104] Fig. 2 shows a system 200 (which may contain at least one part incorporation system 100) which is depicted herein as being subdivided into a server side 202, a media delivery side 203, a client side 204 and/or a media consumption device side 206. Each of the sides 202, 203, 204 and 206 is a system in itself and may be combined with any other system to obtain another system. Here, the audio information messages are known as Earcons, even though it is possible to generalize them to any type of audio information message.

[0105] The client side 204 may receive at least one Video Stream 106 and/or at least one Audio Stream 116 from the server side 202 via a media delivery side 203.

[0106] The delivery side 203 may be, for example, based on a communication system such as a cloud system, a network system, a geographic communication network or known media transport formats (MPEG-2 TS Transport Stream, DASH, MMT, DASH ROUTE etc.) or even a file-based storage. The delivery side 203 may be capable of performing communications in the form of electrical signals (e.g. cable, wireless etc.) and/or distributing data packets (e.g. according to a specific communication protocol) with bitstreams in which audio and video signals are encoded. The delivery side 203 may, however, be embodied by a point-to-point link, a serial or parallel connection and so on. The delivery side 203 may perform a wireless connection, for example according to protocols such as WiFi, Bluetooth and so on.

[0107] The client side 204 may be associated with a media consumption device, e.g. an HND, for example, into which the user's head may be inserted (other devices may be used, however). Therefore, the user may experience a Video and Audio scene (e.g. a VR scene) prepared by the client side 204 based on the Video and Audio data provided by the server side 202. Other implementations are, however, possible.

[0108] The server side 202 is represented here as having a media encoder 240 (which may encompass video encoders, audio encoders, subtitle encoders, etc.). This encoder 240 may be associated, for example, with an audio and video scene to be represented. The audio scene may be, for example, for recreating an environment and is associated with at least one Audio and Video Data Stream 106, 116, which may be encoded based on the position (or virtual position) reached by the user in the VR, AR, MR environment. In general terms, the Video Stream 106 encodes spherical images, of which only a part (viewports) will be seen by the user according to his/her position and movements. The Audio Stream 116 contains audio data that participates in the representation of the audio scene and is intended to be heard by a user. In accordance with the examples, the Audio Stream 116 may include audio metadata 236 (which refers to at least one audio signal that is intended to participate in the audio scene representation) and/or Earcon metadata 141 (which may describe Earcons to be played only in some cases).

[0109] System 100 is depicted here as being on the client side 204. For the sake of simplicity, media video decoder 112 is not depicted in Fig. 2.

[0110] To prepare for playback of the Earcon (or other audio information messages), the Earcon 141 metadata may be used. The Earcon 141 metadata may be understood as metadata (which may be encoded in an Audio Stream) that describes and provides attributes associated with the Earcon. Therefore, the Earcon (if played) may be based on the attributes of the Earcon 141 metadata.

[0111] Advantageously, metadata processor 132 may be implemented specifically to process metadata from Earcon 141. For example, metadata processor 132 may control the reception, processing, manipulation, and/or generation of metadata from Earcon 141. When processed, the metadata from Earcon may be represented as modified metadata from Earcon 234. For example, it is possible to manipulate the metadata from Earcon to achieve a specific effect and/or to perform audio processing operations, such as multiplexing or blending, to add the Earcon to the audio signal to be represented in the audio scene.

[0112] The metadata processor 132 may control the reception, processing, manipulation of the audio metadata 236 associated with at least one Stream 116. When processed, the audio metadata 236 may be represented as modified audio metadata 238.

[0113] The modified metadata 234 and 238 may be provided to the media audio decoder 112 (or a plurality of decoders in some examples) for playback of the audio scene 118b to the user.

[0114] A synthetic audio generator and/or storage device 246 may be provided as an optional component. The generator may synthesize an Audio Stream (to generate an Earcon that is not encoded in a Stream). The storage device allows storing (in a cache memory) Earcon Streams (for future use) that were generated by the generator and/or obtained in a received Audio Stream.

[0115] Therefore, the ROI processor 120 may decide to represent an Earcon based on the user's current viewport and/or the position and/or orientation and/or movement data of the head 122. However, the ROI processor 120 may also base its decision on criteria involving other aspects.

[0116] For example, the ROI processor may enable/disable Earcon playback based on other conditions, such as user selections or higher layer selections, for example, based on the specific application intended to be consumed. For a video game application, for example, Earcons or other audio information messages may be prevented for high levels of video game. This may be achieved simply by the metadata processor disabling Earcons in the Earcon metadata.

[0117] Furthermore, it is possible to deactivate the Earcons based on the system state: if, for example, the Earcon has already been played, its repetition can be inhibited. A timer can be used, for example, to avoid repetitions that are too fast.

[0118] The ROI processor 120 may also request controlled playback of a sequence of Earcons (e.g., the Earcons associated with all ROIs in the scene), for example, to instruct the user about the elements he/she can see. The metadata processor 132 may control this operation.

[0119] The ROI processor 120 may also modify the position of the Earcon (i.e., the spatial location in the scene) or the type of Earcon. For example, some users may prefer the Earcon to play a specific sound at the exact location/position of the ROI, while other users may prefer the Earcon to always play at a fixed location (in the center or top position “voice of God”, etc.) as a vocal sound indicating the position in which the ROI is located.

[0120] It is possible to modify the gain (e.g., to obtain a different volume) of the Earcon playback. This decision may follow a user's selection, for example. Notably, based on the ROI processor's decision, the metadata processor 132 will perform the gain modification by modifying, among the Earcon metadata associated with the Earcon, the specific attribute associated with the gain.

[0121] The original designer of the VR, AR, MR environment may also be unaware of how the Earcons will actually be rendered. For example, user selections may modify the final rendering of the Earcons. Such operation may be controlled, for example, by metadata processor 132 which may modify Earcon metadata 141 based on decisions from the ROI processor.

[0122] Thus, the operations performed on the audio data associated with the Earcon are therefore in principle independent of at least one Audio Stream 116 used to represent the audio scene and may be managed differently. The Earcons may even be generated independently of the Audio and Video Streams 106 and 116 that constitute the audio and video scene and may be produced by different and independent business groups.

[0123] Therefore, the examples allow to increase the satisfaction of the users. For example, a user can perform his own selections, for example, modifying the volume of the audio information messages, turning off the audio information messages and so on. Therefore, each user can have the experience that best suits his preference. Furthermore, the architecture obtained is more flexible. The audio information messages can be easily updated, for example, by modifying the metadata independently of the Audio Streams, and/or by modifying the Audio Information Message Streams independently of the metadata and the main Audio Streams.

[0124] The obtained architecture is also compatible with legacy systems: Legacy Audio Information Message Streams can be associated with new Audio Information Message Metadata, for example. In case of absence of a suitable Audio Information Message Stream, in the examples, the latter can be easily synthesized (and, for example, stored for later use).

[0125] The ROI processor tracks metrics associated with historical and/or statistical data associated with playback of the audio information message, so as to disable playback of the audio information message if the metrics are above a predetermined threshold (this may be used as a criterion).

[0126] The ROI processor's decision may be based, as a criterion, on a prediction of the user's current viewport and/or position and/or orientation and/or movement data of head 122 relative to the position of the ROI.

[0127] The ROI processor may also be configured to receive at least a first Audio Stream 116 and, in deciding whether the information message should be played, to request an Audio Message Information Stream from a remote entity.

[0128] The ROI processor and/or metadata generator may be configured to determine whether to play two audio information messages at the same time or to select an audio information message with higher priority, which should be played with priority over an audio information message with lower priority. To make this decision, audio information metadata may be used. A priority may, for example, be obtained by the metadata processor 132 based on values in the audio information message metadata.

[0129] In some examples, the media encoder 240 may be configured to search a database, intranet, internet, and/or geographic network for at least one additional Audio Stream and/or audio information message metadata, and upon retrieval, deliver the additional Audio Stream and/or audio information message metadata. For example, the search may be performed on client-side request.

[0130] As explained above, here a solution is proposed for efficient delivery of Earcon messages together with audio content. An optimized receiver behavior is achieved, to use the audio information messages (e.g. Earcons) without affecting the user experience and content consumption. This will lead to an increase in Quality of Experience.

[0131] This can be achieved by using dedicated metadata and system-level metadata handling mechanisms to enable or disable audio information messages in the final audio scenes. The metadata can be used in conjunction with any audio codecs and nicely complement the metadata of next-generation audio codecs (e.g., MPEG-H audio metadata).

[0132] Delivery mechanisms can be various (e.g. DASH/HLS streaming, DASH-ROUTE/MMT/MPEG-2 TS streaming, file playback, etc.). DASH delivery is considered in this application, but all concepts are valid for the other delivery options.

[0133] In most cases, audio information messages do not overlap in the time domain, i.e. at a specific moment only one ROI is defined. However, considering more advanced use cases, for example in an interactive environment where the user can change the content based on his/her selections/movements, there may also be use cases that require multiple ROIs. For this purpose, more than one audio information message may be needed at a moment. Therefore, a generic solution is described to support all the different use cases.

[0134] Delivery and processing of audio informational messages should complement existing delivery methods for next-generation audio.

[0135] One way to transmit multiple audio information messages to multiple ROIs, independent in the time domain, is to mix all the audio information messages into one audio element (e.g., audio object) with associated metadata describing the spatial position of each audio information message at different instances of time. Since the audio information messages do not overlap in time, they can be addressed independently in the single shared audio element. This audio element may contain silence (or no audio data) between the audio information messages, i.e. whenever there is no audio information message. The following mechanisms can be applied in this case: • The audio element of the common audio information message can be delivered in the same Elementary Stream (ES) with the audio scene to which it refers, or it can be delivered in an auxiliary Stream (dependent or not on the main Stream). • If the Earcon audio element is delivered in an auxiliary Stream dependent on the main Stream, the Client may request the additional Stream whenever a new ROI is present in the visual scene. • The Client (e.g., system 100) may, in the examples, request the Stream prior to the scene that requires the Earcon. • The Client may, in the examples, request the Stream based on the current viewport, i.e., if the current viewport matches the ROI, the Client may decide not to request the additional Earcon Stream. • If the Earcon audio element can be delivered in an auxiliary Stream independent of the main Stream, the Client may request, as before, the additional Stream whenever a new ROI is present in the visual scene. Additionally, the two (or more) Streams may be processed using two Media Decoders and a common Render/Mix step to mix the Earcon audio data into the final audio scene. Alternatively, a Metadata Processor may be used to modify the metadata of the two Streams and a Stream Merger to merge the two Streams. A possible implementation of such a Metadata Processor and Stream Merger is described below.

[0136] In alternative examples, multiple Earcons for multiple ROIs, independent in the time domain or overlapping in the time domain, may be delivered in multiple Audio Elements (as Audio Objects) and embedded in an Elementary Stream along with the main audio scene or in multiple Auxiliary Streams, e.g., each Earcon in an ES or a group of Earcons in an ES based on a shared property (e.g., all Earcons located on the left side share a Stream). • If all of the Earcons' audio elements are delivered in multiple Auxiliary Streams dependent on the main Stream (e.g., one Earcon per Stream or a group of Earcons per Stream), the Client may, in the examples, request an additional Stream containing the desired Earcon whenever the ROI associated with the Earcon is present in the visual scene. • The Client may request the Stream with the Earcon prior to the scene that requires the Earcon (e.g., based on the user's movements, the ROI processor 120 may execute the decision even if the ROI is not part of the scene yet). • The Client may request the Stream based on the current viewport, i.e., if the current viewport matches the ROI, the Client may decide not to request the additional Earcon Stream. • If the audio element of the Earcon (or a group of Earcons) can be delivered in an auxiliary Stream independent of the main Stream, the Client may request, as before, the additional Stream whenever a new ROI is present in the visual scene. Additionally, the two (or more) Streams may be processed using two Media Decoders and a common Render/Mix step to mix the decoded Earcon audio data into the final audio scene. Alternatively, a Metadata Processor may be used to modify the metadata of the two Streams and a Stream Merger to merge the two Streams. A possible implementation of this Metadata Processor and Stream Merger is described below.

[0137] Alternatively, a common (generic) Earcon may be used to cue all ROIs in an audio scene. This may be achieved by using the same audio content with different spatial information associated with the audio content at different time instances. In this case, the ROI processor 120 may request the metadata processor 132 to gather the Earcons associated with the ROIs in the scene and control the playback of the Earcons in sequence (e.g., upon selection by a user or upon request from the upper-layer application).

[0138] Alternatively, an Earcon may be transmitted only once and cached on the Client. The Client may reuse it for all ROIs in a video scene with different spatial information associated with the audio content at different time instances.

[0139] Alternatively, the Earcon Audio content may be synthetically generated on the Client. In conjunction with this, a Metadata Generator may be used to create the metadata to signal the Earcon spatial information. For example, the Earcon Audio content may be compressed and fed into a Media Decoder along with the main audio content and the new metadata, or it may be mixed into the final audio scene after the Media Decoder, or multiple Media Decoders may be used.

[0140] Alternatively, the audio content of the Earcon may, in the examples, be synthetically generated at the Client (e.g., under the control of metadata processor 132), while metadata describing the Earcon is already embedded in the Stream. Using Earcon type-specific signaling at the encoder, the metadata may contain the spatial information of the Earcon, the specific signaling for a "Decoder-generated Earcon", but no audio data for the Earcon.

[0141] Alternatively, the Earcon audio content may be synthetically generated at the Client, and a Metadata Generator may be used to create the metadata required to signal the Earcon spatial information. For example, the Earcon audio content may be • compressed and fed to a Media decoder along with the main audio content and new metadata; • or it may be mixed into the final audio scene after the Media Decoder; • or multiple Media Decoders may be used.

6.3 Examples of metadata for audio information messages (Earcons)

[0142] An example of audio information message metadata 141 (Earcons) as described above is provided here.

[0143] A framework to describe Earcon properties and provide possibilities to easily adjust these values:

[0144] Each identifier in the table can be considered to be associated with an attribute of the Earcon 132 metadata.

[0145] Semantics is discussed here.

[0146] numEarcons - This field specifies the number of Earcon Audio Elements available in the Stream.

[0147] Earcon_isIndependent - This flag defines whether the Earcon Audio Element is independent of an Audio Scene. If Earcon_isIndependent == 1 the Earcon Audio Element is independent of the Audio Scene. If Earcon_isIndependent == 0 the Earcon Audio Element is part of the Audio Scene and Earcon_id must have the same value as the mae_groupID associated with the Audio Element.

[0148] EarconType - This field defines the Earcon type. The table below specifies the allowed values

[0149] EarconActive This flag defines whether the Earcon is active. If EarconActive == 1 the Earcon Audio Element should be decoded and rendered in the audio scene.

[0150] EarconPosition This flag defines whether the Earcon has position information available. If Earcon_isIndependent == 0, this position information should be used instead of the Audio object metadata specified in the dynamic_object_metadata() or intracoded_object_metadata_efficient() structures.

[0151] Earcon_azimuth absolute value of the azimuth angle.

[0152] Earcon_elevation absolute value of elevation angle.

[0153] Earcon_radius absolute value of the radius.

[0154] EarconHasGain This flag defines whether the Earcon has a different Gain value.

[0155] Earcon_gain This flag sets the absolute value for the Earcon gain.

[0156] EarconHasTextLabel This flag defines whether the Earcon has an associated text label.

[0157] Earcon_numLanguages This field specifies the number of languages available for the description text label.

[0158] Earcon_Language This 24-bit field identifies the language of the Earcon description text. It contains a 3-character code as specified by ISO 639-2. Either ISO 639-2/B or ISO 639-2/T can be used. Each character is encoded in 8 bits according to ISO/IEC 8859-1 and inserted in order in the 24-bit field. EXAMPLE: French has the 3-character code "fre" which is encoded as: “0110 0110 0111 0010 0110 0101”.

[0159] Earcon_TextDataLength This field defines the length of the following group description in the bit Stream.

[0160] Earcon_TextData This field contains a description of an Earcon, i.e. a string describing the content by a high-level description. The format must follow UTF-8 according to ISO/IEC 10646.

[0161] A framework for identifying system-level Earcons and associating them with existing viewports. The two tables below provide two ways to implement this framework that can be used in different implementations: aligned(8) class EarconSample() extends SphereRegionSample { for (i = 0; i <num_regions; i++) { unsigned int(7) reserved;

[0162] or alternatively: aligned(8) class EarconSample() extends SphereRegionSample

[0163] Semantics:

[0164] hasEarcon specifies whether Earcon data is available for a region.

[0165] numRegionEarcons specifies the number of Earcons available for a region.

[0166] Earcon_id uniquely defines an ID for an Earcon element associated with the sphere region. If the Earcon is part of the Audio Scene (i.e. the Earcon is part of a group of elements identified by a mae_groupID), the Earcon_id MUST have the same value as the mae_groupID. The Earcon_id can be used for identification in the audio file/track, for example in case of DASH delivery, the Adaptation Set with EarconComponent@tag element in the MPD equal to the Earcon_id.

[0167] Earcon_track_id - is an integer that uniquely identifies an Earcon track associated with the sphere region throughout the lifetime of a performance, i.e. if the Earcon track(s) are provided within the same BMFF ISO file, the Earcon_track_id will represent the corresponding track_id of the Earcon track(s). If the Earcon is not provided within the same BMFF ISO file, this value MUST be set to zero.

[0168] For easy identification of the Earcon(s) range at the MPD level, the following Attribute/Element can be used: EarconComponent@tag:

[0169] Summary of MPD Elements and Attributes for MPEG-H Audio

[0170] For MPEG-H Audio this can be implemented by making use of the MHAS packages: • a new MHAS package for Earcon care information can be defined: PACTYP_EARCON carrying the EarconInfo() structure; • a new identification field in a generic MHAS METADATA MHAS package, to carry the EarconInfo() structure.

[0171] With respect to metadata, the metadata processor 132 may have at least some of the following capabilities: extracting the data information message metadata from a Stream; modifying the audio information message metadata to activate the audio information message and/or set/change its position and/or record/modify the audio information message text label; embedding metadata back into a Stream; feeding the Stream to an additional media decoder; extracting audio metadata from at least a first Audio Stream (116); extracting audio information message metadata from an additional Stream; modifying the audio information message metadata to activate the audio information message and/or set/change its position and/or record/modify the audio information message text label; modifying audio metadata from at least a first Audio Stream (116) taking into account the existence of the audio information message and enabling merging; feeding a Stream to the multiplexer or muxer to multiplex them based on information received from the ROI processor.

6.4 Example of Fig. 3

[0172] Fig. 3 shows a system 300 including, on the client side 204, a system 302 (client system) that may incorporate, for example, system 100 or 200.

[0173] The system 302 may include the ROI processor 120, the metadata processor 132, a decoder group 313 formed by a plurality of decoders 112.

[0174] In this example, different Audio Streams are decoded (each by a respective media audio decoder 112) and are subsequently mixed and/or rendered together to provide the final audio scene.

[0175] The at least one Audio Stream is represented here as including two Streams 116 and 316 (other examples may provide a single Stream, as in Fig. 2, or more than two Streams). These are the Audio Streams intended to reproduce the audio scene that the user is expected to experience. Here, reference is made to the Earcons, even though it is possible to generalize the concept to any audio information message.

[0176] Additionally, an Earcon Stream 140 may be provided by the media encoder 240. Based on user movements and ROIs as indicated in the viewport metadata 131 and/or other criteria, the ROI processor will cause the playback of an Earcon from the Earcon Stream 140 (also referred to as an additional Audio Stream as in addition to Audio Streams 116 and 316).

[0177] Notably, an actual representation of the Earcon will be based on the Earcon metadata 141 and the modifications performed by the metadata processor 132.

[0178] In the examples, the Stream may be requested by the system 302 (client) from the media encoder 240 (server) in case of need. For example, the ROI processor may decide whether, based on the user's movements, a specific Earcon will be needed soon and may therefore request an appropriate Earcon Stream 140 from the media encoder 240.

[0179] The following aspects can be noted from this example: • Use case: Audio data is delivered in one or more Audio Streams 116, 316 (such as a main Stream and an auxiliary Stream) while Earcons are provided in one or more additional Streams 140 (dependent or independent of the Audio Stream) • In a client-side implementation 204, ROI Processor 120 and Metadata Processor 132 are used to efficiently process Earcon information • ROI Processor 120 may receive information 122 about the current viewport (user orientation information) from the side of the media consumption device 206 used for content consumption (e.g., based on an HMD). The ROI Processor may also receive information about and the ROI flagged in the metadata (Video Viewports are flagged as in OMAF). • Based on this information, the ROI processor 120 may decide to activate one (or more) Earcons contained in the Earcon Audio Stream 140. Additionally, the ROI processor 120 may decide on a different location of the Earcons and different gain values (e.g., for a more accurate representation of the Earcon in the current space in which the content is consumed). • ROI processor 120 provides this information to metadata processor 132. • Metadata processor 132 may analyze the metadata contained in the Earcon Audio Stream and • enable the Earcon (to allow its playback) • and, if requested by ROI processor 120, modify the spatial position and gain information contained in the Earcon metadata 141. • Each Audio Stream 116, 316, 140 is then decoded and rendered (based on the user's position information) independently and the output from all media decoders is mixed together as a final step by mixer or renderer 314. A different implementation may only decode the compressed audio and provide the decoded audio data and metadata to a common overall renderer for final rendering of all audio elements (including the Earcons). • Additionally, in a Streaming environment, based on the same information, the ROI processor 120 may decide to prompt the Stream Earcon(s) 140 in advance (e.g., when the user looks in the wrong direction a few seconds before the ROI is activated.

6.5 Example of Fig. 4

[0180] Fig. 4 shows a system 400 including, on the client side 204, a system 402 (client system) that may incorporate, for example, system 100 or 200. Here, reference is made to Earcons, even though it is possible to generalize the concept to any audio information message.

[0181] The system 402 may include the ROI processor 120, the metadata processor 132, a Stream multiplexer or muxer 412. In the examples where the multiplexer or muxer 412, the number of operations to be performed by the hardware is advantageously reduced relative to the number of operations to be performed when multiple decoders and a mixer or renderer are used.

[0182] In this example, different Audio Streams are processed, based on their metadata, and multiplexed or merged at element 412.

[0183] The at least one Audio Stream is depicted herein as including two Streams 116 and 316 (other examples may provide a single Stream, as in Fig. 2, or more than two Streams). These are the Audio Streams intended to reproduce the audio scene that the user is expected to experience.

[0184] Additionally, an Earcon Stream 140 may be provided by the media encoder 240. Based on user movements and ROIs as indicated in the viewport metadata 131 and/or other criteria, the ROI processor 120 will cause the playback of an Earcon from the Earcon Stream 140 (also referred to as an additional Audio Stream as in addition to the Audio Streams 116 and 316).

[0185] Each Audio Stream 116, 316, 140 may include metadata 236, 416, 141, respectively. At least some of this metadata may be manipulated and/or processed to be provided to the Stream muxer or multiplexer 412 in which packets from the Audio Streams are merged. Accordingly, the Earcon may be represented as part of the audio scene.

[0186] The Stream muxer or multiplexer 412 may therefore provide an Audio Stream 414 with modified audio metadata 238 and modified Earcon metadata 234, which may be provided to an audio decoder 112 and decoded and played back to the user.

[0187] The following aspects can be noted from this example: • Use case: Audio data is provided in one or more Audio Streams 116, 316 (a main Stream 116 and an auxiliary Stream 316, but also a single Audio Stream can be provided) while Earcons are provided in one or more Streams 140 (dependent or independent of Audio Stream 116) • In a client-side implementation 204, ROI Processor 120 and Metadata Processor 132 are used to efficiently process Earcon information • ROI Processor 120 can receive information 122 about the current viewport (user orientation information) from the media consumption device used for content consumption (e.g., an HMD). The ROI processor 120 may also receive information about the ROI flagged in the Earcon metadata 141 (Video Viewports may be flagged in OMAF Format [Omnidirectional Media Format]). Based on this information, the ROI processor 120 may decide to activate one (or more) Earcons contained in the additional Audio Stream 140. Furthermore, the ROI processor 120 may decide on a different location of the Earcons and different gain values (e.g., for a more accurate representation of the Earcon in the current space in which the content is consumed). The ROI processor 120 may provide this information to the metadata processor 132. The metadata processor 132 may analyze the metadata contained in the Earcon Audio Stream and activate the Earcon and, if requested by the ROI processor, modify the spatial position and/or gain information and/or text labels contained in the Earcon metadata. • The metadata processor 132 may also parse the audio metadata 236, 416 of all Audio Streams 116, 316 and manipulate the audio-specific information so that the Earcon can be used as part of the audio scene (e.g., if the audio scene has a 5.1 channel base and 4 objects, the Earcon Audio element is added to the scene as the fifth object. All metadata fields are updated accordingly). • The audio data of each Stream 116, 316 and the modified audio metadata and Earcon metadata are then provided to a Stream Muxer or multiplexer which may generate based on this an Audio Stream 414 with a set of metadata (modified audio metadata 238 and modified Earcon metadata 234). • This Stream 414 may be decoded by a single audio and media decoder 112 based on the position information of the user 122. • Additionally, in a Streaming environment, based on the same information, the ROI processor 120 may decide to request the Earcon(s) Stream(s) 140 in advance (e.g., when the user looks in the wrong direction a few seconds before the ROI is activated).

6.6 Example of Fig. 5

[0188] Fig. 5 shows a system 500 including, on the client side 204, a system 502 (client system) that may incorporate, for example, system 100 or 200. Here, reference is made to Earcons, even though it is possible to generalize the concept to any audio information message.

[0189] System 502 may include ROI processor 120, metadata processor 132, a Stream multiplexer or muxer 412.

[0190] In this example, an Earcon Stream is not provided by a remote entity (on the client side), but is generated by the Synthetic Audio generator 246 (which may also have the ability to store a Stream, for subsequent reuse or to use a compressed/decompressed version of a natural sound). The Earcon metadata 141 is provided by the remote entity, for example in an Audio Stream 116 (which is not an Earcon Stream). Therefore, the Synthetic Audio generator 246 may be activated to create an Audio Stream 140 based on the attributes of the Earcon metadata 141. For example, the attributes may refer to a type of synthesized voice (natural sound, synthetic sound, spoken text, etc.) and/or text labels (the Earcon may be generated by creating synthetic sound based on the text in the metadata). In the examples, after the Earcon Stream is created, it may be stored for future reuse. Alternatively, the synthetic sound may be a generic sound that is permanently stored on the device.

[0191] A Stream muxer or multiplexer 412 may be used to merge packets of the Audio Stream 116 (and in the case also of other Streams, such as the auxiliary Audio Stream 316) with the packets of the Earcon Stream generated by the generator 246. Thereafter, an Audio Stream 414 may be obtained that is associated with the modified audio metadata 238 and the modified Earcon metadata 234. The Audio Stream 414 may be decoded by the decoder 112 and played back to the user on the side of the media consumption device 206.

[0192] The following aspects can be noted from this example: Use Case: Audio data is provided in one or more Audio Streams (a main Stream and an auxiliary Stream) No Earcon is provided from the remote device, but Earcon Metadata 141 is provided as part of the main Audio Stream (specific flagging may be used to indicate that the Earcon has no associated audio data) In a client-side implementation, ROI Processor 120 and Metadata Processor 132 are used to efficiently process Earcon information ROI Processor 120 may receive information about the current viewport (user orientation information) from the device used on the content consumption device 206 side (e.g., an HMD). ROI Processor 120 may also receive information about and ROI flagged in the metadata (Video Viewports are flagged as in OMAF). • Based on this information, the ROI processor 120 may decide to activate one (or more) Earcon(s) present in Stream 116. Additionally, the ROI processor 120 may decide on a different location of the Earcons and different gain values (e.g., for a more accurate representation of the Earcon in the current space in which the content is consumed). • ROI processor 120 may provide this information to metadata processor 132. • Metadata processor 120 may analyze the metadata contained in Audio Stream 116 and • activate an Earcon • and, if requested by ROI processor 120, modify the spatial position and gain information contained in the Earcon metadata 141. • Metadata processor 132 may also analyze the audio metadata 236, 417 of all Audio Streams (116, 316) and manipulate the audio-specific information so that the Earcon can be used as part of the audio scene (e.g., if the audio scene has a 5.1 channel base and 4 objects, the Earcon Audio element is added to the scene as the fifth object. All metadata fields are updated accordingly). • The modified Earcon metadata and ROI processor information 120 are provided to the Synthetic Audio Generator 246. The Synthetic Audio Generator 246 can create a synthetic sound based on the information received (e.g., based on the spatial position of the Earcon, a signal is generated spelling out the location). Additionally, the Earcon metadata 141 is associated with the audio data generated in the new Stream 414. • The audio data from each Stream (116, 316) and the modified audio metadata and Earcon metadata are then provided to a Stream Muxer or multiplexer that can generate based on the Audio Stream with a set of metadata (Audio and Earcon). • This Stream 414 can be decoded by a single media audio decoder 112 based on the user position information. • Alternatively or additionally, the Earcon Audio Data may be cached on the Client (e.g., from previous uses of the Earcon) • Alternatively, the output of the Synthetic Audio Generator 246 may be uncompressed audio and may be mixed into the final rendered scene • Additionally, in a Streaming environment, based on the same information, the ROI processor 120 may decide to request the Earcon Stream(s) in advance (e.g., when the user looks in the wrong direction a few seconds before the ROI is activated).

6.7 Example of Fig. 6

[0193] Fig. 6 shows a system 600 including, on the client side 204, a system 602 (client system) that may incorporate, for example, system 100 or 200. Here, reference is made to Earcons, even though it is possible to generalize the concept to any audio information message.

[0194] System 602 may include ROI processor 120, metadata processor 132, a Stream multiplexer or muxer 412.

[0195] In this example, an Earcon Stream is not provided by a remote entity (on the client side), but is generated by the Synthetic Audio generator 236 (which may also have the ability to store a Stream, for subsequent reuse).

[0196] In this example, the Earcon metadata 141 is not provided by the remote entity. The Earcon metadata is generated by a metadata generator 432 that can generate the Earcon metadata to be used (processed, manipulated, modified) by the metadata processor 132. The Earcon metadata 141 generated by the Earcon metadata generator 432 can have the same structure and/or format and/or attribute as the Earcon metadata discussed for the previous examples.

[0197] The metadata processor 132 may operate as in the example of Fig. 5. A synthetic audio generator 246 may be activated to create an Audio Stream 140 based on the attributes of the Earcon metadata 141. For example, the attributes may refer to a type of synthesized voice (natural sound, synthetic sound, spoken text, etc.), and/or the gain, and/or the activation/deactivation state, etc. In the examples, after the Earcon Stream 140 has been created, it may be cached for future reuse. It is also possible to cache the Earcon metadata generated by the Earcon metadata generator 432.

[0198] A Stream muxer or multiplexer 412 may be used to merge packets of the Audio Stream 116 (and in the case also of other Streams, such as the auxiliary Audio Stream 316) with the packets of the Earcon Stream generated by the generator 246. Thereafter, an Audio Stream 414 may be obtained that is associated with the modified audio metadata 238 and the modified Earcon metadata 234. The Audio Stream 414 may be decoded by the decoder 112 and played back to the user on the side of the media consumption device 206.

[0199] The following aspects can be noted from this example: Use case: Audio data is distributed across one or more Audio Streams (e.g., a Main Stream 116 and an Auxiliary Stream 316) No Earcon is provided on the client side 202, No Earcon metadata is provided on the client side 202 This usage may represent a solution to enable Earcons for legacy content that was created without Earcons In a client-side implementation, ROI Processor 120 and Metadata Processor 232 are used to efficiently process Earcon information ROI Processor 120 may receive information 122 about the current viewport (user orientation information) of the device used on the content consumption device side 206 (e.g., an HMD). ROI Processor 210 may also receive information about and the ROI flagged in the metadata (Video Viewports are flagged as in OMAF). • Based on this information, the ROI processor 120 may decide to activate one (or more) Earcon(s) NOT present in the Stream (116, 316). • Additionally, the ROI processor 120 may provide information about the location of the Earcons and gain values to the Earcon Metadata Generator 432. • The ROI processor 120 may provide this information to the metadata processor 232. • The metadata processor 232 may analyze the metadata contained in an Earcon Audio Stream (if present) and may: • activate the Earcon • and, if requested by the ROI processor 120, modify the spatial position and gain information contained in the Earcon metadata. • The metadata processor may also parse the audio metadata 236, 417 from all Audio Streams 116, 316 and manipulate the audio-specific information so that the Earcon can be used as part of the audio scene (e.g., if the audio scene has a 5.1 channel base and 4 objects, the Earcon Audio element is added to the scene as the fifth object. All metadata fields are updated accordingly). • The modified Earcon metadata 234 and information from the ROI processor 120 are provided to the Synthetic Audio Generator 246. The Synthetic Audio Generator 246 may create a synthetic sound based on the information received (e.g., based on the spatial position of the Earcon, a signal is generated spelling out the location). Additionally, the Earcon metadata is associated with the audio data generated in the new Stream. • Similarly as before, the audio data of each Stream and the modified audio metadata and Earcon metadata are then provided to a Stream Muxer or multiplexer 412 which can generate based on this, an Audio Stream 414 with a set of metadata (Audio and Earcon). • This Stream 414 can be decoded by a single media audio decoder based on the user's position information. • Alternatively, the Earcon Audio Data can be cached on the Client (e.g. from previous Earcon uses) • Alternatively, the output of the Synthetic Audio Generator can be uncompressed audio and can be mixed into the final rendered scene • Furthermore, in a Streaming environment, based on the same information, the ROI processor 120 can decide to request the Earcon Stream(s) in advance (e.g. when the user looks in the wrong direction a few seconds before the ROI is activated).

6.8 Example based on user position

[0200] It is possible to implement a function that allows playing an Earcon only when a user does not see the ROI.

[0201] The ROI processor 120 may periodically check, for example, the user's current viewport and/or the position and/or orientation and/or movement data of the head 122. If the ROI is visible to the user, there is no playback of the Earcon.

[0202] If, from the user's current viewport and/or head position and/or orientation and/or movement data, the ROI processor 120 determines that the ROI is not visible to the user, the ROI processor 120 may request a playback of the Earcon. In this case, the ROI processor 120 may cause the metadata processor 132 to prepare the playback of the Earcon. The metadata processor 132 may use one of the techniques described for the examples above. For example, the metadata may be retrieved in a server-side provided Stream 202, may be generated by the Earcon metadata generator 432, and so on. The attributes of the Earcon metadata may be easily modified based on requests from the ROI processor and/or various conditions. For example, if a user's selection has previously disabled the Earcon, the Earcon will not play, even if the user does not see the ROI. For example, if a timer (previously set) has not yet expired, the Earcon will not play, even if the user does not see the ROI.

[0203] If, from the user's current viewport and/or head position and/or orientation and/or movement data, the ROI processor determines that the ROI is not visible to the user, the ROI processor 120 may request that no playback of the Earcon be performed, specifically if the Earcon metadata already contains flagging for an active Earcon.

[0204] In this case, the ROI processor 120 may cause the metadata processor 132 to disable playback of the Earcon. The metadata processor 132 may use one of the techniques described for the examples above. For example, the metadata may be retrieved in a Stream provided by the server side 202, may be generated by the Earcon metadata generator 432, and so on. The attributes of the Earcon metadata may be easily modified based on requests from the ROI processor and/or various conditions. If the metadata already contains the indication that an Earcon should be played, in that case the metadata will be modified to indicate that the Earcon is inactive and should not be played.

[0205] The following aspects can be noted from this example: Use case: Audio data is delivered in one or more Audio Streams 116, 316 (e.g., a main Stream and an auxiliary Stream) while the Earcons are provided in one or more Audio Streams 116, 316 or in one or more additional Streams 140 (dependent or independent of the main Audio Stream) Earcon metadata is set to indicate that the Earcon will always be active at specific times A first generation device that does not include an ROI processor would read the Earcon metadata and cause the Earcon to play regardless of whether the user's current viewport and/or head position and/or orientation and/or movement data indicates that the ROI is visible to the user A new generation device that includes an ROI processor, as described in either system, would utilize the ROI processor's determination. If from the user's current viewport and/or head position and/or orientation and/or movement data, the ROI processor 120 determines whether the ROI is visible to the user, the ROI processor 120 may request that no playback of the Earcon be performed, especially if the Earcon metadata already contains flagging for an active Earcon. In this case, the ROI processor 120 may cause the metadata processor 132 to disable playback of the Earcon. The metadata processor 132 may use one of the techniques described for the examples above. For example, the metadata may be retrieved from a server-side provided Stream 202, it may be generated by the Earcon metadata generator 432, and so on. The attributes of the Earcon metadata may be easily modified based on requests from the ROI processor and/or various conditions. If the metadata already contains an indication that an Earcon should be played, then the metadata will be modified to indicate that the Earcon is inactive and should not be played. • Additionally, depending on the playback device, the ROI processor may decide to request modification of the Earcon metadata. For example, the Earcon spatial information may be modified differently depending on whether the sound is played through headphones or speakers.

[0206] Therefore, the final audio scene experienced by the user will be obtained based on the metadata modifications performed by the metadata processor.

6.9 Example based on Client-Server communication (Figure 5a)

[0207] Fig. 5 shows a system 550 including, on the client side 204, a system 552 (client system) that may incorporate, for example, system 100 or 200 or 300 or 400 or 500. Here, reference is made to Earcons, even though it is possible to generalize the concept to any audio information message.

[0208] System 552 may include ROI processor 120, metadata processor 132, a Stream multiplexer or muxer 412. (In the examples, different Audio Streams are decoded (each by a respective media audio decoder 112) and are subsequently mixed and/or rendered together to provide the final audio scene.)

[0209] The at least one Audio Stream is depicted herein as including two Streams 116 and 316 (other examples may provide a single Stream, as in Fig. 2, or more than two Streams). These are the Audio Streams intended to reproduce the audio scene that the user is expected to experience.

[0210] Additionally, an Earcon Stream 140 may be provided by the media encoder 240.

[0211] Audio Streams can be encoded at different bit rates, which allows efficient adaptation of the bit rate depending on the network connection (i.e. for users using high-speed connection, the high-bit-rate encoded version is provided, while for users with low-speed network connection a lower-bit-rate version is provided).

[0212] The Audio Streams may be stored on a media server 554, where for each Audio Stream, the different encodings with different bit rates are grouped into an Adaptation Set 556 with appropriate data signaling the availability of all the Adaptation Sets created. Audio adaptation sets 556 and Video adaptation sets 557 may be provided.

[0213] Based on the user's movements and ROIs as indicated in the viewport metadata 131 and/or other criteria, the ROI processor 120 will cause the playback of an Earcon from the Earcon Stream 140 (also indicated as an additional Audio Stream as in addition to the Audio Streams 116 and 316).

[0214] In this example: • the client 552 is configured to receive, from the server, data about the availability of all Adaptation Sets, the available Adaptation Sets, including: o at least one audio scene Adaptation Set for an Audio Stream; and o at least one audio message Adaptation Set for an additional Audio Stream containing at least one audio information message • Similar to the other example implementations, the ROI Processor 120 may receive information 122 about the current viewport (user orientation information) from the side of the media consumption device 206 used for content consumption (e.g., based on an HMD). The ROI Processor 120 may also receive information about and the ROI flagged in the metadata (Video Viewports are flagged as in OMAF). o Based on this information, the ROI processor 120 may decide to activate one (or more) Earcons contained in the Earcon Audio Stream 140. o Additionally, the ROI processor 120 may decide on a different location of the Earcons and different gain values (e.g., for a more accurate representation of the Earcon in the current space in which the content is consumed). o The ROI processor 120 may provide this information to a selection data generator 558. • a selection data generator 558 may be configured to create, based on the decision of the ROI processor, selection data 559 identifying which of the Adaptation Sets should be received; the Adaptation Sets, including the Audio Scene Adaptation Sets and the Audio Message Adaptation Sets • The Media Server 554 may be configured to provide instructions to the client 552 to cause the Streaming client to retrieve data for the Adaptation Sets 556, 557 identified by selection data identifying which of the Adaptation Sets are to be received, the Adaptation Sets including the Audio Scene Adaptation Sets and the Audio Message Adaptation Sets • a download and switch module 560 is configured to receive the Audio Streams from the Media Server 554 based on the selection data identifying which of the Adaptation Sets are to be received; the Adaptation Sets including the Audio Scene Adaptation Sets and the Audio Message Adaptation Sets The download and switching module 560 may be further configured to provide the audio metadata and Earcon metadata 141 to the metadata processor 132. The ROI processor 120 may provide this information to the metadata processor 132. The metadata processor 132 may parse the metadata contained in the Earcon Audio Stream 140 and activate the Earcon (to enable its playback) and, if requested by the ROI processor 120, modify the spatial position and gain information contained in the Earcon metadata 141. The metadata processor 132 may also parse the audio metadata of all Audio Streams 116, 316 and manipulate the audio-specific information so that the Earcon may be used as part of the audio scene (e.g., if the audio scene has a 5.1 channel base and 4 objects, the Earcon Audio element is added to the scene as the fifth object. All metadata fields may be updated accordingly). • The audio data of each Stream 116, 316 and the modified audio metadata and Earcon metadata are then provided to a Stream Muxer or multiplexer which can generate based on this, an Audio Stream 414 with a set of metadata (modified audio metadata 238 and modified Earcon metadata 234). • This Stream can be decoded by a single media audio decoder 112 based on the user position information 122.

[0215] An Adaptation Set may be formed by a set of Representations containing interchangeable versions of the respective content, e.g. different audio bitrates (e.g. different streams at different bitrates). While a single Representation may theoretically be sufficient to provide a playable stream, multiple Representations may offer the client the possibility to adapt the media stream to current network conditions and bandwidth requirements and therefore ensure smoother playback.

6.10 Method

[0216] All of the above examples may be implemented by method steps. Here, a method 700 (which may be realized by any of the above examples) is described for completeness. The method may include: In step 702, receiving at least one Video Stream (106) and at least a first Audio Stream (116, 316); In step 704, decoding at least one video signal from the at least one Video Stream (106) into the representation of a VR, AR, MR, or 360-degree Video environment scene (118a) to a user; and In step 706, decoding at least one audio signal from the at least one first Audio Stream (116, 316) into the representation of an audio scene (118b) to the user; receiving a user's current viewport and/or the position and/or orientation and/or head movement data and/or other user-related data (122); and In step 708, receiving viewport metadata (131) associated with a video signal from at least one Video Stream (106), the viewport metadata defining at least one ROI; and In step 710, deciding, based on the user's current viewport and/or orientation and/or head position and/or movement data and the viewport metadata (122) and/or other criteria, whether an audio information message associated with a ROI should be played. In step 712, receiving, processing and/or manipulating the audio information message metadata (141) describing the audio information message, so as to cause playback of the audio information message in accordance with the attributes of the audio information message, such that the audio information message is part of the audio scene.

[0217] Notably, the sequence may also vary. For example, the receiving steps 702, 706, 708 may have a different order, according to the actual order in which the information is provided.

[0218] Line 714 refers to the fact that the physician may be reiterated. Step 712 may be skipped in case the ROI processor decides not to play the audio information message.

6.11 Other implementations

[0219] Fig. 8 shows a system 800 that may implement one of the systems (or a component thereof) or execute method 700. System 800 may include a processor 802 and a non-transitory memory unit 806 that store instructions that, when executed by processor 802, may cause the processor to perform at least the Stream processing operations discussed above and/or the metadata processing operations discussed above. System 800 may include an input/output unit 804 for connecting to external devices.

[0220] System 800 may implement at least some (or all) of the functions of ROI processor 120, metadata processor 232, generator 246, muxer or multiplexer 412, decoder 112m, Earcon metadata generator 432, etc.

[0221] Depending on certain implementation requirements, examples may be implemented in hardware. The implementation may be realized using a digital storage medium, e.g., a floppy disk, a digital versatile disc (DVD), a Blu-Ray disc, a compact disc (CD), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory, with electronically readable control signals stored therein, which cooperate (or are capable of cooperating) with a programmable computer system so that the respective method is executed. Therefore, the digital storage medium may be computer-readable.

[0222] Generally, the examples may be implemented as a computer program product with program instructions, the program instructions being operative to execute one of the methods when the computer program product is executed on a computer. The program instructions may, for example, be stored on a machine-readable medium.

[0223] Other examples include the computer program for executing one of the methods described herein, stored in a machine-readable medium. In other words, an example method is therefore a computer program with program instructions for executing one of the methods described herein, when the computer program is executed on a computer.

[0224] A further example of the methods is therefore a data-bearing medium (or a digital storage medium or a computer-readable medium) including, recorded thereon, the computer program for executing one of the methods described herein. The data-bearing medium, the digital storage medium or the recorded medium are tangible and/or non-transitory, rather than intangible and transient signals.

[0225] Another example includes a processing unit, e.g., a computer or a programmable logic device, that performs one of the methods described in this document.

[0226] Another example includes a computer with the computer program recorded to perform one of the methods described herein.

[0227] Another example includes an apparatus or system that transfers (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device, or the like. The apparatus or system may, for example, include a file server for transferring the computer program to the receiver.

[0228] In some examples, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some examples, a field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods may be performed by an appropriate hardware apparatus.

[0229] The examples described above are illustrative of the principles discussed. It is understood that modifications and variations of the arrangements and details described herein will be apparent. Therefore, the intent is to limit oneself to the scope of the imminent patent claims and not to the specific details presented by way of description and explanation of the examples presented herein.

Claims (31)

1. A system of content consumption devices configured to receive at least one first Audio Stream (116, 316) associated with a scene to be played, at least one audio signal being encoded in the Audio Stream, the system of content consumption devices comprising: at least one media audio decoder (112) for decoding at least one audio signal from a first Audio Stream (116, 316) into the representation of the scene to the user; and a first processor configured to: decide, based on at least user motion data and/or user selection and/or metadata, whether an audio information message should be played, the audio information message being uncompressed and independent of at least one audio signal; and cause, when it is decided that the audio information message should be played, a playback of the audio information message. 2. The content consumption device system of claim 1, further comprising a metadata processor (132) configured to receive and/or generate and/or process and/or modify the metadata of the audio information message so as to cause, upon deciding that the audio information message should be played, the playback of the audio information message in accordance with the metadata of the audio information message. 3. The content consumption device system of claim 2, the system further comprising generating the audio information message based on the metadata of the audio information message. 4. The content consumption device system of any one of claims 1 to 3, the system further comprising an audio generator for generating the audio information message. 5. The content consumption device system of any one of claims 1 to 4, the system further comprising a mixer for mixing the audio information message with at least one audio signal. 6. The content consumption device system of any one of claims 1 to 5, characterized in that the audio information message is associated with an accessibility feature associated with an object in the scene. 7. The content consumption device system of any one of claims 1 to 6, the system further comprising a multiplexer or muxer for merging the audio information message with the at least one audio signal. 8. The content consumption device system of any one of claims 1 to 7, wherein the audio information message is an Earcon. 9. The content consumption device system of claim 2 or 3, wherein the metadata processor is configured to perform at least one of the following operations: embedding metadata into a Stream; extracting audio metadata from at least a first Audio Stream; modifying the metadata of the audio information message to activate the audio information message and/or set/change its position; and modifying the audio metadata of at least a first Audio Stream so as to take into account the existence of the audio information message and enable merging. 10. The content consumption device system of claim 2, 3 or 9, the system further configured to store, for future use, the metadata of the audio information message. 11. The content consumption device system of claim 2, 3, 9, or 10, wherein the audio information message metadata comprises at least one of the following: an identification tag, an audio information message type, a status, an indication of scene dependency/non-dependency, positional data, an indication of the presence of an associated text label, a number of available languages, a data text length, and a description of the audio information message. 12. The content consumption device system of any one of claims 2, 3, 9, 10 or 11, wherein the audio information message metadata includes at least one of the gain data. 13. The content consumption device system of any one of claims 2, 3, 9, 10, 11 or 12, wherein the metadata of the audio information message includes at least one of the languages of the audio information message. 14. The content consumption device system of any one of claims 1 to 13, the system further configured to perform at least one of the following operations: receiving at least one audio metadata describing at least one audio signal encoded in at least one first Audio Stream; processing at least one audio information message associated with at least one accessibility feature; generating audio information message metadata; generating, based on the audio information message metadata, the audio information message; and multiplexing or mixing the at least one first Audio Stream and the audio information message into a single audio signal. 15. The content consumption device system of any one of claims 1 to 14, the system further configured to: process an Audio Stream that includes at least one encoded audio signal and at least one uncompressed audio information message and audio information message metadata; and decode and play back the at least one audio signal and the at least one audio information message based on the audio information message metadata. 16. The content consumption device system of any one of claims 1 to 15, the system characterized in that it is configured to perform a local search for the audio information message and, in case of non-retrieval, request metadata of the audio information message from a remote entity. 17. The content consumption device system of claim 3, the system configured to perform a local search for the audio information message and, if not retrieved, cause the audio generator to generate the audio information message. 18. The content consumption device system of any one of claims 1 to 17, wherein the Audio Streams are formatted in the MPEG-H 3D Audio Stream format. 19. The content consumption device system of any one of claims 1 to 18, the system further configured to: receive data about the availability of a plurality of adaptation sets, the available adaptation sets comprising at least one audio scene adaptation set for at least one first Audio Stream and at least one audio information message adaptation set for the audio information message; create, based on the decision of the first processor, selection data identifying which of the adaptation sets are to be retrieved, the available adaptation sets comprising at least one audio scene adaptation set and/or at least one audio information message adaptation set; and request and/or retrieve the data for the adaptation sets identified by the selection data, wherein each adaptation set groups different encodings for different bit rates. 20. The content consumption device system of any one of claims 1 to 19, the system configured to choose, among a plurality of audio information messages to be played, to play a first audio information message before a second audio information message. 21. The content consumption device system of any one of claims 1 to 20, wherein the cache memory stores an audio information message received from a remote entity or synthetically generated, to reuse the audio information message at different time instances. 22. The content consumption device system of any one of claims 1 to 21, the system being configured to check at least one criterion for playing the audio information message, the criterion comprising a user selection and/or a user configuration. 23. The content consumption device system of claim 22, the system being configured to verify whether the user has disabled settings for playback of audio information messages or system sounds. 24. The content consumption device system of any one of claims 1 to 23, wherein the metadata processor is configured to receive metadata of an audio information message, to receive from the first processor a request to modify the metadata of the audio information message, and to modify the metadata of the audio information message to a modified audio information message metadata in accordance with the request from the first processor; and the first processor is also configured to play back the audio information message in accordance with the modified metadata of the audio information message. 25. The content consumption device system of any preceding claim, the system configured to receive at least one Video Stream associated with the scene to be reproduced; and further comprising at least one media video decoder configured to decode at least one video signal from the at least one Video Stream into the representation of the scene to the user. 26. The content consumption device system of any one of claims 1 to 25, wherein the audio information message is associated with a generic audio message. 27. The content consumption device system of any one of claims 1 to 26, wherein the at least one first processor is configured to decide whether the audio information message should be played based on an indication of an accessibility feature or accessibility feature indication metadata associated with objects in the scene. 28. The content consumption device system of any one of claims 1 to 27, the system configured to generate the uncompressed audio information message. 29. A method for receiving at least a first Audio Stream associated with a scene to be played, with at least one audio signal being encoded in the Audio Stream, comprising: deciding, based on at least user motion data or user selection or metadata, whether an audio information message should be played, wherein the audio information message is uncompressed and independent of at least one audio signal; and causing, upon deciding that the audio information message should be played, playing the audio information message, wherein causing it includes decoding, or causing the decoding of, at least one audio signal of the at least one first Audio Stream into the representation of a scene to a user. 30. A non-transitory storage unit, characterized by storing instructions that, when executed by a processor, cause the processor to perform the following method for receiving at least one first Audio Stream associated with a scene to be played, wherein at least one audio signal is encoded in the Audio Stream, including deciding, based on at least user motion data or user selection or metadata, whether an audio information message should be played, wherein the audio information message is uncompressed and independent of the at least one audio signal; and causing, upon deciding that the audio information message should be played, the playing of the audio information message, wherein causing includes decoding, or causing the decoding of, at least one audio signal from the at least one first Audio Stream into the representation of a scene to a user. 31. The method of claim 29, including processing or modifying the metadata of the audio information message such that causing includes reproducing the audio information message in accordance with the metadata of the audio information message.