RU2815621C1 - Audio device and audio processing method - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: invention relates to computer engineering for processing audio data. Technical result is achieved by converting for reproducing audio elements by generating a second set of audio signals for headphones; selection between conversion for reproduction of the first part of the first audio element through the set of acoustic systems and converting for reproducing said first portion of the first audio element through the headphones in response to the first conversion property indicator for audio reproduction, wherein the first conversion property indicator for audio playback indicates whether said first part of the first audio element should be converted for playback through an acoustic system or converted for playback through headphones.

EFFECT: reduced noise level when playing back audio data.

20 cl, 2 dwg

Description

TECHNICAL FIELD

The invention relates to an audio device and method for processing audio and, in particular, but not exclusively, to the use of such device and method to support an augmented/virtual reality application.

BACKGROUND OF THE ART

With the continuous development and introduction of new services and ways of using and consuming audiovisual content, recent years have seen a significant increase in the variety and range of experiences of such content. In particular, a variety of spatial and interactive services, applications and perceptual approaches are being developed to provide users with a more involved and immersive experience.

Examples of such applications include Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), which are quickly becoming mainstream, with a number of solutions targeting the consumer market. In addition, a number of standards are being developed by many standardization bodies. As part of this standardization effort, standards are being actively developed for various aspects of VR/AR/MR systems, including, for example, streaming, broadcasting, playout conversion, etc.

VR applications typically provide user perceptions consistent with the user being in another world/environment/scene, while AR applications (including Mixed Reality (MR)) typically provide user perceptions consistent with the user being in the current environment , but with the addition of additional information or virtual objects or information. Thus, VR applications typically provide a fully immersive synthetically generated world/scene, while AR applications typically provide a partially synthetic world/scene overlaid on a real scene in which the user is physically present. However, these frequency terms are used interchangeably and have a high degree of overlap. In the following, the term virtual reality/VR will be used to refer to both virtual reality and augmented reality.

As an example, a rapidly gaining popularity service is to provide images and audio in such a way that the user is able to actively and dynamically interact with the system to change rendering parameters for playback so that the images and audio adapt to the user's movement and changes in position and orientation. A very attractive feature of many applications is the ability to change the viewer's effective viewing position and viewing direction so that, for example, the viewer can move and "look around" in the presented scene.

Such a feature may, in particular, provide the user with a virtual reality experience. Thanks to this, the user can move around (relatively) freely in the virtual environment and dynamically change his position and the direction in which he is looking. Typically, such virtual reality applications are based on a 3D model of the scene, with the model dynamically evaluated to provide the specific view requested. This approach is well known, for example, from gaming applications for computers and consoles, for example, from the category of “shooters” or first-person shooting games.

It is also desirable, particularly for virtual reality applications, that the image being presented is a three-dimensional image. Indeed, to optimize viewer immersion, it is generally preferable for the user to experience the presented scene as a 3D scene. After all, the experience of virtual reality should preferably allow the user to choose his own position, camera viewpoint and point in time relative to the virtual world.

Typically, virtual reality applications are inherently limited in that they are based on a predefined model of the scene and usually an artificial model of the virtual world. In some applications, a virtual reality experience can be provided by capturing the real world. In many cases, this approach is typically based on constructing a virtual model of the real world from captured real-world data. In this case, the feeling of virtual reality is formed by evaluating this model.

Many current approaches are generally suboptimal and often tend to place high demands on computing and communication resources and/or provide a suboptimal user experience, for example due to reduced quality or limited freedom.

As an example of application, virtual reality glasses have appeared on the market, which allow viewers to perceive captured 360° (panoramic) or 180° video. Such 360° videos are often pre-shot using a multi-camera setup in which individual images are stitched together into a single spherical display. Common stereo formats for 180° or 360° video are top/bottom and left/right. Just as in non-panoramic stereoscopic video, the left-eye and right-eye images are compressed, for example, as part of a single H.264 video stream.

In addition to the visual experience, most VR/AR applications also provide a corresponding audio experience. In many applications, audio preferably provides spatial perception of sound, with audio sources being perceived as emitting sound from positions that correspond to the positions of corresponding objects in the visual scene. Thus, audio and video scenes are preferentially perceived as coherent, with both providing complete spatial perception.

In terms of audio, until now the main focus has been on headphone playback using conversion technology to play binaural audio. In many cases, headphone playback enables a highly immersive, personalized user experience. Using head tracking, the playback transform can be made to respond to the user's head movements, greatly increasing the feeling of immersion.

Recently, both in the market and in standards discussions, use cases have begun to be proposed that include the “social” or “shared” aspect of VR (and AR), i.e. the ability to share experiences with other people. These may be people in different places, but also people in the same place (or a combination of both). For example, multiple people in the same room may have the same general perception with the presence of a projection (audio or video) of each participant in the VR content/scene. For example, in a game involving multiple people, each player may have a different location in the game scene and therefore a different projection of the audio and video scene.

As a specific example, MPEG is attempting to standardize the bitstream and decoder for realistic, immersive AR/VR experiences with six degrees of freedom. Social VR is an important feature and allows users to interact in a shared environment (games, conference calls, online shopping, etc.). The concept of social VR also aims to make virtual reality a more social experience for users physically located in the same place, but where, for example, a head-mounted display or other VR headset provides an isolated experience from the physical environment.

The disadvantage of headphone playback in such "social" or "shared" AR (or VR) use cases is that when each user wears individual headphones, co-located users are at least partially acoustically isolated from each other. which reduces the “social” part of the experience (for example, it becomes difficult or uncomfortable for people standing next to each other to have a natural conversation).

This problem can be solved by using audio acoustic systems for playback instead of headphones. However, this approach has the disadvantage that audio playback cannot be as easily tailored and tailored to the individual user. For example, it is difficult to dynamically adapt audio playback to head movements and, in particular, to changes in head orientation of each individual user. This effect is very important for the feeling of immersion, and therefore speaker systems are generally not optimal for creating an optimized user experience.

Therefore, an improved approach to audio processing would be useful, particularly for virtual/augmented/mixed reality perception/application. In particular, an approach that would improve performance, increase flexibility, reduce complexity, ease implementation, improve audio experience, improve audio-visual scene matching, improve adaptation, improve personalization, improve virtual reality experience, and/or improve performance would be useful. /or work.

DISCLOSURE OF THE INVENTION

Accordingly, the present invention is directed to preferably attenuating, mitigating or eliminating one or more of the above-mentioned disadvantages, individually or in any combination.

In accordance with an aspect of the present invention, there is provided an audio device comprising: a receiver for receiving data describing an audio scene, the data comprising audio data for a set of audio elements corresponding to audio sources in said scene, and metadata comprising at least a first transform property indicator for audio playback for the first audio element from a set of audio elements; a first playback converter for converting for playback of audio elements by generating a first set of audio signals for a set of speaker systems; a second playback converter for converting for playback of audio elements by generating a second set of headphone signals; and a selector configured to select between the first playback transformer and the second playback transformer for transforming to reproduce at least the first portion of the first audio element in response to the first audio playback transform property indicator; wherein the transform property indicator for audio playback indicates whether the first part of the first audio element is associated with a position-dependent listener pose or with a position-independent listener pose.

This approach may provide an enhanced user experience in many embodiments and may in particular provide an enhanced user experience for many virtual reality (including augmented and mixed reality) applications, including, but not limited to, social or general perception. The present approach can achieve improved performance by using hybrid transformation for reproducibility. For example, in many embodiments, it may enable users in the same room to more easily communicate directly with each other while providing targeted and personalized conversion to reproduce an audio scene.

The transform property indicator for audio playback may indicate whether the first audio element represents an audio source with a spatial property that is head-orientation-bound or head-orientation-unbound (corresponding to a listener's pose-dependent position and a listener's pose-independent position, respectively). This approach can reduce complexity and resource requirements.

In some embodiments, the device may include a first driver for driving a set of speakers with a first set of audio signals and a second driver for driving headphones with a second set of audio signals. The first set of audio signals may be, in particular, a set of surround signals, and the second set of audio signals may be, in particular, a binaural stereo signal.

The first audio playback transform property indicator may indicate a playback transform property to be applied to the first audio element, or a property of the first audio element.

In accordance with an optional feature of the present invention, the device also includes a listener posture receiver for receiving a listener posture indicating a listener posture, wherein the first playback transducer is configured to generate a first set of audio signals regardless of the listener posture, and the second playback transducer is configured to generate a second set of audio signals. audio signals in response to the listener's posture.

The audio device may provide a highly favorable and flexible user experience, allowing close consistency between, for example, the user's movement and the perceived audio scene. Pose may refer to position and/or orientation data and may also be referred to as location. The listener pose may be an indication of the listener's position, an indication of the listener's orientation, or a combined indication of the listener's position and orientation. The pose/position may be represented by one or more values providing an indication of position and/or direction.

In accordance with an optional feature of the present invention, the apparatus is configured to generate audio signals for a plurality of listeners, wherein the first playback converter is configured to generate the first set of audio signals as a common set of audio signals for the plurality of listeners; and the second playback converter is configured to generate a second set of audio signals for headphones of a first listener from the plurality of listeners and generate a third set of audio signals for headphones of a second listener from the plurality of listeners.

An audio device can provide effective support for multiple users. Many applications can benefit from improved support for reducing complexity and resource usage while still providing an attractive user experience, often with a consistent and natural feel to the audio space.

A second set of audio signals may be generated in response to a first listening posture for a first listener, and a third set of audio signals may be generated in response to a second listening posture for a second listener. The first set of signals can be generated regardless of the listeners' postures.

According to an optional feature of the present invention, the first part is a frequency subband of the first audio element.

This may provide improved performance in many embodiments.

In accordance with an optional feature of the present invention, the selector is configured to select different playback transformers from the first playback transformer and the second playback transformer for the first part of the first audio element and for the second part of the first audio element.

This may provide an improved user experience in many implementations. The selector may in particular be configured to select different transducers for reproduction for different frequency ranges of the first audio element.

This can provide an effective approach in many applications. The transform property indicator for audio playback may indicate whether the first audio element is diegetic or not.

In accordance with an optional feature of the present invention, the transform property indicator for audio playback indicates the audio format of the first audio element.

This may provide an improved user experience in many implementations. The transform property indicator for audio playback may indicate an audio format from a set of audio formats comprising at least one audio format from the group: an audio object format, a higher order ambiophonic audio format, and an audio channel signal audio format.

In accordance with an optional feature of the present invention, the transform property indicator for audio playback indicates the type of audio source for the first audio element.

This may provide an improved user experience in many implementations. The transform property indicator for audio playback may indicate an audio source type from a set of audio source types comprising at least one audio source type from the group: speech audio, musical audio, foreground audio, background audio, voiceover audio, and audio of an actor reading text from an author.

According to an optional feature of the present invention, the audio playback transform property indicator indicates a playback transform guiding property for the playback transform of the first audio element.

This may provide improved user experience and/or performance in many embodiments.

In accordance with an optional feature of the present invention, the audio playback transform property indicator indicates whether the first portion of the first audio element is intended to be converted for playback through speakers or headphones.

This may provide improved user experience and/or performance in many embodiments.

According to an optional feature of the present invention, the receiver is also configured to receive visual data indicating a virtual scene corresponding to an audio scene, and a transform property indicator for audio playback indicates whether the first audio element represents an audio source corresponding to an audio scene object.

This may provide improved user experience and/or performance in many embodiments.

In some implementations, the transform property indicator for audio playback may indicate whether the first audio element represents an audio scene corresponding to a scene object that is within the viewport defined for the current listener pose.

In accordance with an optional feature of the present invention, the device also includes a user input for receiving user input, and wherein the selector is configured to select between a first transformer for playback and a second transformer for playback for converting to play back at least a first portion of the first audio element in response to the user input. input.

This may provide an improved user experience in many implementations.

In accordance with an optional feature of the present invention, the selector is configured to determine an audio property of the first audio element and select between a first rendering transform and a second rendering transform for transforming at least a first portion of the first audio element to reproduce at least a first portion of the first audio element in response to the audio property.

This may provide improved user experience and/or performance in many embodiments.

In accordance with an aspect of the present invention, there is provided an audio processing method, comprising: receiving data describing an audio scene, the data comprising audio data for a set of audio elements corresponding to audio sources in said scene, and metadata comprising at least a first transform property indicator for audio playback for the first audio element from a set of audio elements; converting to reproduce audio elements by generating a first set of audio signals for a set of speaker systems; conversions to reproduce audio elements by generating a second set of headphone signals; and selecting between transforming for reproducing at least a first portion of a first audio element for the set of speakers and the set of headphones in response to the first transform property indicator for reproducing audio; wherein the transform property indicator for audio playback indicates whether the first part of the first audio element is associated with a position-dependent listener pose or with a position-independent listener pose.

These and other aspects, features and/or advantages of the present invention will become apparent from the embodiments described later in this document and will be explained with reference to the embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will be described only by way of example with reference to the drawings, in which:

in FIG. Figure 1 shows an example of a virtual reality system based on a client-server architecture; And

in FIG. 2 illustrates an example of audio device elements in accordance with some embodiments of the present invention.

IMPLEMENTATION OF THE INVENTION

Virtual reality experiences (including augmented and mixed reality), which allow the user to navigate through a virtual or augmented world, are becoming increasingly popular, and services are being developed to meet this demand. In many such approaches, visual data and audio data can be generated dynamically to reflect the current position of the user (or viewer).

In the art, the terms "location" and "posture" are used as a general term for position and/or direction/orientation. The combination of position and direction/orientation, such as an object, camera, head, or view, can be called pose or orientation. Thus, a location or pose specification may include up to six values/components, or components/DOF, with each value/component/DOF describing a different property of the position/location or orientation/direction of the corresponding object. Of course, in many situations, location or pose can be represented using fewer components, for example, if one or more components are considered fixed or irrelevant (for example, if all objects are considered to be at the same height and have a horizontal orientation, then four components can provide a complete representation of the object's pose). In the following, the term “pose” is used to refer to position and/or orientation, which can be represented by one to six values (corresponding to the maximum possible degrees of freedom).

Many virtual reality applications are based on a pose that has maximum degrees of freedom, i.e., three degrees of freedom each of position and orientation, resulting in a total of six degrees of freedom. Thus, a pose may be represented by a set or vector of six values representing six degrees of freedom and, therefore, the pose vector may provide an indication of a three-dimensional position and/or a three-dimensional direction. However, it is understood that in other embodiments the pose may be represented by fewer values.

A system or object based on providing maximum degrees of freedom for the viewer is usually called having 6 degrees of freedom. Many systems and objects provide only orientation or position and are commonly referred to as having 3 degrees of freedom.

Typically, a virtual reality application produces 3D output in the form of separate images of the view for the left and right eyes. These can then be presented to the user through suitable means, typically such as separate left and right eye displays of the virtual reality headset. In other embodiments, one or more views of the view may, for example, be presented on an autostereoscopic display, or indeed, in some embodiments, only one 2D image may be generated (eg, using a conventional 2D display).

Likewise, an audio representation of the scene can be provided for a given viewer/user/listener position. The audio scene is typically converted for playback to provide spatial awareness in which audio sources are perceived as originating from desired locations. Audio sources may be static in the scene, and changes in the user's pose will result in a change in the relative position of the audio source relative to the user's pose. Accordingly, the spatial perception of the audio source should be changed to reflect the new position relative to the user. The conversion for audio playback can accordingly be adapted depending on the user's posture.

The viewer or user pose input can be determined in different ways in different applications. In many embodiments, the user's physical movement can be tracked directly. For example, a camera taking a picture of a user's area can detect and track the head (or even the eyes (eye tracking)) of the user. In many embodiments, the user may wear a virtual reality headset that can be tracked externally and/or internally. For example, the headset may contain accelerometers and gyroscopes that provide information about the movement and rotation of the headset and therefore the head. In some examples, the virtual reality headset may transmit signals or contain (eg, visual) identifiers that allow an external sensor to determine the position of the virtual reality headset.

In some systems, the viewer's pose may be provided by manual means, for example, by the user manually operating a joystick or similar manual input means. For example, a user can manually move a virtual spectator around a virtual stage by operating a first analog stick with one hand, and manually control the direction in which the virtual spectator is looking by manually moving a second analog stick with the other hand.

Some applications may use a combination of manual and automated approaches to generate the viewer's input pose. For example, the headset can track head orientation, and the viewer's movement/position in the scene can be controlled by the user using a joystick.

In some systems, the virtual reality application may be provided to the viewer locally, for example, by a standalone device that does not use, or even have any access to, any remote virtual reality data or processing. For example, a device such as a game console may include storage for storing scene data, an input for receiving/generating a viewer's pose, and a processor for generating corresponding images from the scene data.

In other systems, the virtual reality application may be implemented and executed remotely from the viewer. For example, a device local to the user may detect/receive motion/pose data transmitted to a remote device that processes the data to generate the viewer's pose. The remote device can then generate suitable view images for the viewer's pose based on the scene data describing the scene. The view images are then transferred to a device local to the viewer where they are presented. For example, a remote device can directly generate a video stream (typically a stereo/3D video stream) that is directly presented by the local device.

Likewise, the remote device can generate an audio scene that reflects the virtual audio environment. In many embodiments, this can be done by generating audio elements that correspond to the relative positions of different radio sources in a virtual audio environment, and converting them for playback so that they are perceived in the corresponding positions.

For example, the remote device may generate audio data representing an audio scene, and may transmit audio components/audio objects/audio signals or other audio elements corresponding to different audio sources in the audio scene, together with position information indicating the position of these sources (which may, for example, change dynamically for moving objects) . The audio elements may include audio elements associated with specific positions, but may also include elements for more distributed or diffuse audio sources. For example, audio elements representing general (non-localized) background sound, ambient sound, diffuse reverberation, etc. may be provided.

In such a case, the local VR device may transform the audio elements for playback accordingly, for example, by applying appropriate binaural processing reflecting the relative position of the audio sources for the audio components.

With respect to audio in a VR service, in some embodiments, a central server may appropriately generate audio data representing an audio scene, and may, in particular, represent that audio scene using a number of audio elements that can be converted for playback by a local client/device.

In FIG. 1 shows an example of a VR system in which a central server 101 communicates with a number of remote clients 103, for example, through a network 105, such as the Internet. The central server 101 may be configured to support a potentially large number of remote clients 103 simultaneously.

This approach can provide improved trade-offs, such as between complexity and resource requirements across devices, communication requirements, etc., in many scenarios. For example, viewer pose and corresponding scene data may be transmitted at longer intervals, with the local device processing viewer pose and received scene data locally to provide low-latency, real-time perception. This can, for example, significantly reduce the required communication bandwidth while providing low-latency perception while allowing scene data to be centrally stored, generated, and maintained. This may, for example, be suitable for applications in which virtual reality experiences are provided across multiple remote devices.

In FIG. Figure 2 shows elements of an audio device that can provide advanced conversion for audio playback in many applications and scenarios. In particular, the audio device may provide improved playback conversion for many VR applications, and the audio device may be particularly configured to perform audio playback processing and conversion for the VR client 103 of FIG. 1.

The audio device shown in FIG. 2 is configured to be converted to reproduce an audio scene by generating a hybrid set of output signals, wherein a first (sub-)set of output signals is generated to be converted for playback by a set of speakers, and a second (sub-)set of output signals is generated to be converted for playback by headphones. The first set of audio signals may be, in particular, a set of surround signals to be converted for playback on a surround sound speaker. The second set of audio signals may be, in particular, a binaural stereo signal to be converted for playback on headphones.

The audio device in FIG. 2 may be part of a hybrid audio playback system for VR/AR, which uses a combination of headphone and speaker playback to provide an audio scene representation.

This approach can work effectively in many implementations. For example, in many scenarios, using a combination of headphone and speaker playback, rather than playback in either one or the other, can provide a highly immersive AR (or VR/MR) experience for each individual user, while at the same time without compromising for the "social" or "general" aspect of perception. For example, this allows the audio converted for playback to be tailored to individual users and the current context for the user. For example, this could allow the positions of audio sources to be precisely adapted to match the user's head movements/rotations. At the same time, it can reduce the complexity required for binaural processing, since significant portions of the audio scene can be converted for playback using less complex audio channel/surround processing. In addition, this could be based, for example, on the use of headphones with low attenuation of external sound, which, for example, facilitates direct interaction between users in the same environment/room.

The following discussion will focus on implementations in which the system renders an audio scene using a combination of a surround sound setup (e.g., a 5.1 or 7.1 system) that is common to all local users and individual (open-back or semi-open-back) headphones for individual users (where “custom headphones” means headphones that convert for playback a signal that has been generated or adapted for the user wearing the headphones).

In particular, the device will be described with reference to an example of using the "social" or "shared" aspect of a VR/AR/MR application with a common perception by many people. They can be in different places, but, more interesting in this example, they can also be in the same place (for example, in the same room). As a specific use case, multiple people are in the same room and have the same general AR experience that is "projected" within their shared real-world environment. For example, a couple sitting together on the couch watching an immersive movie virtually projected onto the wall of their living room. They may be wearing clear glasses to allow them to see each other and their environment, as well as open-back headphones to allow both dedicated personalized conversion for playback and hearing audio in the environment, including that generated by the surround sound setup.

In particular, the device in FIG. 2 includes a receiver 201 that is configured to receive data describing a virtual scene. The data may comprise data providing a visual description of the scene, and may comprise data providing an audio description of the scene. Thus, the received data can provide an audio scene description and a visual scene description.

The receiver 201 is connected to a visual rendering converter 203, which proceeds to convert to render images corresponding to the current viewing pose of the viewer. For example, the data may comprise 3D image spatial data (eg, image and depth or scene model description), and from this, renderer renderer 203 may generate stereo images (an image for the user's left and right eyes) as known to one skilled in the art. Images may be presented to the user, for example, using separate displays for the left and right eyes of the VR headset.

The received data contains audio data describing the scene. Specifically, audio data contains audio data for a set of audio elements corresponding to audio sources in a scene. Some audio elements may represent localized audio sources in a scene that are associated with a specific position in the scene (of course, the position may change dynamically in the case of a moving object). Often, an audio element may represent audio generated by a particular scene object in a virtual scene and therefore may represent an audio source at a position corresponding to the position of the scene object (eg, a person speaking).

Other elements may represent more distributed or diffuse audio sources, such as, for example, ambient or background noise, which may be diffuse. As another example, some audio elements may fully or partially represent non-spatially localized audio components from localized audio sources, such as, for example, diffuse reverberation from a spatially well-defined audio source.

Audio elements may be encoded audio data, such as encoded audio signals. The audio elements can be different types of audio elements, including different types of signals and components, and even in many embodiments, the first receiver 201 can receive audio data that defines different types/formats of audio. For example, audio data may contain audio represented by audio channel signals, individual audio objects, Higher Order Ambisonics (HOA), etc.

The audio may, for example, be represented as encoded audio for a given audio component that needs to be converted for playback. The audio data may also include position data that indicates the source position of the audio component. The position data may, for example, comprise absolute position data defining the position of the audio source in the scene.

The device also includes two converters 205, 207 for playback.

The first playback converter 205 is configured to convert for playback of audio elements through a set of speakers. In particular, the first playback converter 205 may generate a first set of audio signals for a set of speakers, the first set of audio signals being, for example, a set of surround signals for an acoustic surround setup.

Thus, the first playback converter 205 can generate audio signals that are intended to be converted for playback by a particular configuration of speaker system. The first playback converter 205 may generate a signal for each speaker system of the surround configuration and therefore be converted for playback from a specific location corresponding to a speaker position in the configuration.

The first playback converter 205 may be configured to generate audio signals such that the conversion for playback of a given audio element is performed in such a way that the combined effect gives the impression of the audio element being converted for playback from a desired position. Typically, the received data may, for at least some audio elements, contain specific position indications, and the first renderer 205 may render the audio elements so that they are perceived as originating from the specified position. Other audio elements may be, for example, distributed and diffuse, and may be rendered for playback as such.

It will be appreciated that a variety of algorithms and conversion approaches are known to those skilled in the art for reproducing spatial audio using speaker systems and, in particular, in surround sound systems, and that any suitable approach can be used without prejudice to the present invention.

For example, the first playback converter 205 may generate audio signals for five speaker systems in a surround configuration with a center speaker, a left front speaker, a right front speaker, a left surround speaker, and a right surround speaker. The first playback converter 205 may generate an audio signal set containing an audio signal for each speaker system. The signals can then be amplified to generate driving signals for a separate speaker system.

In some embodiments, the audio element converted for speaker playback may be received as, for example, stereo downmix, and the first playback converter 205 may perform upmixing to generate surround signals, which in some cases may be directly converted for playback. This approach may be useful, for example, for audio elements representing diffuse sound that is not directly related to the user's posture. For example, an audio element representing overall diffuse surround audio may be provided as a stereo downmix that is directly upmixed to provide the proper surround audio channels. Each of the signals resulting from the upmix can be combined with the corresponding speaker signals generated from other audio elements to form a set of output signals.

Some audio elements that are converted for playback through an acoustic installation may be provided, for example, in the form of audio objects. Such an audio object may be represented by audio data that describes a particular audio and associated with position data that describes the position of the audio source. Based on position data and speaker positions (whether actual positions or nominal positions for a surround acoustic setup), the first renderer 205 may determine coefficients for a matrix or vector mapping the audio signal to different surround channels.

In some embodiments, the first playback transducer 205 may also be configured to adapt the generated audio signals based on data from the acoustic environment. For example, if the provided data indicates that the current environment is a highly reflective environment (e.g., a bathroom or similar highly reflective acoustic environment), then the first rendering converter 205 may generate and apply a filter having an impulse response corresponding to transfer function for the environment (first reflections, etc.). In some embodiments, the filter may be applied to each of the generated audio signals for individual surround channels, or in some embodiments may be applied to an audio element prior to upmixing the different audio channels.

In some embodiments, the first playback transformer 205 may alternatively or additionally be configured to add reverberation, which may in particular be based on environmental data received with the audio element. For example, the first rendering transducer 205 may employ a synthetic reverb, such as a Jot reverb, with parameters set depending on acoustic environmental data (eg, the duration of the reverberation as indicated by the data). Typically, reverb can be applied to an audio element before any upmixing or mapping to surround channels. The second playback converter 207 is configured to generate a second set of audio signals for headphones. The second set of audio signals may be, in particular, a binaural stereo signal.

In many embodiments, the playback conversion by the second playback transformer 207 is a binaural playback conversion process using suitable binaural transfer functions to provide the desired spatial effect to the user wearing the headphones. For example, the second renderer 207 may be configured to generate an audio component that is to be perceived as coming from a particular position using binaural processing.

As is known, binaural processing is used to provide spatial perception by virtually positioning sound sources using separate signals to the listener's ears. With proper binaural conversion processing for playback, the signals needed at the eardrums for the listener to perceive sound from any desired direction can be calculated, and these signals can be converted for playback so that they provide the desired effect. These signals are then played back to the eardrum using either headphones or a crosstalk reduction technique (suitable for conversion for playback on speakers placed close together). Binaural conversion for playback can be considered an approach to shaping signals to the listener's ears, resulting in the human auditory system giving the false impression that the sound is coming from the desired positions.

Binaural conversion for playback is based on binaural transfer functions, which vary from person to person due to the acoustic properties of the head, ears, and reflective surfaces such as shoulders. For example, binaural filters can be used to create a binaural recording that simulates multiple sources in different locations. This can be implemented by convolving each sound source with a pair of Head Related Impulse Response (HRIR) that correspond to the position of the sound source.

A well-known method for determining binaural transfer functions is binaural recording. It is a method of recording audio that uses a specially designed microphone arrangement and is designed to be played back using headphones. Recording is accomplished either by placing microphones in the subject's ear canal or by using a headform with built-in microphones - a bust containing the pinnae (outer ears). The use of such a headform containing the auricles provides a spatial experience very similar to that of a person listening to the recordings if they were present during the recording.

Suitable binaural filters can be determined by measuring, for example, responses from sound sources at a specific location in 2D or 3D space on microphones placed in or near a person's ears. Based on such measurements, binaural filters can be generated that reflect the acoustic transfer functions to the user's ears. Binaural filters can be used to create a binaural recording that simulates multiple sources in different locations. This can be implemented by convolving each sound source with a pair of measured impulse responses for the desired sound source position. To create the illusion of a sound source moving around the listener, a large number of binaural filters with sufficient spatial resolution, for example 10 degrees, are usually required.

Binaural head transfer functions can be represented, for example, as Head Related Impulse Response (HRIR), or equally as Head Related Transfer Functions (HRTF), or as binaural impulse response responses. Binaural Room Impulse Response (BRIR), or Binaural Room Transfer Function (BRTF). The (estimated or estimated) transfer function from a given position to the listener's ears (or eardrums) may, for example, be specified in the frequency domain, in which case it is usually called HRTF or BRTF, or in the time domain, in which case it is usually called HRIR or BRIR. In some scenarios, binaural head transfer functions are defined to include aspects or properties of the acoustic environment and, in particular, the room being measured, while in other examples only user characteristics are considered. Examples of functions of the first type are BRIR and BRTF.

The second playback transformer 207 contains a store of binaural transfer functions, typically for a large number of different positions, with each binaural transfer function providing information about how the audio signal should be processed/filtered to be perceived as originating from that position. To form an audio scene with multiple audio sources located at appropriate positions in the audio space, binaural processing can be used to individually apply to multiple audio signals/audio sources and combine the result.

For a given audio element that is to be perceived as originating from a given position relative to the user's head, the second renderer 207 may select and retrieve a stored binaural function that most closely matches the desired position (or, in some case, may generate it by interpolation between multiple close binaural transfer functions). It can then apply the selected binaural transfer function to the audio signal of the audio element, thereby generating a left ear audio signal and a right ear audio signal.

The generated stereo output signal in the form of a left ear and a right ear signal is then suitable for conversion for playback on headphones and can be amplified to generate driving signals that are supplied to the user's headset. The user will then perceive the audio element as originating from the desired position.

It will be understood that in some embodiments the audio element may also be processed, for example to add environmental acoustic effects. For example, as described for the first playback transformer 205, the audio element may be processed to add reverberation or, for example, decorrelation/spaciousness. In many implementations, this processing may be performed on the generated binaural signal rather than on the element audio signal itself.

Thus, the second playback converter 207 may be configured to generate audio signals such that a given audio element is converted for playback such that a user wearing headphones perceives the audio element to be perceived from a desired position. Typically, the second rendering transformer 207 may render the audio elements so that they are perceived as originating from a position indicated in the positional data included in the audio data. It is possible that other audio elements may be, for example, distributed and diffuse, and may be converted for playback as such.

The device may suitably be part of a client 103 that receives data containing audio data describing an audio scene from a central server 101. In many embodiments, the central server 101 may provide a number of audio elements in the form of audio objects, audio channels, audio components, HOAs, audio signals, etc. In many situations, some audio elements may correspond to a single audio source having a specific position. Other audio elements may correspond to more diffuse and less well-defined and more distributed audio sources.

It will be understood that a variety of algorithms and conversion approaches are known to those skilled in the art for spatial audio playback using headphones, and in particular for binaural conversion for playback, and that any suitable approach can be used without prejudice to the present invention.

In such a case, the device shown in FIG. 2 may be used in the client 103 to process the received audio data for conversion to reproduce the desired audio scene. In particular, it can process each audio element based on the required position data (when appropriate) and then combine the results.

Accordingly, to generate audio representing a scene, the device in FIG. 2 uses two different conversion methods for playback. Different rendering conversion methods may have different properties, and the apparatus shown in FIG. 2 includes a selector 209 configured to select which audio elements are converted for playback by the first playback transformer 205 and which audio elements are converted by the second playback transformer 207. In particular, for a given first audio element, the selector 211 may select which playback converter 205, 207 should be used for playback converter. The selector 209 may accordingly receive the first audio element and supply it to the first playback transformer 205 or the second playback transformer 207 depending on the selection.

In the system, receiver 201, in addition to audio data (and possibly visual data), is configured to receive metadata containing transform property indicators for audio playback for at least one of the audio elements and often for most or even all of the audio element. In particular, for the first audio element, at least a first transform property indicator for audio playback is included.

The selector 209 is configured to select which playback transformer should be used depending on the received metadata and transform property indicators for audio playback. In particular, the selector 209 is configured to take into account the first transform property indicator for audio playback and decide whether the first audio element should be converted by the first playback transformer 205 or the second playback transformer 207, i.e., whether it should be converted to playback using speakers or headphones.

As a low complexity example, the data for an audio element may include encoded audio data as well as metadata comprising an indication of a position (typically the position of an audio source corresponding to the audio element) and an indication of an audio playback transform property for the audio element, wherein the audio playback transform property indication in a particular example may simply be a binary indication of whether the first audio element is to be converted for playback by the first playback transformer 205 or by the second playback transformer 207 . The selector 209 can then evaluate this binary indication and select the specified transformer 205, 207 for reproduction. The playback converter 205, 207 can then generate the appropriate output signals for the speakers and headphones, respectively, so that the audio elements are perceived as coming from the position indicated by the position indicator. The contributions of each of the audio elements indicated to be converted for playback using the playback transformer 205 may then be combined to form a first set of speaker audio signals, and the contributions of each of the audio elements indicated to be converted for playback using the second playback converter 207 may then be combined to form a second set of headphone audio signals.

Thus, the audio device in FIG. 2 can convert an audio scene for playback through a hybrid audio rendering conversion system comprising both speakers and headphones. In addition, control of the distribution of audio elements across headphones and speaker systems can be controlled/guided remotely. For example, the VR experience service provider may also manage and decide how audio elements should be converted for playback. Since the service provider typically may have additional information about the specific nature of the audio source for each audio element, this may allow it to choose how to transform each audio element for playback based on the additional information and knowledge of what may not be available on the client side. This approach can provide improved conversion for playback in many situations and can provide improved user experience in many scenarios. This approach can, for example, provide an accurate and natural conversion to reproduce an audio scene, while at the same time, for example, allowing people in the same room to talk to each other more naturally.

Thus, in many implementations, the audio rendering transform property indicator may provide guidance to the client and audio device on how to transform received audio data for playback. The audio playback transform property indicator may indicate a playback transform guiding property for the transform for playback of the first audio element. In many embodiments, the playback transform guide property may be a preferred, suggested, or nominal renderer transform property that is recommended for use by the local renderer. Thus, the playback transform guidance property may be control data that can be used by the client to set the playback transform parameter of the playback transform.

In some implementations, the playback transform guiding property may be intended to be a required playback transform property that must be used when transforming to reproduce an audio element, but in other implementations, the playback transform guiding property may be a proposed property that can be used or not used by the client. Thus, in many embodiments, an audio device may choose whether to adapt its playback transform to conform to a playback transform guiding property, or may choose to use a different value. However, this approach provides an approach that allows the device to adapt its operation under the guidance of a remote server/service provider. This can achieve improved performance in many implementations because the remote server/service provider can have additional information. For example, it may also allow centralized manual optimization or analysis to potentially improve the playout transform while allowing the client to maintain freedom and flexibility in the playout transform.

In the specific example mentioned above, the audio playback transform property indicator indicates whether the first audio element is intended to be converted for playback through speakers or whether it is intended to be converted for playback through headphones. The selector 209 may be configured to select, for a first audio element, a first playback transformer 205 for conversion for playback if the first playback transform indicator for the first audio element indicates that the first audio element is to be converted for playback by speakers, and select a second playback transformer 207 for playback. a first audio element if the first playback conversion indicator indicates that the first audio element is to be converted for playback by headphones. The selector 209 may then provide it to the selected converter 205, 207 for playback for conversion for playback.

Thus, in many embodiments, the audio rendering transform property indicator indicates the rendering transform property to be applied to the first audio element, and in particular, the rendering transform indicator for the audio element may indicate whether the audio element is intended to be transformed for playback by speaker systems. or headphones.

In some embodiments, accordingly, metadata in the content stream can explicitly communicate whether an audio element should be converted for playback through speakers or through headphones in the case of a hybrid playback system. This may be an explicit artistic choice made by the content producer, and therefore may provide improved conversion control/guidance for playback.

In the device of FIG. 2, the transformation for audio playback (like the transformation for visual playback) can depend on the viewer's posture. In particular, the device includes a listener pose receiver 211, which is configured to receive a listener pose indicating a listener pose. The pose of the listener may in particular be represented by the pose of the headset, for example determined by tracking a VR headset worn by the user/listener. It will be appreciated that any suitable method for generating, estimating, receiving and providing a listening posture may be used without prejudice to the present invention.

The listener pose receiver 211 is connected to the visual renderer 203 and is used to generate a visual output corresponding to a particular pose. In addition, the listening posture receiver 211 is connected to the second playback converter 207 and is used for conversion for playback of headphone audio elements. Thus, the second playback transformer 207 is configured to generate a second set of audio signals in response to the listener's posture.

The second renderer 207 may in particular perform binaural conversion for playback such that audio elements are converted for playback as originating from appropriate positions relative to the current orientation and position of the listeners. For example, for a first audio element, the second playback transformer 207 may first determine a position in scene space indicated by a position indicator received from the first audio element in the stream data. The relative position of the first audio element relative to the user can then be determined by analyzing the listener's current pose and the corresponding pose in scene space. The second renderer 207 may then extract the HRTFs corresponding to this relative position and filter the first audio signal using the extracted HRTFs to generate a binaural stereo signal for the first audio element. The components can then be added to corresponding components formed from other audio elements to form binaural stereo output signals.

It will be understood that many other approaches are known for generating headphone signals (and in particular binaural signals) corresponding to audio sources at spatial locations, and that such a suitable approach or algorithm can be used by the second converter 207 for playback.

Unlike the second playback transformer 207, the playback transform of the first playback transformer 205 (ie, the playback transform for speakers) is independent of the listening position, and therefore the first playback transformer 205 in the example of FIG. 2 is configured to generate a first set of audio signals regardless of the listener's posture.

The first playback transformer 205 may particularly take into account the position indication for an audio element to be converted for playback by the first playback transformer 205 and map it to a position in the transform space for speaker playback. The first playback transformer 205 may then generate signals for the speakers to provide spatial awareness of the audio element corresponding to the specific position.

It will be understood that many other approaches are known for generating speaker signals (and in particular surround signals) corresponding to audio sources at spatial locations, and that such a suitable approach or algorithm may be used by the first converter 205 for reproduction.

Thus, in this example, the headphone signals are continuously generated to reflect the movement and rotation of the listener's head, thereby providing a continuous and consistent user experience. At the same time, the conversion for speaker playback remains unchanged with respect to the movement and rotation of the listeners' heads, which also further ensures a consistent approach. This approach can provide an approach in which different rendering conversion approaches provide a consistent representation of the audio scene relative to a non-static listener.

The previous examples focus on a situation in which a device produces a representation of an audio scene for a single user. However, in many embodiments, the device may generate a representation of an audio scene for multiple users, particularly, for example, two or more users in the same room.

In such a case, the first playback transformer 205 may be configured to generate a common set of audio signals for multiple users, while the second playback transformer 207 is configured to generate separate headphone signals for each user.

Thus, for audio elements that are selected to be converted for playback by the first playback converter 205, only one set of output signals is generated for all users, for example, only one speaker signal is generated for each speaker system in the configuration, and they typically may do not depend on any user-specific properties. In particular, the first set of audio signals generated for conversion for reproduction by speaker systems is generated without taking into account any listener postures. The same transformation for displaying the audio scene is generated for all users.

However, for audio elements that are converted for playback by the second playback converter 207, a different set of audio signals may be generated for each user. In particular, a binaural stereo signal can be generated for each user. These individual signals may be generated to reflect properties or specific characteristics for an individual listener and may be particularly configured to reflect the listening posture of an individual listener. In this way, binaural signals can be generated that reflect the user's current position and orientation.

Therefore, the device can, in particular, provide very effective support for multi-user scenarios. The required audio processing to support multiple users can be significantly reduced. For example, binaural processing is typically relatively complex and resource-intensive, and the number of signals that must be generated using binaural processing can be significantly reduced, significantly reducing the complexity and computational burden in many implementations.

Thus, in an example in which the device supports two users in the same room, the first playback transformer 205 may be configured to generate a common first set of audio signals to be converted for playback using speakers, and the second playback transformer 207 may be configured to generate a second set of headphone audio signals for a first listener and generate a third set of headphone audio signals for a second listener. A first set of audio signals may be generated regardless of the listening posture of the first and second listener, and a second set of audio signals may be generated in response to the listening posture of the first listener, and a third set of audio signals may be generated in response to the listening posture of the second listener.

The audio rendering transform property indicator provided in the received data stream may represent different data in different implementations.

The transform property indicator for audio playback indicates whether the first part of the first audio element is associated with a listener pose-dependent position or a listener pose-independent position. The transform property indicator for audio playback may specifically indicate whether the first audio element is diegetic or not.

As a specific example, in some embodiments, the selector 209 may be configured to allocate audio elements between the first playback transform 205 and the second playback transform 207 based on whether the audio playback transform property indicator for the first audio element indicates that it is “bound.” "head orientation" or "head orientation untethered" in MPEG terminology."

An audio element specified by the audio playback transform property pointer as "head-bound" is an audio element that must have a fixed location relative to the user's head. Such audio elements may be converted for playback using the second playback transformer 207 and may be converted for playback regardless of the listener's posture. Therefore, the transformation to reproduce such audio elements does not take into account the orientation (orientation changes) of the user's head, in other words, such audio elements are audio elements for which the relative position does not change when the user turns his head (for example, non-spatial audio such as ambient noise or e.g. ,music, is designed to follow the user without changing relative position).

An audio element indicated by the audio playback transform property indicator as "head-free" is an audio element that must have a fixed location in the (virtual or real) environment, and therefore its playback transform dynamically adapts to (changes in) the user's head orientation . In many embodiments, realism can be enhanced when such an audio element is converted for playback as a binaural headphone signal that is adapted based on the listener's current posture. For example, the perception of the position of an audio source being converted to be reproduced by a surround sound speaker may depend on the user's position and orientation, and therefore, transforming to reproduce a specified "head-mounted" audio element by a surround speaker may result in the audio source being perceived as moving when user's head movement.

Thus, in some embodiments, the "head-orientation-neutral" elements may be rendered for playback through the user's headphones, with their positions adapted for each individual user according to the user's tracked head orientation. On the other hand, "head-based" elements may be converted for playback through speakers and do not adapt to users' head movements.

The advantage of this implementation is that the "head orientation" elements, which are now mostly presented through speakers (rather than through headphones), are primarily responsible for the acoustic isolation that is felt when all elements are converted for playback through headphones . This is because "head orientation" sounds (mostly music and atmospheric sounds such as crowds, wind, rain, thunder, etc.) are often continuous and ubiquitous in nature. resulting in a sound "curtain" that isolates the user from their physical surroundings. On the other hand, "head-orientation-neutral elements" are often more localized and dispersed in space and time, and therefore mask the user's physical acoustic environment much less.

In some practical implementations, the user's perception of the "head orientation-locked" sounds that are converted for playback through speakers may differ slightly from how they are typically perceived when played back through headphones. However, this usually does not pose a problem since the "head-orientation-locked" sounds that are converted for playback by speakers are typically non-directional or non-critical in terms of spatial localization.

Which audio elements are “head-orientation-free” and which are “head-orientation-bound” can be communicated explicitly via metadata in the audio content stream.

Typically in the context of AR (and VR) audio playback, the term "diegetic" is also used to describe whether an audio element should be "head-oriented" or not. The term "diegetic" describes elements that must remain in the same virtual position when the user moves their head (this means that the position relative to the user's head must be changed for reproduction). The term "non-diegetic" describes elements for which it is not important or even preferable that their positions do not take into account the movements of the user's head (i.e., they will move with or are "attached" to the user's head).

In some implementations, the audio rendering transform property indicator for an audio element may indicate the audio format of the audio element. The selector 209 may be configured to select whether to use the first playback transformer 205 or the second playback transformer 207 to convert for playback of an audio element based on the audio format of the audio element. The transform property indicator for audio playback may, for example, indicate that the audio element has an audio format from the group: audio object format, higher order ambiophonic audio format, and audio channel signal audio format.

In some embodiments, selector 209 may be configured to distinguish between elements to be converted by headphones or speakers based on the format of the audio elements.

For example, channel-based or higher order ambiophony (HOA) elements that are often used to convey background sounds such as music and atmospheric sounds can be converted for playback through speakers, while object elements that are typically used to convey key audio elements of a scene (often representing audio sources with clearly defined positions) can be converted for playback through headphones for each user individually. This will also allow the user to not only change the orientation of their head, but also interact with individual audio objects (if the content producer intended the objects to be interactive).

This embodiment can be viewed as an alternative or complement to providing transform property indicators for audio playback that directly determine which playback transformer should be used. For example, in situations where an explicit signaling element about whether an audio element is “head-orientation-unbound”/“head-orientation-bound” is not included, selector 209 may evaluate the audio format to determine which renderer 205, 207 should be used for playback. .

Approaches and different transform property indicators for audio playback can be combined, for example, channel elements, HOA elements, or elements that are explicitly reported as "head orientation bound" are converted for playback through speakers, whereas objects and "unbound" to head orientation” elements are converted for playback through headphones.

In some implementations, the transform property indicator for audio playback may indicate the source type for the first audio element. For example, a transform property indicator for audio playback may indicate whether the audio element is an audio source type of a set including, for example, one or more of: speech audio; foreground audio; background audio; voice-over audio; and audio of an actor reading text from the author.

In some implementations, the distribution of audio elements to speakers and headphones may be based on indications in the content stream of the source types for the audio elements, for example, metadata such as "speech" or "music" or "foreground" or "background sounds" . In this example, "speech" sources can be converted for playback through headphones, while "music" and "background" sources can be converted for playback through speakers. A special case might be speech that is labeled as being a "voice-over" or "actor reading from the author" that would be best converted for playback through speakers (since it is not intended to be present at a specific location in space, but rather rather, to be present “everywhere”).

In some embodiments, receiver 201 may, as previously described, also receive visual data indicating a virtual scene corresponding to an audio scene. This data may be provided to the visual renderer 203 for rendering transform using a suitable rendering transform method, such as generating stereo images corresponding to the user's current pose.

In some implementations, an audio rendering transform property indicator for an audio element may indicate whether the first audio element represents an audio source corresponding to a visual object in the scene. A scene visual object may be an object for which the visual data contains a visual representation.

In an example in which visual data provides visual data for a viewport, a transform property indicator for audio playback may indicate whether the audio element is associated with an object within the viewport.

If the transform property indicator for audio playback indicates that the object corresponding to the audio element is visible in the scene, selector 209 may decide to transform it for playback using headphones, and otherwise it may transform it for playback using headphones. In some implementations, the transform property indicator for audio playback may directly indicate whether the object is visible. However, in other implementations, the transform property indicator for audio playback may provide an indirect indication of whether the audio element corresponds to a visible scene object.

For example, a transform property indicator for audio playback may include an indication of a scene object that is represented by the received visual data. The selector 209 may then proceed to judge whether an object associated with the audio element is visible for the current listener pose. If yes, then you can move on to converting it for playback using headphones, and if not, the object can be converted for playback using speakers.

In some embodiments, the distribution of audio elements to speakers and headphones may be based on an indication in the received content stream of whether the audio element is associated with a visual element/objects in the content stream. If the indicator indicates that this is the case, the audio element is converted for playback through headphones. If the indicator indicates that this is not the case, the audio elements are converted for playback through the speakers.

In the previous examples, the selector 209 was configured to select the appropriate converter 205, 207 for playback based only on the received data. However, it is understood that in many embodiments, additional considerations and, in particular, other data may be taken into account.

In many implementations, the device may include a user input function that can receive user input. In such embodiments, the selector 209 may be configured to select between the first renderer 205 and the second renderer 207 based on user input. For example, the user input may be a direct indication of the preferred conversion for playback, such as, for example, an explicit indication that a particular audio element should be converted through headphones rather than through speakers. In other implementations, user input may be more indirect and may, for example, change the selection criterion or bias the selection towards one of the transformers 205, 207 for playback. For example, user input may indicate that it is desirable to convert more of the audio elements for playback by headphones, and selector 209 may change the decision criterion to effect this.

Thus, in some embodiments, the user may be able to directly influence the distribution of elements across the speakers and headphones. One example is to allow users to manually assign individual items to be played through headphones or speakers.

Another example of user control of the distribution is to provide the user with two or more modes from which he can select, for example, a "personal perception" mode and a "general perception" mode. In the case where the user selects the "shared experience" mode, any of the above-described implementations for determining which audio elements should be converted for playback through speakers and headphones, respectively, can be used in combination.

In some implementations, the selector 209 may itself be configured to analyze audio elements and determine based on this which transducer 205, 207 should be used. For example, if a transform property indicator for audio playback is not received for a given audio element, selector 209 may proceed to analyze the audio elements to determine an audio property, such as, for example, the number of audio elements in the scene, the number of channels per audio element, the position of the audio elements, the distances of the audio elements from listeners ( or to each speaker system) or the movement of audio elements. The selector 209 may then proceed to make a decision about which transformer 205, 207 to use based on that audio property or a plurality of them.

In a specific example configuration, hereinafter referred to as configuration X, the selector 209 may select a renderer for each audio element to create the most accurate spatial representation of the audio scene. For example, if an audio element is in a position fairly close to the position of one of the physical speakers, then it can be converted for playback on that particular speaker. Conversely, if an audio element falls into an area not covered by any speaker system, then it can be converted for playback through headphones. The fact that an audio element has the same direction as the speaker (from the listener's point of view) can also be used in the same way for a single listener as for many listeners, provided they are all on the same line with speaker. However, this is generally not practical since the user may change positions over time. In this particular configuration X, the angular accuracy of the (binaural) headphone transformer 207 may be taken into account by the selector 209 to make this decision.

Thus, in some embodiments, selection of the appropriate converter 205, 207 for playback may be further based on analysis of the audio signals. For example, a means for estimating the acoustic property of audio signals may be used to determine properties such as the distance (or speed) of an audio object/audio source (particularly in the case of multi-channel signals) or reverberation time. Audio classifiers such as speech/music classifiers, music genre classifiers, or audio event classifiers may also be used. Also, type-specific classifiers can be used to determine what type of microphones (HOA, lavalier, omnidirectional, XY...) were used to record a given signal. Frequency distribution analysis of the audio signal can also be used to decide which audio system (headphones or speaker) is more suitable for conversion to reproduce the entire audio element.

In the previous example, the selector 209 is configured to select either the first playback transformer 205 or the second playback transformer 207 for audio elements on an element-by-element basis. However, it is clear that this is not necessary or essential. For example, in some embodiments, the selector 209 may be configured to select which playback transformer 205, 207 to use for a group of audio elements.

Additionally, in some embodiments, selector 209 may be configured to separately select between transformers 205, 207 for different portions of a single audio element. For example, for some audio elements, one portion may be converted for playback by the first playback transformer 205, and another portion may be converted for playback by the second playback transformer 207.

It will be understood that the audio element may be divided into different parts in different ways depending on the requirements and preferences of the individual implementation. For example, in some embodiments, an audio element may be received as a combination or group of different parts, and selector 209 may individually select transformer 207 to convert for each part. For example, an audio element may represent a particular audio source with a first component that represents an audio source with a well-defined position (eg, corresponding to direct audio), and a second component that represents a more diffuse and distributed sound (eg, corresponding to reverberant sound). In such a scenario, selector 209 may be configured to reproduce the first component and the second component using speakers.

In other embodiments, selector 209 may be configured to divide an audio element into different parts for conversion for playback. For example, a received audio element may correspond to an audio signal that can be analyzed to separate into different parts, which can then be converted for playback separately.

In particular, in many embodiments, different portions of an audio element may correspond to different frequency ranges. For example, the selector 209 may be configured to select, for a given first portion corresponding to a particular frequency range, which converter 205, 207 to use for playback. The same could be done for a different frequency range, which could result in different playback converters 205, 207 being used for the first and second frequency ranges.

In some implementations, different audio rendering transform property indicators may be provided for different portions of an audio element, and selector 209 may consider a particular audio rendering transform property indicator for a given portion when deciding how to transform it for playback. In other embodiments, a transform property indicator for audio playback may be provided for the audio element as a whole, but using different decision criteria for different parts. For example, for the mid to high frequency range, the choice between headphones and speakers is made depending on the received audio playback transform property indicator for the audio element, while for the very low frequency range, the first playback transformer 205 is used to transform for playback of the signal through the speakers. whatever the transform property indicator for audio playback indicates (this reflects the fact that low frequencies tend to provide much less meaningful spatial cues).

For example, a signal may be divided into a low-frequency portion and a high-frequency portion using high-pass filtering, with the low-frequency portion being sent to speakers and the high-frequency portion being sent to headphones depending on the transform property indicator for audio playback. In some embodiments, advanced audio source division may be used (eg, dividing each time-frequency point between converters for playback).

Using filtering that conserves energy at each time-frequency point can allow a physical hybrid rendering system to reduce possible errors created by filtering.

The described approach may provide a number of advantageous effects, including the previously described enabling a perceptually accurate spatial transformation to reproduce an audio scene while providing/facilitating direct interaction to users in the same location.

This approach can reduce complexity and resource usage in many scenarios by potentially reducing the amount of binaural processing required. Another benefit that is often achievable is a reduction in the energy used by the headphone playback system, for example in terms of amplifier power and/or processing load for the on-chip playback converter, which can be critical in the case of untethered headphones (e.g. battery-powered headphones ).

Another interesting property of a hybrid audio playback system for a VR application is that it tends to provide improved security. Indeed, unlike wearing closed-back headphones, participants are not completely cut off from the potential dangers of the actual environment around them. This can be an important factor in many practical situations.

Another advantage of a hybrid system such as the systems described is that a portion of the audio content is converted for playback through a common acoustic installation, which typically enhances the user's overall experience. This approach typically provides an improved user experience.

It will be understood that in the foregoing description, embodiments of the present invention are set forth for clarity with reference to different functional circuits, blocks, and processors. However, it is understood that any suitable distribution of functionality between different functional circuits, blocks or processors can be used without prejudice to the present invention. For example, functionality shown to be implemented by separate processors or controllers may be implemented by the same processor or controllers. Therefore, references to specific functional blocks or diagrams should be considered only as references to suitable means to provide the functionality described, and not as an indication of a strict logical or physical structure or organization.

The present invention may be implemented in any suitable form, including hardware, software, firmware, or any combination thereof. Optionally, the present invention may be implemented, at least in part, in the form of computer software running on one or more processors and/or digital signal processors. The elements and components of an embodiment of the present invention may be physically, functionally, and logically implemented in any suitable manner. In reality, functionality may be implemented in a single block, in multiple blocks, or as part of other functional blocks. Therefore, the present invention may be implemented in a single block or may be physically or functionally distributed among different blocks, circuits and processors.

Although the present invention has been described in connection with certain embodiments, this should not be construed as limiting it to the particular form set forth herein. Rather, the scope of the present invention is limited only by the appended claims. In addition, although it may appear that a feature is described in connection with specific embodiments, one skilled in the art will understand that various features of the described embodiments can be combined in accordance with the present invention. In the claims, the term “comprising/comprising” does not exclude the presence of other elements or steps.

In addition, although multiple means, elements, circuits, or method steps are listed separately, they may be implemented, for example, by a single circuit, block, or processor. Further, although individual features may be included in different claims, they may be effectively combined, and inclusion in different claims does not mean that the combination of features is infeasible and/or disadvantageous. In addition, the inclusion of a feature in one category of claims does not imply limitation to that category, but rather indicates that the feature may equally be applied to other categories of claims when appropriate. In addition, the order of features in a claim does not imply the specific order in which those features must be processed, and, in particular, the order of individual steps in a method claim does not mean that the steps must be performed in that order. Rather, the steps may be performed in any suitable order. In addition, references in the singular do not exclude the plural. Therefore, references using means of indicating the singular, singular numerals “first”, “second”, etc. do not exclude the multitude. Reference numerals in the claims are provided solely by way of clarifying example and should not be construed as limiting the scope of the claims in any way.

Claims (68)

1. An audio device containing: receiver circuit, wherein the receiver circuit is configured to receive data, which describe the audio scene, and contains audio data and metadata, wherein the audio data is audio data for a set of audio elements corresponding to audio sources in the scene, the set of audio elements includes a first audio element, and the metadata comprises at least a first transform property indicator for audio playback, which refers to the first audio element; a first converter circuit, wherein the first converter circuit is configured to convert to reproduce audio elements by generating a first set of audio signals for a set of speaker systems; a second converter circuit, wherein the second converter circuit is configured to convert to reproduce audio elements by generating a second set of headphone audio signals; And a selector circuit that is configured to select between the first transform circuit and the second transform circuit, so that in response to the first transform property indicator for audio playback, transform is provided to reproduce at least a first portion of the first audio element; wherein the first transform property indicator for audio playback indicates whether the first portion of the first audio element is to be converted for playback through a speaker system or converted for playback through headphones. 2. Audio device according to claim 1, which is configured to generate audio signals for multiple listeners, wherein the first converter circuit is configured to generate the first set of audio signals in the form of a common set of audio signals for the specified plurality of listeners, wherein the second converter circuit is configured to generate a second set of headphone audio signals for a first listener of said plurality of listeners, wherein the second converter circuit is configured to generate a third set of headphone audio signals for a second listener from said plurality of listeners. 3. The audio device according to claim 1, wherein said first part is a frequency subband of the first audio element. 4. Audio device according to claim 1, wherein the selector circuit is configured to select between the first transducer circuit and the second transducer circuit for said first portion of the first audio element, and the selector circuit is configured to select between the first converter circuit and the second converter circuit for the second portion of the first audio element. 5. The audio device of claim 1, wherein the first transform property indicator for audio reproduction indicates an audio format of the first audio element. 6. The audio device of claim 1, wherein the first audio playback transform property indicator indicates a playback transform guide property for the playback transform of the first audio element. 7. The audio device of claim 6, wherein the transform property indicator for audio reproduction explicitly indicates that said first portion of the first audio element is intended to be converted for reproduction through speakers, or explicitly indicates that said first portion of the first audio element is intended for conversion for playback via headphones. 8. Audio device according to claim 1, wherein the receiver circuit is configured to receive visual data, the visual data indicates the virtual scene corresponding to the audio scene, and a first transform property indicator for audio playback indicates whether the first audio element represents an audio source corresponding to a visual object in the scene. 9. The audio device according to claim 1, further comprising a user input circuit, wherein the user input circuit is configured to receive user input, and the selector circuit is configured to select between the first converter circuit and the second converter circuit for conversion to reproduce at least the first audio element in response to user input. 10. Audio device according to claim 1, wherein the first transform property indicator for audio reproduction indicates the audio source type of said first portion of the first audio element, and the selector circuit is configured to select between the first converter circuit and the second converter circuit for converting to reproduce said first portion of the first audio element based on the type of audio source of said first portion of the first audio element. 11. Audio device according to claim 1, wherein the first transform property indicator for audio playback indicates whether said first portion of the first audio element is foreground audio or background audio, the selector circuit is configured to determine that said first portion of the first audio element is to be converted for playback through speakers when it is indicated that said first portion of the first audio element is background audio, and the selector circuit is configured to determine that said first portion of the first audio element should be converted for playback through headphones when it is indicated that said first portion of the first audio element is foreground audio. 12. Audio device according to claim 1, wherein the selector circuit is configured to determine an audio property of the first audio element and selecting between a first rendering transformer and a second rendering transformer for transforming to reproduce at least said first portion of the first audio element in response to the audio property. 13. A method for processing audio, including: receiving data describing the audio scene, wherein the audio data is audio data for a set of audio elements, wherein the set of audio elements corresponds to audio sources in the specified scene and metadata, and the metadata comprises at least a first transform property indicator for audio playback, which refers to the first audio element of the set of audio elements; converting to reproduce audio elements by generating a first set of audio signals for a set of speaker systems; converting to reproduce audio elements by generating a second set of headphone audio signals; And selecting between transforming to reproduce at least a first portion of a first audio element through a set of speakers and transforming to reproduce at least said first portion of a first audio element through headphones in response to a first transform property indicator for audio reproduction, wherein the first audio playback transform property indicator indicates whether said first portion of the first audio element is to be converted for playback through a speaker system or converted for playback through headphones. 14. A computer program stored on a non-transitory medium, which, when executed by a processor, implements the method according to claim 13. 15. The method according to claim 13, additionally including: generating audio signals for multiple listeners; generating a first set of audio signals as a common set of audio signals for said plurality of listeners; generating a second set of headphone audio signals for a first listener of said plurality of listeners; And generating a third set of headphone audio signals for a second listener from said plurality of listeners. 16. The method according to claim 13, wherein said first part is a frequency subband of the first audio element. 17. The method of claim 13, further comprising selecting a different rendering transform for said first portion of the first audio element and for the second portion of the first audio element. 18. The method of claim 13, wherein the first transform property indicator for audio playback explicitly indicates that said first portion of the first audio element is intended to be converted for playback through speakers, or explicitly indicates that said first portion of the first audio element is intended to to convert for playback via headphones. 19. Method according to paragraph 13, whereby the first transform property indicator for audio reproduction indicates the audio source type of said first portion of the first audio element, and the choice between conversion to reproduce at least the first part of the first audio element through a set of speakers and conversion to reproduce at least the first part of the first audio element through headphones is carried out in response to the type of audio source. 20. Method according to paragraph 13, whereby the first transform property indicator for audio playback indicates whether said first portion of the first audio element is foreground audio or background audio, wherein the first audio element is converted for playback through speakers when it is indicated that the first portion of the first audio element is background audio, and said first portion of the first audio element is converted for playback via headphones when said first portion of the first audio element is indicated to be foreground audio.

Images