KR20110002005A - Scalable techniques for providing real-time per-avatar streaming data in virtual reality systems that employ per-avatar rendered environments - Google Patents

Abstract

Extensible techniques for rendering the represented representation using segments of streaming data are disclosed wherein the emissions are potentially recognizable from many recognition points, and the emissions and recognition points have relationships that change in real time. The technique filters the segments by determining with respect to the time slice whether a given emission is recognizable at a given recognition point. If not recognizable, the segments of streaming data representing the emission are not used to render the emission as recognized from a given recognition point. The techniques are used in networked virtual environments to render audio emission at clients in a networked virtual reality system. In the case of audio emissions, one deciding factor as to whether a given emission is recognizable at a given point of recognition is whether or not the psychoacoustic characteristics of the other emissions obscure the given emission.

Description

SCALABLE TECHNIQUES FOR PROVIDING REAL-TIME PER-AVATAR STREAMING DATA IN VIRTUAL REALITY SYSTEMS THAT EMPLOY PER-AVATAR RENDERED ENVIRONMENTS}

<Cross Reference to Related Application>

The subject matter of this patent application relates to the following United States patent provisional applications, which are hereby incorporated by reference in their entirety, and claim their priority.

United States Patent Provisional Application No. 61/021729 to Rafal Boni et al., "Relevance routing system", filed Jan. 17, 2008.

<Statement of research or development funded by federal funds>

None

<Reference to sequence list>

None

<Technical Field>

The techniques disclosed herein relate to virtual reality systems, and more particularly to rendering of streaming data in a multi-avatar virtual environment.

Virtual environment { virtual environments }

The term virtual environment (abbreviated as VE ) refers to an environment created by a computer system in this context that behaves in a way that meets the expectations of the user's real world environment of the computer system. And a computer system to create a virtual environment is referred to as a virtual reality system in the following, production of a virtual environment according to the virtual reality system is referred to as rendering of the virtual environment. The virtual environment may include an avatar , which in this context is an entity belonging to the virtual environment, having a point of perception within the virtual environment. The virtual reality system may render, for the avatar, the virtual environment as recognized from the recognition point of the avatar. The user of the virtual environment system can be associated with a particular avatar in the virtual environment. An overview of the history and evolution of virtual environments can be found in IEEE Computer (October 2007), "Generation 3D: Living in Virtual Worlds."

In many virtual environments, a user associated with an avatar can interact with the virtual environment through the avatar. In addition to recognizing the virtual environment from the recognition point of the avatar, the user may change the recognition point of the avatar in the virtual environment, change the relationship between the avatar and the virtual environment differently, or change the virtual environment itself. In the following, such a virtual environment is referred to as Interactive (interactive) virtual environment. With the advent of high-performance personal computers and high-speed networking, virtual environments—particularly multi-avatar interactive virtual environments where avatars for multiple users are simultaneously interacting with virtual environments—are widely used by engineering laboratories and specialized application areas. Has changed. Examples of such multi-avatar virtual environments include significant graphical and visual content, such as in massively-multiplayer on-line games (MMOGs) such as World of Warcraft ^® and user-defined virtual environments such as Second Life ^®. It includes an environment having a. In such systems, each user of the virtual environment is represented by an avatar of the virtual environment, each avatar having a recognition point in the virtual environment based on the avatar's virtual location and other aspects in the virtual environment. Users of the virtual environment control their avatars and interact within the virtual environment through client computers, such as PCs or workstation computers. The virtual environment is also implemented using server computers. Rendering for the user's avatar is generated on the user's client computer according to the data sent from the server computer. Data is transmitted as data packets over a network between server computers and client computers in the virtual reality system.

Most of these systems present a visual image of the virtual environment to the user's avatar. Some virtual environments display additional information, such as the sound a user's avatar hears in the virtual environment, or output from the virtual tactile sense of the avatar. " Neuentwicklungen auf dem Gebiet der Audio Virtual Audible output to users or primarily audible output, as produced by the LISTEN system developed by the Fraunhofer Institute, as described in " Reality " (Fraunhofer-Institut fuer Medienkommunikation, Germany, July 2003). Virtual environments and systems are also devised.

When the virtual environment is interactive, the appearance and behavior of the avatar for the user is what other avatars in the virtual environment perceive as-representing or listening to-the appearance and behavior of the user. Of course, the avatar does not need to be shown or recognized as resembling any particular entity, and the avatar for the user may appear deliberately different from the user's actual appearance, which is why many users interact with the virtual environment. It is one of the things that makes it more attractive compared to the interaction in the "real world".

Because each avatar in the virtual environment has a separate point of recognition, the virtual reality system must render the virtual environment differently from other avatars in the multi-avatar virtual environment. What the first avatar recognizes (eg, sees, etc.) would be from one recognition point, and what the second avatar would recognize would be different. For example, the avatar "Ivan" can "see" the avatars "Sue" and "David" and the virtual table from a specific position and virtual orientation, but not the avatar "Lisa" because the avatar is in a virtual environment. For Ivan's "behind" it is "out of the field of view." Another avatar "Sue" will see avatars Ivan, Sue, Lisa and David and two chairs from a completely different angle. The other avatar "Maurice" may be in a completely different virtual location within the virtual environment at that moment and may not see any of the avatars Ivan, Sue, Lisa or David (also they do not see Maurice), but instead Maurice is Maurice See other avatars near the same virtual location with. In this discussion, other avatars and other rendering it is referred to as the Avatar Star (per-avatar) rendering.

2 illustrates an example of avatar-specific rendering for a specific avatar in an exemplary virtual environment. 2 is a fixed image from rendering-in practice, the virtual environment will render the scene dynamically and in color. The recognition point in this rendering example is that of the avatar that the virtual reality system is rendering in FIG. 2. In this example, a group of avatars for eight users "goed" to a specific place in the virtual environment, which includes two-level single phases at 221 and 223. In this example, users-who could be in very distant real world locations-planned to "meet" (via their avatars) in a virtual environment for a meeting to discuss something, so that their avatars were Express presence.

You can see seven of the eight avatars-in this example all the avatars shown are human figures, and the avatar that the virtual reality system is rendering is not visible because the rendering is created from the avatar's point of recognition. to be. For convenience, the avatar for which rendering is being made for him is referred to as 299 in FIG. 2. The figure includes an independent label 299 that encloses the entire image in braces to indicate that the rendering was made from the point of the avatar indicated by "299".

Four avatars, including avatars labeled 201, 209 and 213, are shown standing on stage 221. The remaining three avatars, including the avatar labeled 205, appear to stand between the two phases.

As can be seen in FIG. 2, the avatar 209 is standing behind the avatar 213. When rendering this scene with respect to the recognition point of avatar 213, either avatar 209 or avatar 299 is out of view of avatar 213, so none of them will be visible.

The example of FIG. 2 is for a virtual reality system in which users can interact via their own avatars, but avatars do not speak. Instead, such a virtual reality system allows users to "talk" their avatar by typing text on the keyboard. The virtual environment renders the text within a "balloon" over the avatar for the user, and optionally a bubble with the name of the user's avatar is rendered in a similar manner. An example for the avatar 201 is shown at 203.

In this particular example virtual reality system, users can use arrow keys on the keyboard to cause their avatar to move or walk from one virtual location to another, or to turn in another direction. There are also keyboard inputs that allow avatars to gesture by moving their hands and arms. Two examples of such gestures are shown: Avatar 205 is gesturing as shown by raising hands and arms (indicated in a circle at 207), and Avatar 209 is a hand indicated in circle at 211. He is gesturing as shown by the position of the arm.

Thus, users can move through their avatars, gesture, and communicate with each other. Users can move to other virtual locations and places through their avatars, meet other users, hold meetings, make friends, and participate in many aspects of "virtual life" within the virtual environment.

Plenty Multi Avatar Rendered  Problems in Implementing the Environment

There are some problems in implementing an environment in which many multi-avatars are rendered. Some of them are listed below.

A purely large number of different individual renderings that the virtual environment must create for many avatars

The need to provide a networked implementation with multiple connections with limitations on delay and available data bandwidth

As shown by the fact that the virtual reality system of FIG. 2 uses speech bubbles to handle speech, live sound presents challenges to current virtual reality systems. One reason that live sound presents a difficulty is that it is referred to hereinafter as an emission, that is an example of the output generated by an entity in the virtual environment and recognizable by the avatars in the virtual environment. One example of such an emission is a speech produced by one avatar in the virtual environment that other avatars in the virtual environment can hear. The characteristic of the emission is that it is represented by streaming data in the virtual reality system. Streaming data in the present context is any data that has a high data rate and changes unpredictably in real time. Because streaming data is constantly changing, it must always be sent in a constant stream. In the context of a virtual environment, there may be multiple sources that emit streaming data at one time. In addition, the virtual location for the emission and the recognition point for the recognizable avatars may change in real time.

Examples of types of emissions in a virtual environment include audible audible emissions, visible visible emissions, haptic haptic emissions, smellable smells, and tasteable tastes. Emissions, and virtual environment specific emissions such as virtual telepathy or force field emissions. Most of the characteristics of the emission intensity (intensity). The type of intensity will of course depend on the type of emission. For example, in emission of speech, the intensity is expressed as loudness. Examples of streaming data are data representing voice (audio data), data representing moving images (video data), and also data representing continuous force or touch. New types of streaming data continue to be developed. Emissions in the virtual environment may come from sources in the real world, such as voices from users associated with avatars, or from sources created or recorded.

The source of the emission in the virtual environment can be any entity in the virtual environment. Taking sound as an example, examples of audible emissions in a virtual environment are sounds made by entities in the virtual environment-e.g., an avatar emits what the user of the avatar speaks into the microphone; Generated wheezing, explosions emitted by virtual bombs, endangered clicks emitted by virtual high heels on the virtual floor, and background sounds-for example, an area of the virtual environment. A background sound of a virtual breeze or wind emitted by or a background sound emitted by a group of virtual ruminants.

The sounds in the sound sequence, the relative location of the emission source and avatar, the quality of the sounds emitted by the sources, the audibility and apparent loudness of the sound relative to the avatars, and potentially each The direction of a potentially-perceiving avatar can actually change all in the real world. The same is true for other types of emission and other types of streaming data.

The problem of rendering emissions as they are individually recognized by each avatar in a virtual environment is complex. These problems may arise when sources and destination avatars move in a virtual environment while sources are emitting, for example, when a user speaks through his avatar and moves the avatar that is emitting it, or when other users It is even worse if you move your avatar while recognizing the emission. The latter aspect, where the recognizing avatar moves in the virtual environment, even affects emissions from a static source in the virtual environment. Not just the ever-changing, only the streaming data representing the Emmy Sean, what will happen is rendered illustration Emmy, and the avatar will be recognized subject to the rendering of which the emission is also constantly changing. The rendering and recognizing avatar changes not only when the potentially recognizing avatars move in the virtual environment, but also when the sources of the emission move in the virtual environment.

At the first level of complexity, whether a potentially recognizing avatar can actually recognize a sound sequence that is emitted by the source at a given moment depends at least on the volume of the sound that is emitted by the source at each moment. do. It also depends on the distance in virtual reality between the source and potentially recognizing avatars at each moment. As in the "real world", sounds that are too soft for a recognition point in the virtual environment will not be heard by the avatar at that recognition point. Sounds coming "away" are heard or perceived as softer than they come from closer distances. In this context, the degree of sound is audible yeorige with distance is referred to as a distance weight factor (a distance-weight factor). Intensity of the sound at the source is referred to as the specific size (intrinsic loudness) of sound. The intensity of the sound at the point of recognition is called the apparent loudness .

At the second level, whether the sound being emitted can be heard by a particular avatar is also determined by other aspects of the location of the particular avatar relative to the source, by sounds that the recognizing avatar is listening to from other sources simultaneously, or It can be determined by the quality of. For example, the principles of psychoacoustics include the fact that louder sounds in the real world can mask or mask less loud sounds (based on the apparent size of an individual listener). This is referred to as the relative loudness or volume of the sound, where the apparent loudness of one sound may be larger relative to the apparent loudness of the other sound. In addition, psychoacoustic effects include that certain qualities of sound tend to be heard better than others: for example, people are particularly aware of or listen to the crying of a baby, This is true even when other louder sounds are present at the same time.

Further, by another complex, to sound to render the sound to be recognized as coming from the appropriate relative direction with respect to all the sound is the avatar for gatgoseo directivity (directionally) is rendered, all avatars for all avatars that can hear the sound It may be desirable. Thus, directionality depends not only on the virtual location of the avatar where the sound can be heard, but also on the location of all sources of potential audible sound in the virtual environment, and also on the direction in which the avatar is "facing" in the virtual environment.

Prior art virtual reality systems that can perform rendering of emissions to and from a few small sources and avatars as they are can be done with tens of thousands of sources and avatars in an environment where a large number of multi-avatars are rendered. You simply won't cope. Generally speaking, such a system is not scalable to handle a very large number of sources and avatars.

In summary, avatar-specific rendering of emissions from multiple sources, such as audible emissions from multiple sources within a virtual environment, results in streaming data representing emissions from each source.

● almost constantly being emitted and changing,

● have a high data rate accordingly,

• must be rendered from multiple separate sources at once,

● must be rendered individually once for each avatar listening;

● complex or expensive to render,

● Difficult to handle when there are multiple sources and avatars

Points out special problems.

Multi Avatar Rendered  Current technologies for handling streaming data in the environment

Current techniques for rendering streaming data in a virtual environment provide limited success in addressing the issues mentioned. As a result, implementers of a multi-avatar virtual environment are forced to make one or more unsatisfactory compromises.

• No support for emissions that must be represented using streaming data, such as audible or visible emissions.

Because providing audio interactions is overly difficult or expensive, the virtual environment can only support "text chat" or "instant messages" in a broadcast or point-to-point manner, with no audio interactions between users through avatars. Do not.

● Limit the size and complexity of the rendered environment:

The virtual environment implementation may allow only a small maximum number of avatars for the virtual environment, or split the avatars so that only a small maximum number can exist at any given time in a given "scene" within the virtual environment, or only a limited number of users at a time. Emissions from this streaming data can be used to allow interaction.

● No avatar-specific rendering of streaming data:

Avatars are limited to talking and listening only on an open "party line", all sounds from all scenes or "scenes" in the virtual environment are always present, and all avatars are provided with the same rendering of all sounds. .

● unrealistic rendering:

Avatars may only audibly interact when the users of the avatars participate in an optional “chat session” such as, for example, a virtual intercom, and the user's voice of the avatars is independent of the virtual location of the avatars in the environment. Rendered without direction at the original volume.

● limited implementation of environmental media:

Due to the difficulty in supporting streaming data, environmental media, such as background sounds for waterfalls, are not meant for virtual environments, but rather in client components for each user, such as playing digital recordings of loops. It can only be supported as a locally generated sound.

 Undesirable side effects from the control of streaming media:

In many existing systems that provide support for streaming data, a separate control protocol used in the network is used to manage the flow of streaming data. One side effect is a control event to change the flow of streaming data, in part due to a known problem of transmission delay in the network, eg, to "mute" streaming data from a particular source, or Changing the delivery from being delivered to the first avatar to being delivered to the second avatar may prevent the change from being made until after a noticeable delay. Control and transfer operations are not sufficiently synchronized.

It is an object of the present invention to provide an extensible technique for dealing with emissions in virtual reality systems that generate avatar-specific rendering. Another object of the invention is to filter the emissions using psychoacoustic principles. It is yet another object of the present invention to provide a technique for rendering emissions in devices at the edge of networked systems.

In one aspect, the object of the present invention is achieved by a filter in a system that renders an emission represented by a segment of streaming data. Emissions are rendered by the system as they are recognized at one point in time from the recognition point at which they can potentially be recognized. The filter's features include:

The filter is associated with the recognition point.

● The filter

     Current emission information for the emission represented by the segment of streaming data at that point in time; And

     O Current recognition point information for the recognition point of the filter at that time represented by the segment of streaming data.

Has access to

The filter determines from the current recognition point information and the current emission information whether the emission represented by the streaming data of the segment is recognizable at the recognition point of the filter. If the determination indicates that the emission represented by the streaming data of the segment is not recognizable at the filter's recognition point at that point, the system does not use that segment to render the emission at the filter's recognition point. .

In another aspect, a filter is a component of a virtual reality system that provides a virtual environment, in which sources in the virtual environment emit emissions that are potentially recognized by avatars in the virtual environment. The filter is associated with the avatar and determines whether the emission represented by the segment is recognizable in the virtual environment by the avatar at the avatar's current recognition point. If not recognizable, the segment representing the emission is not used to render the virtual environment for the avatar's recognition point.

Those skilled in the art to which the present invention pertains will recognize other objects and advantages by reading the following drawings and detailed description .

1 is a conceptual overview of a filtering technique.
2 depicts a scene in an exemplary virtual environment. In this scene, users of the virtual environment represented by the avatars are meeting by having their avatars meet at a specific location within the virtual environment.
3 is a conceptual diagram of the contents of a segment of streaming data in the preferred embodiment.
4 shows a specification of a portion of the SIREN14-3D V2 RTP payload format.
5 illustrates the operation of stage 1 and stage 2 filtering.
6 illustrates stage 2 filtering in more detail.
7 shows an adjacency matrix.
Reference numerals in the drawings have three or more numbers, and the two numbers on the right are the reference numbers in the drawings indicated by the remaining numbers. Thus, an item with reference numeral 203 first appears as item 203 in FIG.

DETAILED DESCRIPTION The following detailed description discloses an embodiment in which the virtual environment includes sources of audible emissions and the audible emissions are represented by streaming audio data.

The principles of the techniques described herein can be used with any type of emission.

Overview of the Technology of the Invention

In this preferred embodiment, a virtual reality system, such as the type that can take Second Life as an example, is implemented in a networked computer system. The techniques of the present invention are integrated into a virtual reality system. Streaming data representing acoustic emissions from sources in the virtual environment are communicated in data packets as segments of streaming audio data. Information about the source of the segment associated with determining the recognizability of the segment of the emission for the avatar is associated with each segment. The virtual reality system performs avatar-specific rendering on a rendering component such as a client computer. Rendering for the avatar is done on the client computer, and only segments that the avatar can hear are sent over the network to the client computer. There, the segments are converted to audible output through headphones or speakers for the user of the avatar.

The avatar does not necessarily have to be associated with the user, it can be any entity that the virtual reality system can render. For example, the avatar can be a virtual microphone in a virtual environment. Recordings made using a virtual microphone would be a rendering of a virtual environment made up of audio emissions within the virtual environment that were heard by the virtual microphone.

1 is a conceptual overview of a filtering technique.

As shown at 101, segments of streaming data representing emissions from different sources within the virtual environment are received and filtered. Each segment is associated with information about the source of the emission, such as the location of the source of the emission in the virtual environment and how strong the emission is at the source. In a preferred embodiment, the emission is an audible emission and the intensity is the magnitude of the emission at the source.

These segments are aggregated into a combined stream of all segments by the segment routing component shown at 105. Segment routing component 105 has a segment stream combiner component 103 that combines the segments into an aggregated stream as shown at 107.

As shown at 107, the aggregate stream (consisting of segments of all voice streams) is transmitted to multiple filter components. Two examples of filter components are shown at 111 and 121, with the remainder indicated by ellipses. There is a filter component corresponding to each avatar for which the virtual reality system is creating a rendering. The filter component 111 is a filter component for rendering for the avatar i. Details of filter 111 are shown in 113, 114, 115, and 117, with the other filters operating in a similar manner.

The filter component 111 filters the aggregate stream 107 for the segments needed for rendering the virtual environment appropriately for the avatar i among the streaming data for a given type of emission. The filtering is based on the current avatar information 113 and the current streaming data source information 114 of the avatar i. The current avatar information 113 is any information about the avatar that affects the ability of the avatar i to recognize the emission. Recognition of the current avatar information depends on the nature of the virtual environment. For example, in a virtual environment with the concept of location, the current avatar information may include the location in the virtual environment of the avatar's organ for detecting the emission. In the following, locations in the virtual environment will often be referred to as virtual locations. Of course, if virtual locations exist, there are also virtual distances between those locations.

Current streaming data source information is current information about sources of streaming data that affect the avatar i's ability to recognize emissions from a particular source. One example of current streaming data source information 114 is the virtual location of the emission generation component of the source. Another example is the intensity of the emission at the source.

As shown at 115, only segments having the streaming data necessary for rendering the virtual environment for the avatar i at 119 are recognized by the avatar i are output from the filter 111. In a preferred embodiment, the recognizability may be based on the virtual distance between the source and the recognizing avatar, and / or the relative loudness of the recognizable segments. The segments remaining after filtering by the filter 111 are provided as input to the rendering component 117 which renders the virtual environment with respect to the avatar i's current recognition point in the virtual environment.

Details of the preferred embodiment

In the present preferred embodiment, the emissions of the sources are an audible sound and the virtual reality system is a networked system in which the rendering of the sound for the avatar is done by the client computer used by the user represented by the avatar.

desirable In the embodiment Of segments  survey

As mentioned above, the user's client computer digitizes the streaming acoustic input and transmits segments of the streaming data in packets over the network. Packets for transmitting data over a network are known in the art. In the following, the content (also known as payload) of streaming audio packets in the preferred embodiment is discussed. This discussion describes aspects of the techniques of this invention.

3 illustrates in conceptual form the payload of a streaming audio segment.

In the preferred embodiment, the avatars can recognize audible emissions as well as be their source. Also, the virtual position of the avatar's voice generator may be different from the virtual position of the avatar's voice detector. As a result, the avatar may have a virtual location as a source of voice, which is different from the virtual location as a voice recognizer.

Component 300 illustrates in conceptual form the payload of a streaming data segment used in the preferred embodiment. Curly braces 330 and 340 represent two main portions of the segment payload, namely a header having metadata information about the streaming audio data represented by the segment, and the streaming audio data itself. The metadata includes information such as speaker location and intensity. In a preferred embodiment, the segment's metadata is part of the current streaming data source information 114 for the source of the emission represented by the streaming data.

In a preferred embodiment, the metadata 330 includes the following.

A userID value 301 identifying the entity that is the source of the speech represented by the streaming data in the segment. For a source that is an avatar, this identifies the avatar.

● session ID value to identify the session (302). In this context, a session is a collection of sources and avatars. The set of flags 303 represents additional information, such as information about the state of the source when the emission represents this segment of streaming data. One flag indicates the nature of the position value 305, namely the "speaker" or "listener" position.

The current virtual location of the source of the emission represented by the segment in the virtual environment, or location 305 providing the avatar with the current virtual location of the “listen” portion of the avatar.

A value 307 for the intensity of the acoustic energy, or the intrinsic magnitude of the emitted sound.

If there is additional metadata, it is represented at 309.

In a preferred embodiment, the intensity value 307 for the audible emission is calculated from the inherent loudness of the sound according to principles known in the art. Different types of emissions may use different values to express the strength of the emission. For example, for an emission that appears as text in a virtual environment, the intensity value may be input separately by the user, or an intensity value greater than mixed case or all lower case text may be provided for text that is otherwise all uppercase. have. In an embodiment in accordance with the techniques of the present invention, the intensity values may be selected as a matter of design such that the intensity of different types of emissions can be compared to each other in filtering and the like.

Streaming data segments are shown in 340 and associated braces. Within a segment, the data portion of the segment is shown starting at 321, continuing with all the data in the segment, and ending at 323. In a preferred embodiment, the data in the streaming data portion 340 represents the emitted sound in a compressed format, wherein the client software that creates the segment also converts the audio data into a compressed representation so that less data (and thus Fewer or smaller segments) are sent over the network.

In a preferred embodiment, a compressed format based on the Discrete Cosine Transform (DCT) is used to transform the signal data from the time domain to the frequency domain, and to quantize multiple subbands according to psychoacoustic principles. Such techniques are known in the art and are described in " Polycom ® Siren14 ™, Information for Prospective Licensees ",

 For the SIREN14 codec standard.

Any expression of the emission can be used. Representations can be in other presentation areas, and emissions can also be rendered in other areas, where speech emissions can be represented or rendered as text using a speech-to-text algorithm, or vice versa, acoustic emission This may be visually represented or rendered, or vice versa, virtual telepathy emission may be represented or rendered as different types of streaming data, and so forth.

Architectural Overview of Preferred Embodiments

5 is a system diagram of a preferred embodiment, illustrating the operation of stage 1 and stage 2 filtering. 5 will be described schematically below.

As mentioned in the discussion of FIG. 3, in the preferred embodiment, the segment has a field for session ID 302. Each segment that includes streaming data 320 belongs to a session and has an identifier in field 320 for the session to which the segment belongs. A session identifies a group of avatars and sources, referred to as members of the session. The set of sessions of which the source is a member is included in current source information 114 for that source. Similarly, the set of sessions for which the avatar is a member is included in the current avatar information 113 for that avatar. Techniques for representing and managing members of a group, and also for implementing a system for doing so, are familiar in the art. In a preferred embodiment, the representation of session membership is referred to as a session table.

In a preferred embodiment, there are two types of sessions: positional sessions and static sessions . A location-related session is a session whose members are sources of emission and avatars that are at least potentially detectable within the virtual environment for the emission from those sources. In a preferred embodiment, a given source of audible emission, and any avatar that can potentially hear an audible emission from that given source, must be a member of the same location-related session. The preferred embodiment has only a single location related session. Other embodiments may have more than one location related session. A fixed session is a session whose membership is determined by users of the virtual reality system. Any audible emission made by an avatar belonging to a fixed session is heard by all other avatars belonging to the fixed session, regardless of the position of the avatars in the virtual environment. Thus, a fixed session functions like a conference call. The virtual reality system of the preferred embodiment provides a user interface that allows the user to specify a fixed session to which his avatar belongs. Other embodiments of the filter 111 may include other types of sessions or no sessions at all. One extension to the implementation of sessions in this preferred embodiment would be a set of session ID special values that represent a group of sessions rather than a single session.

In the preferred embodiment, the type of session specified by the session ID of the segment determines how the segment will be filtered by the filter 111. If the session ID specifies a location related session, the segments are filtered to determine whether the avatar for the filter can recognize the source in the virtual environment. The segments that the avatar for the filter can recognize are then filtered by the relative size of the sources. In the latter filter, segments from location-related sessions that can be recognized by the avatar of the filter are filtered together with segments from the fixed session of which the avatar is a member.

In a preferred embodiment, all sources of audible emissions in the virtual environment make segments for the audible emission with the session ID for the location-related session, where the source is also a member of the fixed session and the emission can be heard in that fixed session. If so, the source further makes a copy of each of the segments for the audible emission with the session ID for that fixed session. Thus, an avatar who is able to recognize audible emissions in a virtual environment and is also a member of a fixed session that can hear them can receive more than one copy of the segment in its filter. In a preferred embodiment, the filter detects duplicates and passes only one of the segments to the avatar.

Returning to FIG. 5, components 501 and 509 are two of a number of client computers. Client computers are generally 'personal' computers with hardware and software for integrated system implementations with virtual environments. For example, a client computer has an attached microphone, keyboard, display, and headphones or speakers, and software for performing client operations of the integrated system. Client computers are connected to the network as shown at 502 and 506, respectively. Each client can control the avatar as instructed by the user of that client. The avatar may emit sound in the virtual implementation and / or hear sound emitted by the sources. The streaming data representing the emission in the virtual reality system is generated at the client when the avatar of the client is the source of the emission, and rendered at the client when the avatar of the client can recognize the emission. This is illustrated by bidirectional arrows between client computers and networks, such as between client 501 and network 502, and between client 509 and network 506.

In a preferred embodiment, network connections for segments and streaming data between components such as client 501 and filtering system 517 use standard network protocols such as RTP and SIP network protocols for audio data. The SIP protocol and many other techniques for network connection and connection management are known in the art. An important feature of RTP in this context is that it supports RTP to manage data by its arrival time, and upon request for data that includes a time value, the data having an arrival time that is equal to or less recent than that time value. Can return. The segments that the virtual reality system of the preferred embodiment requests from RTP as described above are referred to as current segments below.

The networks at 502 and 506 are shown as separate networks in FIG. 5, but may of course be the same network or an interconnected network.

Referring to component 501, when a user associated with an avatar in a virtual environment speaks to the microphone at a client computer, such as 501, the software of the computer may segment the stream of streaming data in a compressed format with metadata for that sound. To the filtering system 517 via the network.

In a preferred embodiment, the filtering system 517 is in the server stack of the integrated system, separate from the server stacks of the unrealized virtual reality system.

In the following, compressed format and metadata are described. The filtering system has avatar-specific filters 512 and 516 for the avatars of the clients. Each avatar-specific filter filters streaming data representing audible emissions from multiple sources in the virtual environment. The filtering determines segments of streaming data representing audible emissions that a particular client's avatar can hear and sends streaming audio for the audible segments to the avatar's client over the network. As shown at 503, segments that an avatar representing a user of client 501 can hear are transmitted to client 501 via network 502.

Each source of emissions is associated with current emission source information , ie current information about the emission and its source and / or information about its source in the case where the information can change in real time. Examples include the quality of the emission at the source, the intensity of the emission at the source, and the location of the emission source.

In the present preferred embodiment, the current emission source information 114 is obtained from metadata in the segments representing the emission from the source.

In a preferred embodiment, the filtering is performed in two stages. The filtering process used in the filtering system 517 is generally as follows.

For segments belonging to location sessions:

Stage 1 filtering: For segments and avatars, the filtering process determines a virtual distance that separates the source of the segment from the avatar, and whether the source of the segment is within the threshold virtual distance of the avatar. The threshold distance defines an audible vicinity of the avatar, and emissions from sources outside this proximity are inaudible to the avatar.

Segments outside the threshold are not passed to Filtering 2. This determination is efficiently made by considering metadata information for the segment, such as the session ID described above, current source information for source 114, and current avatar information for avatar 113. In general, such filtering reduces the number of segments that should be filtered as described for Filtering 2 below.

For segments with session ID of fixed session:

Stage 1 filtering: For segments and avatars, the filtering process determines the membership of the session in which the avatar of the filter is identified by the session ID of the segment. If the avatar of the filter is a member of the session, the segment is passed to filtering 2. In general, this filtering reduces the number of segments to be filtered as described for Filtering 2 below.

For all segments within the filter's threshold for the avatar or belonging to a session in which the avatar is a member:

Stage 2 filtering: The filtering process determines the apparent size of all segments for that avatar conveyed by stage 1 filtering. The segments are then sorted according to their apparent size, duplicate segments from different sessions are removed, and a subset consisting of the three segments with the largest apparent size is sent to the avatar for rendering. The size of the subset is a matter of design choice. The decision is made efficiently by considering the metadata. Duplicate segments are segments with the same userID and different session IDs.

Components of the filter system 517 that filter only segments belonging to the location-related session are indicated by upper braces 541, indicated by braces at the top and right at 541, and components that filter only segments belonging to the fixed session are lower. Denoted by braces 542.

Components related to stage 1 filtering are indicated by braces at the bottom left at 551, and components performing stage 2 filtering are indicated by braces at the bottom right at 552.

In a preferred embodiment, filter system component 517 is located at a server in the virtual reality system of the preferred embodiment. However, the filter for the avatar can generally be located at any point in the path between the source of the emission and the rendering component for the avatar with which the filter is associated.

The session manager 504 receives all input packets and provides them to the segment routing 540, which is an avatar suitable for stage 2 filtering segments that a given avatar can recognize, whether via a location related session or a fixed session. Stage 1 filtering is performed by directing to a star filter.

As shown at 505, sets of segments output from segment routing component 540 are input to representative avatar-specific filters 512 and 516 for each avatar. Each avatar capable of recognizing the type of emission represented by the streaming data has a corresponding avatar-specific filter. Each avatar-specific filter selects, from the segments belonging to each source, segments that the target avatar can hear, sorts about their apparent size, removes any duplicate segments, and adds the largest three among the remaining segments. Send the segment to the avatar's client over the network.

Streaming audio Of segments  Details of the content

4 shows a more detailed description of the relevant aspects of the payload format for these techniques. In a preferred embodiment, the payload format may also include non-streaming data used by the virtual reality system. The integrated system of the preferred embodiment is an illustration of some of the many ways in which the techniques can be integrated with a virtual reality system or other application. The format used for this integration is called the SIREN14-3D format. The format uses encapsulation to carry multiple payloads in one network packet. Techniques of encapsulation, headers, flags, and other general aspects of packet and data formats are known in the art and thus are not described in detail herein. For clarity, where details of integration with the virtual environment or the operation of the virtual environment are not closely related in describing the techniques of the present invention, such details are omitted in this discussion.

Component 401 relates to a V2 RTP version, part of this specification being the preferred SIREN14-3D version of this format, wherein one or more encapsulated payloads are carried by a network packet transmitted over a network using an RTP network protocol. State that

In this preferred embodiment, the SIREN14-3D version of the V2 RTP payload consists of an encapsulated media payload with audio data, followed by zero or more other encapsulated payloads. The content of each encapsulated payload is given by the headerFlags flag bit 414 described below.

Component 410 describes the header portion of the encapsulated payload in V2 format. The details of component 410 describe the individual components of metadata within header 410.

As shown at 411, the first value in the header is a 32-bit userID value, which identifies the source of the emission for that segment.

This is followed by a 32-bit entry named session ID 412. This value identifies the session to which the segment belongs.

This is followed by an entry for the intensity value for this segment, named smoothedEnergyEstimate (413). Component 413 is a metadata value for the intensity value for the unique size of the segment of audio data following the header, which is an integer value in units of a particular system implementation.

를 스무딩 처리하고 세그먼트의 에너지에 대한 세기 값을 생성한다. 세그먼트에 대한 세기 값을 계산 또는 할당하는 다른 방법들이 설계 선택에 따라 이용될 수 있다.In a preferred embodiment, the smoothedEnergyEstimate value 413 is a "smooth" value in long-term determined by smoothing together multiple original or "raw" values from the streaming acoustic data. This is otherwise undesirable, which can result from sudden moments of noise ("clicks" or the like) or data artifacts caused by the digitization process of acoustic data for the client computer that may be present in the audio data. Prevent filter results. The value in this preferred embodiment is calculated for the segment using known techniques for calculating the audio energy reflected by the acoustic data of the segment. In a preferred embodiment, instantaneous sample energy using a first-order Infinite Impulse response filter with an 'alpha' value of 0.125.

Smoothing and generate the intensity value for the energy of the segment. Other methods of calculating or assigning intensity values for the segments may be used depending on the design choices.

HeaderFlags 414, consisting of 32 flag bits, follow element 413. A number of these flag bits are used to indicate the type of data and format that follows the header in the payload.

420 shows a portion of a set of flag bit definitions that may be set in headerFlags 414.

Element 428 describes a flag for the AUDIO_ONLY payload with a numeric flag value of 0x1, which indicates that the payload data consists of 80 bytes of audio data in a compressed format for a segment of streaming audio. do.

Element 421 is a flag for the SPEAKER_POSITION payload whose numeric flag value is 0x2 (this flag indicates that the meta data that the payload data constitutes the "mouse" of the source avatar or the current virtual position of the speaking portion). Indicates that it contains data). This may be followed by 80 bytes of audio data in a compressed format for a segment of streaming audio. The location update data consists of three values for the X, Y and Z locations in the coordinates of the virtual environment.

In a preferred embodiment, each source that is an avatar transmits a payload with SPEAKER_POSITION information 2.5 times per second.

Element 422 is a flag for a LISTENER_POSITION payload whose numeric flag value is 0 × 4 (this flag indicates that the payload data includes metadata consisting of the avatar's “ears” or the listening virtual's current virtual location. Will be described). This may be followed by 80 bytes of audio data. The location information allows the filter implementation to determine which sources are in the "audible proximity" of the particular avatar. In a preferred embodiment, each source that is an avatar transmits a payload with LISTENER_POSITION information 2.5 times per second.

Element 423 is a flag for a LISTENER_ORIENTATION payload with a numeric flag value of 0x10, which includes metadata for which the payload data consists of the current virtual orientation or is directed towards the listening portion of the user's avatar. Explain). This information allows, in filter implementations and virtual environments, to extend virtual reality so that the avatar can have a "directional hearing", or a special virtual anatomical structure for hearing, such as rabbits or cat ears. do. Element 424 describes a flag for the SILENCE_FRAME payload whose numeric flag value is 0x20, which indicates that the segment indicates silence.

In the preferred embodiment, if the source does not have audio emission segments to transmit, the source transmits payloads of SILENCE_FRAME payloads as required above to transmit the SPEAKER_POSITION and LISTENER_POSITION payloads along with the location metadata.

Filtering  For action Segment  Additional aspects of the format

In a preferred embodiment, the audio emission from the avatar is not rendered for that same avatar and does not go into any filtering of streaming audio data for that avatar (this is a matter of design choice). This selection is consistent with known practice of not rendering or suppressing "side-tone" audio or video signals in digital telephony and video communications. An alternative embodiment may process and filter the emissions from a source that is also an avatar when determining what can be recognized for that same avatar.

As will be readily appreciated, the filtering techniques described herein may incorporate management functions of a virtual environment to obtain greater efficiency in both filtering of streaming data and managing the virtual environment.

Details of the filter action

The operation of filtering system 517 will now be described in detail.

In a 20 millisecond period, session manager 504 reads the time value from the authoritative master clock. This session manager then obtains from the connections to the incoming segments all segments having a time of arrival equal to or faster than its time value. If more than one segment from a given source is returned, less recent segments from that source are abandoned. The remaining segments are referred to as the current set of segments. Session manager 504 then provides a set of current segments to segment routing component 540, which routes the current segments to specific per-avatar filters. The operation of the segment routing component will be described below. Segments that are not provided to the segment routing component 540 are not filtered and therefore not delivered for rendering to the avatar.

Segment routing component 540 is an adjacency, which is a data table that records which sources are within which avatar's audible proximity (where Avatar's audible proximity is part of a virtual environment within a certain virtual distance of the avatar's hearing). A stage 1 filtering is performed on the segments belonging to the location session using the matrix 535. In a preferred embodiment, this virtual distance is 80 units within the virtual coordinate units of the virtual reality system. Acoustic emissions farther from the avatar's hearing than this virtual distance are inaudible to the avatar.

The adjacency matrix 535 is illustrated in detail in FIG. 7, where the adjacency matrix 535 is a two-dimensional data table. Each cell represents a source / avatar combination and includes a distance-weight value for the source-avatar combination. The distance weight is a factor for adjusting the intrinsic sound intensity or intensity value for the segment according to the virtual distance between the source and the avatar (the distance weighting factor becomes smaller as the virtual distance is larger). In this preferred embodiment, the distance weight is calculated by the clamped formula for roll-off as a linear function of distance. Other formulas can be used instead, including effects such as approximate or clamping, or minimum and maximum intensities, more dramatic or less dramatic roll-off effects, or other effects for more efficient operation. The formula can be chosen. For example, any formula suitable for a particular application may be used depending on the design choice, which is one of the following exemplary references:

The adjacency matrix has one row for each source, shown in FIG. 7 along the left side at 710 with A, B, C, and the like. There is one column for each destination or avatar, as shown at 720 and above as A, B, C and D. In the preferred embodiment, since the avatar is also a source, there is a row B at 730, like column B at 732 for avatar B, but more or fewer sources than avatars, and sources other than avatars and vice versa. There may be a case.

Each cell in the adjacency matrix is at the intersection of rows and columns (source, avatar). For example, row 731 is the row for source D and column 732 is the column for avatar B.

Each cell in the adjacency matrix is a distance weight of zero indicating that the source is not in the audible proximity or inaudible to the avatar, or a distance weight between 0 and 1 (this value is the distance weight calculated according to the formula described above, The factor that must be multiplied by the intensity value to determine the apparent loudness for the emission from that source at the destination. Cell 733 at the intersection of rows and columns holds the value of the weight for (D, B), shown in this example as 0.5.

The weighting factor is calculated using the current virtual position of the source represented by the row of cells and the current virtual position of the "ear" of the avatar represented by the column. In a preferred embodiment, in accordance with the processing for sidetone audio known in the field of digital communications, the cell for each avatar and itself is set to zero and unchanged, and the sound from the source entity is the destination. It is not sent to the entity. This is shown in diagonal set 735 of values, all of which are zero (as with all other cells in this diagonal, the distance weighting factor in cell (source = A, avatar = A) is 0. Values in cells along diagonal line) 735 is shown in bold text for better readability.

In a preferred embodiment, the sources and other avatars transmit segments of streaming data 2.5 times per second with location data for virtual locations. If the segment includes a location, the session manager 504 may retrieve the userID and location values of the segment 114 to update the location information associated with the source or other avatar of the segment in the adjacency matrix 535, as shown at 532. Send to neighbor matrix updater 530.

The neighbor matrix updater 530 periodically updates the distance weighting factors in all cells of the neighbor matrix 521. In a preferred embodiment, this is done at 2.5 cycles per second as follows.

Adjacency matrix updater 530 obtains relevant positional information for each row of adjacency matrix 535 from adjacency matrix 535. After obtaining this location information for the row, the adjacent matrix updater 530 obtains the location information for the hearing unit of the avatar for each column of the adjacent matrix 535. Obtaining location information is shown at 533.

After acquiring positional information about the hearing unit of the avatar, the adjacent matrix updater 530 determines a virtual distance between the source position and the position of the hearing unit of the avatar. If the distance is greater than the threshold distance for the audible proximity, then as shown, the distance weight for the cell corresponding to the row of the source and the column of the avatar in the adjacency matrix 535 is set to zero. If the source and avatar are the same, the value remains unchanged as 0 as described above. Otherwise, the virtual distance between the source X and the destination Y is calculated, and the distance weight (the distance weight for the cell is set to this value) according to the above formula. Updating the distance weights is illustrated at 534.

If the segment routing component 540 determines that the source is outside the avatar's audible proximity, then the segment routing component 540 does not route the segments from the source to the stage 2 filter for the avatar, so these segments are not rendered for the avatar. Will not.

Returning to session manager 504, session manager 504 also segments current segments belonging to static sessions for potential delivery to stage 2 filter components, such as those illustrated at 512 and 516. To the routing component 540.

Segment routing component 540 determines the set of avatars for which a particular segment for an emission should be sent and transmits the segment to one stage 2 filters for those avatars. Segments from a particular source sent to a particular stage 2 during a particular time slice may include segments from different sessions and may include duplicate segments. If the session ID value indicates a fixed session, the segment routing component accesses the session table described below to determine the set of all avatars that are members of the session. This is shown at 525. The segment routing component then sends a segment to each of the stage 2 filters associated with the avatars.

If the session ID value is the value of the location session, the segment routing component accesses the adjacency matrix 535. From the row of the adjacency matrix corresponding to the source of the packet, the segment routing component determines all the columns of the adjacency matrix with nonzero distance weighting factors and the avatars of each such column. This is shown at 536 labeled "adjacent matrix." The segment routing component then sends a segment to each of the stage 2 filters associated with the avatars.

Stage 1 filtering for fixed sessions is performed using segment routing component 540 and session table 521. Session table 521 defines membership in sessions. The session table is a two-column table, where the first column contains the session ID value and the second column contains the entity identifier, such as the identifier for the source or avatar. The entity is a member of all sessions identified by the session ID value in all rows in which the identifier of the entity is in the second column. Members of a session are all entities that appear in the second column of all rows with the session ID of the session in the first column. The session table is updated by the session table updater component 520 in response to fixed session membership by adding rows to or removing rows from the session update table. Numerous techniques are known to those skilled in the art for the implementation of both session table 521 and session table updater 520. If the table 521 indicates that the source and avatar for the segment belong to the same fixed session, the session router 540 routes the segment to the stage 2 filter for the avatar.

6 illustrates the operation of a stage 2 filtering component, such as 512 of the preferred embodiment. Each Stage 2 filtering component is associated with a single avatar. 600 shows a set of current segments 505 that are passed to the stage 2 filtering component. A set of representative segments 611, 612, 613, 614 and 615 are shown. Ellipses indicate that there may be any number of segments.

The beginning of filtering 2 processing is shown at 620. The next set of current segments 505 is obtained as input. Steps 624, 626, 628 and 630 of elements are performed for each segment in the current set of segments obtained in step 620. 624 shows obtaining from each segment an energy value of the segment and a source id of the segment. At 626, for each segment, a session ID value is obtained. If the session ID value is the value of the location session, the next step is 628, as shown. If the session ID value is the value of the fixed session, the next step is 632.

628 shows obtaining a weight from the adjacency matrix 535 from the cell of the adjacency matrix 535 for the avatar that is the source of this segment, and the avatar whose avatar is the stage 2 filter component. This is indicated by dashed arrows at 511.

630 illustrates adjusting the energy value for the segment by multiplying the segment's energy value by the distance weight from the cell. After all segments have been processed by steps 624, 626, 628, and 630, processing continues to step 632.

632 shows sorting all segments obtained in step 622 by the energy value of each segment. After the segments are sorted, all but one of any set of replicas is removed. 634 shows outputting a subset of the segments obtained at 622 as an output of the filtering of Filtering 2. In a preferred embodiment, the subset is three segments with the largest energy values determined by the sorting step 632. The output is shown at 690, which shows representative segments 611, 614, and 615.

Of course, in accordance with the techniques of the present invention, the selection of segments to be output to the avatar may include different sorting and selection criteria than those used in the preferred embodiment. Processing continues from 634 to step 636 before continuing to the starting step at 620 from 636 in the loop. 636 shows that the loop is executed periodically at intervals of 20 milliseconds in the preferred embodiment.

Client Behavior for Rendering

In this preferred embodiment, segments representing audio emissions recognizable for a given avatar are rendered for that avatar according to the avatar's point of recognition. In the case of an avatar for a particular user, rendering is performed on the user's client computer, and the streams of audio data are appropriate apparent volume and stereophonic depending on the source and the virtual distance to the user's avatar and the associated direction. ) Or in the binaural direction. Since the segments sent to the renderer include metadata for the segment, the metadata used for filtering may also be used at the renderer. In addition, energy values of the segments that can be adjusted during filtering 2 can be used in the rendering process. Thus, since there is no need to transcode or modify the encoded audio data originally transmitted by the source, the rendering has no loss of fidelity or intelligibility. The rendering is, of course, greatly simplified by the reduction in the number of segments to be rendered resulting from the filtering.

The rendered sound is output to the user by playing the sound through headphones or speakers of the client computer.

desirable Example  Other aspects

As will be readily appreciated, there are a number of ways to implement or apply the techniques of the present invention, and the examples provided herein are not limiting. For example, filtering may be implemented in separate embodiments, in a parallel manner, or using virtualization of computer resources. In addition, the filtering according to the present techniques may be selected and performed at various points in the system and in various combinations when required to make the best use of the network bandwidth and / or processing power of the virtual reality system.

Additional kinds Filtering , And multiple kinds of Filtering  Combination

Any kind of filtering technique may be used that separates segments representing emissions that a particular avatar can recognize from segments representing emissions that a particular avatar cannot recognize. As indicated in the preferred embodiments described above, multiple types of filtering can be used individually, continuously, or in combination using the techniques of the present invention. In addition, filtering according to the techniques of the present invention may be used in any kind of virtual environment in which the relationship between an emission source and a sensor for an emission may change in real time using any kind of emission. Indeed, a preferred embodiment of the use of relative sized filtering with segments belonging to a fixed segment corresponds to an example of using techniques in a situation where filtering does not depend on location. Techniques used with fixed segments can be used, for example, in conference call applications.

As is apparent, the ease and low cost of the techniques herein can be applied to many kinds of communication and streaming data, which are some of the advantages of the techniques over the prior art.

Type of application

Of course, the techniques of the present invention cover a wide range of applications.

An easy example is

Improvements to audio mixing and rendering of numerous audio inputs for recording, for example to render collective audio for recognition points in a virtual audio space environment, such as a virtual concert hall.

Text messaging communication, for example, where streams of text messaging data from numerous avatars must be displayed or rendered simultaneously in a virtual environment. This is one of many possible examples of streaming virtual data to which the techniques can be applied.

● Filtering and rendering of streaming data for real-time conferencing systems, such as for telephone / audio virtual conferencing environments.

● Filtering and rendering of streaming data for sensory input in a virtual sensory environment.

Distribution of streaming data based on real-time geographic proximity to real entities, which are related to avatars in the virtual environment.

The kind of information needed to filter the emissions from these sources will depend on the nature of the virtual environment, which may depend on the intended application. For example, in a virtual environment for a conferencing system, the relative position of the meeting participants with each other may not be important, and in such an environment, filtering may be associated with the relative inherent size of the audio portion of the meeting attendees and their association with the particular session of the meeting participants. It can be done based only on information such as.

With filtering  Combination and Integration of Different Processing

In addition, filtering can be combined with other processing to produce excellent effects. For example, a stream of media data in a virtual environment may be identified as a "background sound" such as the sound of running water in a virtual fountain in the virtual environment. As part of the integration of these technologies, the designer of a virtual environment may not have the background sound filtered equally to other streaming audio data, but also to prevent other data from being filtered, but instead masked and filtered in a different way. If other streaming data is present, the data for the background sound may be filtered and processed to be rendered at a weaker size. Such an application for filtering techniques allows the background sound to be generated by the server component in the virtual environment system, instead of being generated locally by rendering the component within the client component.

In addition, it is readily apparent that the same filtering according to these techniques can be applied to different kinds of emission and streaming. For example, different users can communicate by different kinds of emission through a virtual environment-a hearing impaired user communicates by visual text messaging in a virtual environment, and another user In this case, the designer can select the same filtering to be applied to the two types of streaming data in an integrated manner. In such implementations, filtering for two different types of emissions according to current avatar information and metadata, such as, for example, source position, intensity, and avatar position, irrespective of two different kinds of emissions. This can be done. All that is required is comparable intensity data.

As mentioned above, the techniques of the present invention can be used to reduce the amount of data to be rendered, thereby increasing the likelihood of moving the rendering of real-time streaming data to the "edge" of the network virtual reality system. Rendering on the destination client, not rendering that burdens the server component. In addition, designers can use these techniques to reduce the amount of data so that functions such as recording previously implemented on the client can be performed on the server component, thereby reducing the client cost for the designer for a particular application, or It is possible to choose to provide virtual functions that are not supported by the client computer or its software.

The flexibility and power of combining filtering with routing and other processing, and the resulting improvement in implementation costs, are among the many advantages of the new techniques disclosed herein.

Summary of some additional aspects of the application of the present technology

In addition to the foregoing, there are of course other useful aspects of the technique. Some additional examples are mentioned here.

In a preferred embodiment, for example, the current emission source information provided by the metadata, relating to position and orientation, may be more useful for rendering the streaming media data stereo or stereophonically at the final point of rendering. The rendered sound is perceived as coming from the appropriate relative direction from the left, from the right, from above, and so on. Thus, the inclusion of such relevant information for filtering may further have synergistic advantages in rendering, in addition to those described above.

In part due to these advantages and novel simplicity over the prior art, systems using the techniques of the present invention can operate very quickly, and designers can also quickly understand and recognize their own techniques. Some of these techniques are particularly helpful in the implementation of special hardware or firmware. As a matter of design choice, these techniques can be integrated with the same infrastructure as a network packet routing system: thus, these new techniques are very efficient, a new kind of components that will be readily and widely available, and will be available in the future. Can be implemented by a new kind of component. In addition, these techniques may be applied to types of emissions that are not yet known and types of virtual environments that are not yet implemented.

conclusion

The foregoing detailed description discloses to those skilled in the art how to use the inventor's adjustable techniques to provide real-time avatar-specific streaming data in a virtual reality system using an avatar-rendered environment, and also the inventors implementing these techniques. It discloses the best mode currently known to.

Those skilled in the art will readily recognize that there are as many applications as possible for these techniques in any area where streaming data is rendered and needs to reduce the processing resources and / or network bandwidth needed to deliver or render streaming data. Could be. The present filtering technique is particularly useful when streaming data represents emissions from sources in the virtual environment and is rendered when required for different sensing points in the virtual environment. Of course, the basis on which filtering is done will depend on the nature of the virtual environment and the nature of the emission. The psychoacoustic filtering techniques disclosed herein are more useful not only in a virtual environment, but in any situation where audio from multiple sources is rendered. Finally, in both cases of filtering and rendering the streaming data in the renderer, the technique of using metadata in the segments containing the streaming data results in a substantial reduction in both network bandwidth requirements and processing resources.

In addition, those skilled in the art will readily appreciate that there are many ways to implement the techniques of the present inventors as long as the implementers exist. Details of the implementations given in the present technology include what the streaming data represents, the type of environment, what is virtual or otherwise, what are the technologies used, and the amount of processing resources and available network bandwidth of the system. And with respect to location will depend on the performance of the system component in which the techniques are used.

For all of the above reasons, the present description is to be considered in all respects only as illustrative and not restrictive, and the scope of the invention disclosed herein is to be accorded the broadest interpretation of the claims as permitted by the patent law and not by the detailed description. It must be decided.

Claims (18)

As a filter in a virtual reality system, the virtual reality system renders a virtual environment that is perceived by an avatar in a virtual environment, wherein the virtual environment is the virtual environment in which the applicability of the virtual environment by the avatar changes in real time. A source of emission in an environment, the emission being represented in the virtual reality system by segments comprising streaming data-,
The filter is associated with the avatar,
The filter,
Current emission source information for the emission represented by the streaming data of the segment; And
Access current avatar information for the avatar of the filter,
The filter determines, from the current avatar information and the current emission source information on the streaming data of the segment, whether the emission represented by the streaming data of the segment is recognized by the avatar, and in the determination And the virtual reality system does not use the segment when rendering the virtual environment if the emission represented by the streaming data of the segment appears to be unrecognizable to the avatar. The method of claim 1,
And determining whether the emission is recognized is based on a physical characteristic of the emission in the virtual environment. The method of claim 1,
And the avatar further recognizes an emission that the avatar cannot recognize in the virtual environment based on a member of the minimum avatar group. The method of claim 2,
The physical property is a distance between the emission and the avatar in the virtual environment rendering an emission not recognized by the avatar. The method of claim 1,
A plurality of emissions exist in the virtual reality that the avatar can recognize;
The determination of whether the emission is recognizable comprises determining whether the emission is psychologically recognizable by the avatar compared to other recognizable emissions. The method of claim 5,
When recognized by the avatar, the plurality of emissions have different intensities;
Whether or not the emission is psychologically recognizable by the avatar is determined by the relative strength of the emission relative to the strengths of other emissions the avatar is recognizable. The method of claim 1,
A plurality of emissions exist in the virtual reality that the avatar can recognize;
The filter,
A first judgment determining whether the emission is recognized in the virtual environment based on a physical characteristic of the emission; And
And a second determination that determines whether the emission is psychologically recognizable by the avatar relative to other recognizable emissions. The method of claim 7, wherein
And said filter makes said second determination only if said emission is determined to be recognizable in said first determination. The method according to any one of claims 1 to 8,
And said emission is an audible emission which is audible in said virtual environment. The method according to any one of claims 1 to 8,
And the emission is a visible emission visible in the virtual environment. The method according to any one of claims 1 to 8,
And the emission is a haptic emission recognized by a touch in the virtual environment. The method according to any one of claims 1 to 8,
The virtual reality system is a distributed system of a plurality of components, the components are mutually accessible by a network, the emission is generated in the plurality of first components and used to render the virtual environment in another component, Segments are transported between the component and other components through the network, and the filter can be located anywhere between the first component and the second component of a distributed system. The method of claim 12,
The components of the distributed system include at least one client and a server, wherein the emission is generated and / or rendered for the server containing the filter and the avatar of the client, the server from the client Receiving the segments representing an emission and selecting segments provided to the client to be rendered for the avatar using the filter. The method according to any one of claims 1 to 8,
And the current emission source information for the emission represented by the streaming data of the segment is also included in the segment. The method according to any one of claims 1 to 8,
Wherein said segments further comprise segments of current avatar information from which said filter obtains current avatar information for the avatar of said filter. A filter in a system that renders an emission represented by a segment of streaming data, wherein the emission is rendered by the system if the emission is recognized at that point in time from a potentially recognized recognition point;
The filter is associated with the recognition point,
The filter,
Current emission information for the emission represented by the streaming data of the segment at this point in time; And
Access current recognition point information for the recognition point of the filter at this point,
The filter determines, from the current recognition point information and the current emission information, whether the emission represented by the streaming data of the segment can be recognized at the recognition point of the filter, and the determination is the If the emission represented by the streaming data of the segment appears to be unrecognizable at the recognition point of the filter, then the system does not use the segment when rendering the emission at the recognition point of the filter. Featured filter. Sound from a plurality of sources, the sound being from a source having a characteristic that changes in real time and from each of a plurality of sources represented as segments in a segment stream generated by the source. As a filter in the system,
The filter receives a time-sliced segment stream from the sources;
The filter selects from the stream the segments belonging to the time slice for rendering in accordance with a psychoacoustic effect resulting from the interaction of a characteristic with respect to the sound represented by the segments belonging to the time slice. Featured filter. A renderer for rendering emissions from a plurality of sources, wherein the emissions change in real time, and the emissions from each of the sources are represented by segments comprising streaming data;
A segment from a source includes information about the emission of the source in addition to the streaming data, and the information about the emission of the source in the segment also includes a subset of segments representing emissions from the plurality of sources. Used to filter the segment to make it available to the renderer;
And the renderer renders the segments belonging to the subset using information about the emission of the source of the segments belonging to the subset.

Images