CN116051788A - Method and device for realizing virtual reality - Google Patents
Method and device for realizing virtual reality Download PDFInfo
- Publication number
- CN116051788A CN116051788A CN202211430089.6A CN202211430089A CN116051788A CN 116051788 A CN116051788 A CN 116051788A CN 202211430089 A CN202211430089 A CN 202211430089A CN 116051788 A CN116051788 A CN 116051788A
- Authority
- CN
- China
- Prior art keywords
- event
- virtual
- sound
- character
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000007613 environmental effect Effects 0.000 claims abstract description 29
- 230000008569 process Effects 0.000 claims abstract description 14
- 230000009471 action Effects 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 12
- 230000006399 behavior Effects 0.000 claims description 8
- 230000003213 activating effect Effects 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 abstract description 10
- 230000003993 interaction Effects 0.000 abstract description 8
- 238000001514 detection method Methods 0.000 description 10
- 238000013528 artificial neural network Methods 0.000 description 7
- 230000014509 gene expression Effects 0.000 description 7
- 230000001960 triggered effect Effects 0.000 description 7
- 241000282326 Felis catus Species 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 239000011521 glass Substances 0.000 description 3
- 238000001228 spectrum Methods 0.000 description 3
- 206010011224 Cough Diseases 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- APTZNLHMIGJTEW-UHFFFAOYSA-N pyraflufen-ethyl Chemical compound C1=C(Cl)C(OCC(=O)OCC)=CC(C=2C(=C(OC(F)F)N(C)N=2)Cl)=C1F APTZNLHMIGJTEW-UHFFFAOYSA-N 0.000 description 2
- 239000000779 smoke Substances 0.000 description 2
- 206010041232 sneezing Diseases 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 230000001747 exhibiting effect Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000010408 sweeping Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
- G06N5/022—Knowledge engineering; Knowledge acquisition
- G06N5/025—Extracting rules from data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
- G06N5/046—Forward inferencing; Production systems
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Evolutionary Computation (AREA)
- Signal Processing (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Processing Or Creating Images (AREA)
Abstract
A method and apparatus for implementing virtual reality are provided. The method for realizing virtual reality comprises the following steps: acquiring sound in a real environment; determining an ambient sound event corresponding to the sound; generating a virtual event for presenting a process and/or result of the ambient sound event; the virtual event is executed in a virtual space. According to the method and the device for realizing virtual reality in the exemplary embodiment of the disclosure, the detected environmental sound event can be fed back to the virtual character and the virtual environment where the virtual character is located in a visual driving mode, so that interaction between the virtual world and the real world is achieved, expressive force of the virtual environment and the virtual character is improved, and interestingness of a virtual reality system and immersive experience of a user are increased.
Description
Technical Field
The disclosure relates to the technical field of internet of things. More particularly, the present disclosure relates to a method and apparatus for implementing virtual reality.
Background
The prior meta-universe related technology focuses on the display and interaction in a virtual space, and the presentation of user information (actions, positions and the like) in a real environment mainly depends on controller interaction and mapping.
In the related art, in one case, things in the meta-universe are controlled by recognizing an issued voice command. In another case, metauniverse is controlled by parsing a sound spectrum, a sound size range so that the sound is digitized.
Disclosure of Invention
According to an exemplary embodiment of the present disclosure, there is provided a method of implementing virtual reality, including: acquiring sound in a real environment; determining an ambient sound event corresponding to the sound; generating a virtual event for presenting a process and/or result of the ambient sound event; the virtual event is executed in a virtual space.
Optionally, the determining an environmental sound event corresponding to the sound includes: identifying and classifying the sound to obtain a classification label of the sound; detecting the sound source of the sound to obtain the sound emitting position; the ambient sound event is determined based on the classification tag of the sound and the emission location of the sound, wherein the ambient sound event includes at least one of an event execution subject, an event action, and an event location.
Optionally, the generating a virtual event for presenting the process and/or result of the ambient sound event may include: and reasoning based on the environment sound event, the state of the virtual space and the rule base to obtain the virtual event, wherein the virtual event comprises at least one of an event execution subject, an event action and an event position.
Optionally, the reasoning based on the environmental sound event, the state of the virtual space and the rule base to obtain the virtual event may include: sequentially matching a fact library comprising the environmental sound event and the state of the virtual space with rules in a rule library through a pattern matcher to obtain an agenda, wherein the agenda sequentially stores the rules meeting the conditions; and activating rules in the agenda through an execution engine to obtain the virtual event.
Optionally, the rule base is a knowledge set expressed in terms of rules, including facts required to perform reasoning, virtual events that need to be generated after the rules are satisfied.
Optionally, the event execution body of the virtual event includes a virtual character, a virtual accessory, a virtual environment, and a virtual special event body.
Optionally, the state of the virtual space includes at least one of a virtual character state, a virtual accessory state, a special event state, and a virtual environment state.
Optionally, the classification tag includes at least one of a character including at least one of a character behavior and a character utterance, a character accessory including at least one of a pet and a character, an environmental sound including at least one of a natural environment, a background, and an electrical appliance, and a special sound including at least one of an emergency sound and a special event sound.
According to an exemplary embodiment of the present disclosure, there is provided an apparatus for implementing virtual reality, including: a sound acquisition unit configured to acquire sound in a real environment; an event determination unit configured to determine an ambient sound event corresponding to the sound; an event generation unit configured to generate a virtual event for presenting a process and/or result of the ambient sound event; and an event execution unit configured to execute the virtual event in a virtual space.
Optionally, the event determination unit is configured to: identifying and classifying the sound to obtain a classification label of the sound; detecting the sound source of the sound to obtain the sound emitting position; the ambient sound event is determined based on the classification tag of the sound and the emission location of the sound, wherein the ambient sound event includes at least one of an event execution subject, an event action, and an event location.
Optionally, the event generating unit is configured to: and reasoning based on the environment sound event, the state of the virtual space and the rule base to obtain the virtual event.
Optionally, the event generating unit is configured to: sequentially matching a fact library comprising the environmental sound event and the state of the virtual space with rules in a rule library through a pattern matcher to obtain an agenda, wherein the agenda sequentially stores the rules meeting the conditions; and activating rules in the agenda through an execution engine to obtain the virtual event.
Optionally, the rule base is a knowledge set expressed in terms of rules, including facts required to perform reasoning, virtual events that need to be generated after the rules are satisfied.
Optionally, the event execution body of the virtual event includes a virtual character, a virtual accessory, a virtual environment, and a virtual special event body.
Optionally, the state of the virtual space includes at least one of a virtual character state, a virtual accessory state, a special event state, and a virtual environment state.
Optionally, the classification tag includes at least one of a character including at least one of a character behavior and a character utterance, a character accessory including at least one of a pet and a character, an environmental sound including at least one of a natural environment, a background, and an electrical appliance, and a special sound including at least one of an emergency sound and a special event sound.
According to an exemplary embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, implements a method of implementing virtual reality according to an exemplary embodiment of the present disclosure.
According to an exemplary embodiment of the present disclosure, there is provided a computing device including: at least one processor; at least one memory storing a computer program that, when executed by the at least one processor, implements a method of implementing virtual reality according to an exemplary embodiment of the disclosure.
According to an exemplary embodiment of the present disclosure, a computer program product is provided, instructions in which are executable by a processor of a computer device to perform a method of implementing virtual reality according to an exemplary embodiment of the present disclosure.
According to the method and the device for realizing virtual reality in the exemplary embodiment of the disclosure, through acquiring the sound in the real environment, determining the environmental sound event corresponding to the sound, generating the virtual event for presenting the process and/or result of the environmental sound event, and executing the virtual event in the virtual space, the detected environmental sound event is fed back to the virtual character and the virtual environment where the virtual character is located in a visualized driving mode, so that the expressive force of the virtual environment and the virtual character is improved, and the interaction between the virtual world and the real world is completed.
Additional aspects and/or advantages of the present general inventive concept will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the general inventive concept.
Drawings
The foregoing and other objects and features of exemplary embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings which illustrate the embodiments by way of example, in which:
FIG. 1 illustrates a flow chart of a method of implementing virtual reality according to an exemplary embodiment of the present disclosure;
FIG. 2 illustrates an example of a deep neural network identifying and classifying sounds according to an exemplary embodiment of the present disclosure;
FIG. 3 illustrates an example of a category label according to an exemplary embodiment of the present disclosure;
fig. 4A and 4B illustrate examples of generating a virtual event and executing a virtual event according to an exemplary embodiment of the present disclosure;
FIG. 5 illustrates an example of reasoning by the reasoning engine, according to an example embodiment of the present disclosure;
fig. 6A illustrates an example of a flow of implementing virtual reality according to an example embodiment of the present disclosure;
fig. 6B illustrates an example of a virtual reality implementing a multi-person chat in accordance with an exemplary embodiment of the present disclosure;
fig. 7-12 illustrate various examples of implementing virtual reality according to an exemplary embodiment of the present disclosure;
fig. 13 shows a block diagram of an apparatus implementing virtual reality according to an exemplary embodiment of the disclosure; and
fig. 14 shows a schematic diagram of a computing device according to an exemplary embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The embodiments will be described below in order to explain the present disclosure by referring to the figures.
The prior meta-universe related technology focuses on the display and interaction in a virtual space, and the presentation of user information (actions, positions and the like) in a real environment mainly depends on controller interaction and mapping.
In the present disclosure, sound events are generated from sound detection (type & location) in the user's real environment, opening up real world and virtual world boundaries. Finally, the virtual characters and virtual environments in the metauniverse can be driven by sound events in the real world, and the metauniverse user experience (interaction and reflection between the real world and the virtual world) is more vivid and visual.
In this disclosure, it relates to capturing and detecting ambient sound. Sound is captured from all connected microphone-enabled devices in the user's surroundings. Acoustic features detect sound events through a deep neural network while using a directional algorithm to calculate the position and direction of the sound source. The detected sound event contains more semantic information than acoustic features (spectrum, volume) and more information can be reflected from the real world into the meta universe.
In this disclosure, these detected ambient sound events are inferred and reflected into the metauniverse, with related events driving the virtual character and virtual environment in a visual manner, while other users in the metauniverse can see that the reflection is caused by real events around one of the users.
Fig. 1 illustrates a flowchart of a method of implementing virtual reality according to an exemplary embodiment of the present disclosure. Fig. 2 illustrates an example of a deep neural network that identifies and classifies sound according to an example embodiment of the present disclosure. Fig. 3 illustrates an example of a category label according to an exemplary embodiment of the present disclosure. Fig. 4A and 4B illustrate examples of generating a virtual event and executing a virtual event according to an exemplary embodiment of the present disclosure. Fig. 5 illustrates an example of reasoning by the reasoning engine, according to an example embodiment of the present disclosure.
Referring to fig. 1, in step S101, sounds in a real environment are acquired.
In exemplary embodiments of the present disclosure, all devices with auditory capabilities in a real-world environment (e.g., a home environment) may be utilized, for example, any device with a microphone (such as a cell phone, television, sweeping robot, refrigerator, wearable device, etc.). Sounds in a real-world environment may be traded/collected by a sound sharing framework.
Specifically, sound may be traded for by the following manner of sound sharing:
(1) Sound sharing (applicable to powerful devices) based on a sound event monitoring model of the device;
(2) Sound sharing based on edge/center device mode (for lightweight devices);
(3) Sound sharing based on cloud monitoring (applicable to lightweight devices).
In step S102, an ambient sound event corresponding to the sound is determined. For example, the ambient sound event may be classified into a user own event, a user surrounding object sound event, a background sound event, a special sound event, and the like.
In an exemplary embodiment of the present disclosure, when determining an ambient sound event corresponding to the sound, the sound may be first identified and classified to obtain a classification tag of the sound, sound source detection is performed on the sound to obtain an emission position of the sound, and then the ambient sound event is determined based on the classification tag of the sound and the emission position of the sound. Here, the ambient sound event includes at least one of an event execution subject, an event action, and an event location. The sound may be identified and classified by extracting acoustic features. For example, the sound may be identified and classified by extracting Mel-frequency cepstral coefficients (Mel-frequency cepstral coefficients, MFCC for short).
In an exemplary embodiment of the present disclosure, the classification tag includes at least one of a character including at least one of a character behavior and a character utterance, a character accessory including at least one of a pet, a character-related, an environmental sound including at least one of a natural environment, a background, an electrical appliance, and a special sound including at least one of an emergency sound, a special event sound.
In identifying and classifying the sounds, the acquired sounds may be detected and classified based on a deep neural network. As shown in fig. 2, the deep neural network may include 2 convolutional layers, 2 pooled layers, and 1 fully connected layer. The sound is firstly converted into a Mel frequency spectrum in the deep neural network, and then sequentially passes through a convolution layer, a pooling layer, another convolution layer, another pooling layer and a full connection layer, and the output result of the deep neural network is a classification label of the sound. The classification tag may be, for example, but not limited to, a voice of a person speaking, a voice of an animal, a bell, noise, or the like. In addition, the class label may further include a sub-label. As shown in fig. 3, the classification tags may include classification tags of "person", "person accessory", "ambient sound", "special sound", and the like. The class label "persona" may include the sub-labels "persona behavior", "persona vocalization". The class label "persona accessory" may include the sub-labels "pet", "persona related". The class label "ambient sound" may include the sub-labels "natural environment", "background", "electrical appliance". The class label "special sound" may include the sub-labels "emergency", "special event". The "character behavior" sub-labels may include "applause", "footsteps", "ringing fingers", and the like.
In the sound source detection of the sound, the sound emission position can be calculated by the delay time between the arrival of the sound at two microphones on the device and the energy attenuation value of the signal.
In step S103, a virtual event is generated for presenting the course and/or results of the ambient sound event. And deducing the event which is required to be additionally triggered according to the influence possibly caused by the environmental sound event on the virtual character or the virtual environment. In one embodiment, the final virtual event is generated by combining the ambient sound event with the inferred additional events that need to be triggered.
In exemplary embodiments of the present disclosure, when generating a virtual event for presenting a process and/or result of the ambient sound event, reasoning may be performed based on the ambient sound event, a state of a virtual space, a rule base, resulting in the virtual event. Here, the virtual event includes at least one of an event execution subject, an event action, and an event location.
In an exemplary embodiment of the present disclosure, when reasoning is performed based on the environmental sound event, the state of the virtual space, and the rule base, and the virtual event is obtained, the fact base including the environmental sound event and the state of the virtual space may be sequentially matched with rules in the rule base through the pattern matcher to obtain an agenda, where rules that have satisfied the conditions are sequentially stored in the agenda, and then the rules in the agenda are activated through the execution engine to obtain the virtual event.
In an exemplary embodiment of the present disclosure, the rule base is a knowledge set expressed in terms of rules, including facts required to perform reasoning, virtual events that need to be generated after the rules are satisfied.
In an exemplary embodiment of the present disclosure, the event execution body of the virtual event includes a virtual character, a virtual accessory, a virtual environment, and a virtual special event body.
In an exemplary embodiment of the present disclosure, the state of the virtual space includes at least one of a virtual character state, a virtual accessory state, a special event state, and a virtual environment state.
By way of example, by constructing a rule base of virtual events, current state information of an environmental sound event, a virtual environment and a virtual character is input into an inference engine, the inference engine is matched with rules according to the environmental sound event, and the rules meeting the conditions are triggered to finally generate a derived virtual event.
As shown in fig. 4A and 4B, the virtual event is derived by reasoning based on facts (e.g., sound event in fig. 4A, fact library in fig. 4B) and rules (e.g., sound event rules in fig. 4A, rule library in fig. 4B). In FIG. 4B, the rule base is a knowledge set expressed in terms of rules, containing facts required to perform reasoning, and events that need to be generated after the rules are satisfied. The fact repository is a collection of facts input from outside, containing data required to perform reasoning. The rule matcher is used to compare rules in the rule base with facts in the fact base, and to decide whether the current rule (e.g., rete matching algorithm) is satisfied. Agenda means that rules that have satisfied the conditions are stored in order. The execution engine means that the rules in the agenda are activated to finally output the result.
As shown in fig. 5, when an ambient sound event "infant crys, distance: 5m "is entered into the inference engine, rule 1 is triggered and a virtual event is generated that" the avatar takes a child holding action, the avatar doll is at the avatar's arm ". When the environmental sound event "clap, distance: when 0.2m "is input into the inference engine, rule 2 is triggered and a virtual event of" virtual character clapping, virtual character exhibiting a happy expression "is generated. When the environmental sound event "glass breaks, distance: when 3m "is input into the inference engine, rule 3 is triggered, and a prompt flag is generated that" trigger object breakage ", the virtual character exhibits a virtual event of a startle expression". When an ambient sound event "thunderstorm, distance: 500m "is entered into the inference engine, rule 4 is triggered and a virtual event is generated that" the virtual environment presents a lightning background, the virtual character presents a panic expression ".
Further, in generating a virtual event for presenting a process and/or result of the ambient sound event, an event rule corresponding to the ambient sound event and at least one execution subject in a virtual space may be first determined, an action of the at least one execution subject in the virtual space corresponding to the event content information may be predicted based on the event rule and a state of the at least one execution subject in the virtual space, and then a virtual event will be generated based on the at least one execution subject and the action.
Further, in determining an event rule corresponding to the ambient sound event and at least one execution subject in a virtual space, the event rule corresponding to the ambient sound event may be first determined, and then the execution subject involved in the event rule may be determined as the at least one execution subject. The at least one execution body includes at least one of a plurality of virtual objects, a plurality of virtual accessories in a virtual space. In determining the event rule corresponding to the ambient sound event, the event rule corresponding to the ambient sound event may be selected from a predetermined relationship list (or virtual event rule base) between the ambient sound event and the event rule.
In step S104, the virtual event is executed in the virtual space. Here, the virtual event may be performed with the virtual character as a relative reference point.
In an exemplary embodiment of the present disclosure, the at least one execution body in the virtual space may be controlled to perform the action when the virtual event is executed in the virtual space. The at least one execution subject may include at least one of a virtual character, a virtual accessory, a virtual environment, and a virtual special event subject in the virtual space.
Fig. 6A illustrates an example of a flow of implementing virtual reality according to an exemplary embodiment of the present disclosure. Fig. 6B illustrates an example of a virtual reality implementing a multi-person chat in accordance with an exemplary embodiment of the present disclosure. As shown in fig. 6A, sound in a real environment is first acquired. Then, after predicting an event in a real environment through a deep neural network model, classifying the event based on an event body to enable the sound event to be structured; and the relative positions of the sound distance and the sound direction are acquired by a sound source positioning method, and the position and the direction information of the sound are added into the sound event for enriching the content of the sound event. And then, the sound event information abstracted in reality is sent to the virtual world, and the virtual world maps the acquired event into an event in the virtual space. And then, based on the virtual event ontology, deducing the derived virtual event through an inference engine. And finally, activating the virtual event through the driving module, and displaying the virtual character as a relative reference point in the virtual world according to the event type, the position and the distance of the sound event. Based on the real environmental sounds, the environmental sounds are identified, and the event body is used for classifying and filtering the events, so that a user can select which environmental sounds can be mapped to the virtual system, the environmental sound events are prevented from being directly exposed in the meta universe, and the privacy of the participants is better protected.
As shown in fig. 6B, sounds in the real environment are first monitored. And then, carrying out sound event recognition and sound source detection on the monitored sound. Thereafter, reasoning is performed by a rule-based reasoning engine. Finally, the virtual characters, virtual environments and the like in the meta-universe are driven to change according to the reasoning result.
Fig. 7-12 illustrate various examples of implementing virtual reality according to an exemplary embodiment of the present disclosure.
In fig. 7, "cat" in a real environment is acquired, and sound event recognition and sound source detection are performed on the "cat". Then, by means of the reasoning engine based on rules, the rule 2 is matched, when the cat call event occurs, the distance is smaller than 5m, the pitch angle is in the range of 90-120 degrees, then the virtual cat in the virtual accessory is in the southeast direction, and the virtual cat is inferred to be in the southeast direction. Finally, in the virtual space, the virtual accessories are driven such that one virtual cat is in the southeast direction.
In fig. 8, "thunderstorm sounds" in a real environment are acquired, and sound event recognition and sound source detection are performed on the "thunderstorm sounds". Then, by reasoning through a rule-based reasoning engine, rule 1 is matched, when a thunderstorm event occurs, the virtual environment appears as a thunderbolt background, the virtual character appears as a panic expression, and the virtual environment appears as a thunderbolt background, and the virtual character appears as a panic expression. Finally, in the virtual space, triggering the lightning background, and enabling the expression of the virtual character to change into frightening.
In fig. 9, the "clapping, ringing, coughing, sneezing" sounds in the real environment are acquired, sound events generated by the metauniverse itself are confirmed to participate through sound event detection and sound source localization, reasoning is performed through a rule-based reasoning engine, and actions and expressions of tasks in the metauniverse are driven in real time through a driving module (e.g., a avatar driving module), so that virtual characters (or virtual dolls) in the virtual space are driven to execute clapping, ringing, coughing, sneezing.
In fig. 10, the sound of the surroundings of the user is captured and detected, the sound of "baby cry, cat and dog" in the real environment is obtained, the event type of the sound and the distance and direction of the sound are determined by the sound event recognition module and the sound source detection module, the reasoning is performed by the rule-based reasoning engine, and finally, the specific event is dynamically displayed at the appropriate position around the meta-space virtual character by the virtual prop driving module, so that the virtual accessories in the virtual space are driven to respectively execute baby cry, cat and dog.
In fig. 11, background sounds in a real environment are captured, background sounds of music, thunder and wind are obtained, sound events and positions of occurrence are obtained through a sound event and sound source detection module, reasoning is performed through a rule-based reasoning engine, and the background events are dynamically displayed in meta-universe in real time through a virtual environment driving module, namely, pictures of playing music, thunder and wind are driven to be executed by a virtual environment in a virtual space, so that the background events are dynamically displayed in real time.
In fig. 12, a special sound of 'glass breakage and smoke alarm' in a real environment is acquired, the type and the occurrence position of the sound event are acquired through a sound event and sound source positioning module, reasoning is carried out through a rule-based reasoning engine, and a virtual environment in a virtual space is driven to present a picture of glass breakage and smoke alarm, so that the special event is reminded to a user in a visual mode.
According to the method for realizing virtual reality in the exemplary embodiment of the disclosure, based on the recognition of the real environment sound event by the depth model and the sound source detection, the real environment sound event is fed back to the virtual character and the virtual environment in which the virtual character is positioned in a visual driving mode, the expressive force of the virtual character and the virtual environment is improved, and the interestingness of a virtual reality system and the immersive experience of a user are increased. For example, the event of the user is fed back to the avatar in a visual driving manner through the avatar driving module. And feeding back the sound event of the object around the user to the virtual scene where the virtual character is located in a visual driving mode through the virtual prop driving module. And feeding back the background sound event to the virtual scene where the virtual character is located in a visual driving mode through the virtual environment driving module. And feeding back the special sound event to the virtual scene where the virtual character is located in a visual driving mode through the special event driving module.
Furthermore, according to an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a computer program, which when executed, implements a method of implementing virtual reality according to an exemplary embodiment of the present disclosure.
In an exemplary embodiment of the present disclosure, the computer-readable storage medium may carry one or more programs, which when executed, may implement the steps of: acquiring sound in a real environment; determining an ambient sound event corresponding to the sound; generating a virtual event for presenting a process and/or result of the ambient sound event; the virtual event is executed in a virtual space.
The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing. The computer readable storage medium may be embodied in any device; or may exist alone without being assembled into the device.
Furthermore, according to an exemplary embodiment of the present disclosure, a computer program product is provided, in which instructions are executable by a processor of a computer device to perform a method of implementing virtual reality according to an exemplary embodiment of the present disclosure.
A method of implementing virtual reality according to an exemplary embodiment of the present disclosure has been described above in connection with fig. 1 to 12. Hereinafter, an apparatus for implementing virtual reality and units thereof according to an exemplary embodiment of the present disclosure will be described with reference to fig. 13.
Fig. 13 shows a block diagram of an apparatus implementing virtual reality according to an exemplary embodiment of the disclosure.
Referring to fig. 13, the apparatus implementing virtual reality includes a sound acquisition unit 131, an event determination unit 132, an event generation unit 133, and an event execution unit 134.
The sound acquisition unit 131 is configured to acquire sound in a real environment.
The event determination unit 132 is configured to determine an ambient sound event corresponding to the sound.
In an exemplary embodiment of the present disclosure, the event determination unit 132 is configured to: identifying and classifying the sound to obtain a classification label of the sound; detecting the sound source of the sound to obtain the sound emitting position; the ambient sound event is determined based on the classification tag of the sound and the emission location of the sound. Here, the ambient sound event includes at least one of an event execution subject, an event action, and an event location.
In an exemplary embodiment of the present disclosure, the classification tag includes at least one of a character including at least one of a character behavior and a character utterance, a character accessory including at least one of a pet, a character-related, an environmental sound including at least one of a natural environment, a background, an electrical appliance, and a special sound including at least one of an emergency sound, a special event sound.
The event generation unit 133 is configured to generate virtual events for presenting the process and/or results of the ambient sound event.
In an exemplary embodiment of the present disclosure, the event generation unit 133 is configured to: and reasoning based on the environment sound event, the state of the virtual space and the rule base to obtain the virtual event.
In an exemplary embodiment of the present disclosure, the event generation unit 133 is configured to: sequentially matching a fact library comprising the environmental sound event and the state of the virtual space with rules in a rule library through a pattern matcher to obtain an agenda, wherein the agenda sequentially stores the rules meeting the conditions; and activating rules in the agenda through an execution engine to obtain the virtual event.
In an exemplary embodiment of the present disclosure, the rule base is a knowledge set expressed in terms of rules, including facts required to perform reasoning, virtual events that need to be generated after the rules are satisfied.
In an exemplary embodiment of the present disclosure, the event execution body of the virtual event includes a virtual character, a virtual accessory, a virtual environment, and a virtual special event body.
In an exemplary embodiment of the present disclosure, the state of the virtual space includes at least one of a virtual character state, a virtual accessory state, a special event state, and a virtual environment state.
The event execution unit 134 is configured to execute the virtual event in a virtual space.
In an exemplary embodiment of the present disclosure, the event execution unit 134 may be configured to control the at least one execution body in the virtual space to perform the action. In an exemplary embodiment of the present disclosure, the at least one execution subject may include at least one of a virtual character, a virtual accessory, a virtual environment, and a virtual special event subject in the virtual space.
An apparatus for implementing virtual reality according to an exemplary embodiment of the present disclosure has been described above in connection with fig. 13. Next, a computing device according to an exemplary embodiment of the present disclosure is described in connection with fig. 14.
Fig. 14 shows a schematic diagram of a computing device according to an exemplary embodiment of the present disclosure.
Referring to fig. 14, a computing device 14 according to an exemplary embodiment of the present disclosure includes a memory 141 and a processor 142, the memory 141 having stored thereon a computer program which, when executed by the processor 142, implements a method of implementing virtual reality according to an exemplary embodiment of the present disclosure.
In an exemplary embodiment of the present disclosure, the computer program may, when executed by the processor 142, implement the steps of: acquiring sound in a real environment; determining an ambient sound event corresponding to the sound; generating a virtual event for presenting a process and/or result of the ambient sound event; the virtual event is executed in a virtual space.
Computing devices in embodiments of the present disclosure may include, but are not limited to, devices such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, and the like. The computing device illustrated in fig. 14 is merely an example and should not be taken as limiting the functionality and scope of use of embodiments of the present disclosure.
Methods and apparatuses for implementing virtual reality according to exemplary embodiments of the present disclosure have been described above with reference to fig. 1 through 14. However, it should be understood that: the virtual reality implementing apparatus shown in fig. 13 and its units may be configured as software, hardware, firmware, or any combination of the above to perform specific functions, respectively, the computing apparatus shown in fig. 14 is not limited to include the above-shown components, but some components may be added or deleted as needed, and the above components may also be combined.
According to the method and the device for realizing virtual reality in the exemplary embodiment of the disclosure, through acquiring the sound in the real environment, determining the environmental sound event corresponding to the sound, generating the virtual event for presenting the process and/or result of the environmental sound event, and executing the virtual event in the virtual space, the detected environmental sound event is fed back to the virtual character and the virtual environment where the virtual character is located in a visual driving mode, so that the interaction between the virtual world and the real world is completed, the expressive force of the virtual environment and the virtual character is improved, and the interestingness of a virtual reality system and the immersive experience of a user are increased.
While the present disclosure has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the following claims.
Claims (18)
1. A method of implementing virtual reality, comprising:
acquiring sound in a real environment;
determining an ambient sound event corresponding to the sound;
generating a virtual event for presenting a process and/or result of the ambient sound event;
the virtual event is executed in a virtual space.
2. The method of claim 1, wherein the determining an ambient sound event corresponding to the sound comprises:
identifying and classifying the sound to obtain a classification label of the sound;
detecting the sound source of the sound to obtain the sound emitting position;
the ambient sound event is determined based on the classification tag of the sound and the emission location of the sound, wherein the ambient sound event includes at least one of an event execution subject, an event action, and an event location.
3. The method of claim 1, wherein the generating a virtual event for presenting a process and/or result of the ambient sound event comprises:
and reasoning based on the environment sound event, the state of the virtual space and the rule base to obtain the virtual event, wherein the virtual event comprises at least one of an event execution subject, an event action and an event position.
4. The method of claim 3, wherein said reasoning based on the ambient sound event, the state of the virtual space, the rule base, and deriving the virtual event comprises:
sequentially matching a fact library comprising the environmental sound event and the state of the virtual space with rules in a rule library through a pattern matcher to obtain an agenda, wherein the agenda sequentially stores the rules meeting the conditions;
and activating rules in the agenda through an execution engine to obtain the virtual event.
5. A method according to claim 3, wherein the rule base is a knowledge set expressed in terms of rules, including facts required to perform reasoning, virtual events that need to be generated after the rules are satisfied.
6. The method of claim 3, wherein the event execution body of the virtual event comprises a virtual character, a virtual accessory, a virtual environment, and a virtual special event body.
7. The method of claim 4, wherein the state of the virtual space comprises at least one of a avatar state, a virtual accessory state, a special event state, and a virtual environment state.
8. The method of claim 2, wherein the classification tag comprises at least one of a character, a character accessory, an ambient sound, a special sound, the character comprising at least one of a character behavior and a character vocalization, the character accessory comprising at least one of a pet, a character association, the ambient sound comprising at least one of a natural environment, a background, an electrical appliance, the special sound comprising at least one of an emergency sound, a special event sound.
9. An apparatus for implementing virtual reality, comprising:
a sound acquisition unit configured to acquire sound in a real environment;
an event determination unit configured to determine an ambient sound event corresponding to the sound;
an event generation unit configured to generate a virtual event for presenting a process and/or result of the ambient sound event; and
and an event execution unit configured to execute the virtual event in a virtual space.
10. The apparatus of claim 9, wherein the event determination unit is configured to:
identifying and classifying the sound to obtain a classification label of the sound;
detecting the sound source of the sound to obtain the sound emitting position;
the ambient sound event is determined based on the classification tag of the sound and the emission location of the sound, wherein the ambient sound event includes at least one of an event execution subject, an event action, and an event location.
11. The apparatus of claim 9, wherein the event generation unit is configured to:
and reasoning based on the environment sound event, the state of the virtual space and the rule base to obtain the virtual event.
12. The apparatus of claim 11, wherein the event generation unit is configured to:
sequentially matching a fact library comprising the environmental sound event and the state of the virtual space with rules in a rule library through a pattern matcher to obtain an agenda, wherein the agenda sequentially stores the rules meeting the conditions;
and activating rules in the agenda through an execution engine to obtain the virtual event.
13. The apparatus of claim 11, wherein the rule base is a knowledge set expressed in terms of rules, including facts required to perform reasoning, virtual events that need to be generated after the rules are satisfied.
14. The apparatus of claim 11, wherein the event execution body of the virtual event comprises a virtual character, a virtual accessory, a virtual environment, and a virtual special event body.
15. The apparatus of claim 12, wherein the state of the virtual space comprises at least one of a avatar state, a virtual accessory state, a special event state, and a virtual environment state.
16. The apparatus of claim 10, wherein the classification tag comprises at least one of a character, a character accessory, an ambient sound, a special sound, the character comprising at least one of a character behavior and a character vocalization, the character accessory comprising at least one of a pet, a character association, the ambient sound comprising at least one of a natural environment, a background, an electrical appliance, the special sound comprising at least one of an emergency sound, a special event sound.
17. A computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method of implementing virtual reality of any one of claims 1 to 8.
18. A computing device, comprising:
at least one processor;
at least one memory storing a computer program which, when executed by the at least one processor, implements the method of implementing virtual reality of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211430089.6A CN116051788A (en) | 2022-11-15 | 2022-11-15 | Method and device for realizing virtual reality |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211430089.6A CN116051788A (en) | 2022-11-15 | 2022-11-15 | Method and device for realizing virtual reality |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116051788A true CN116051788A (en) | 2023-05-02 |
Family
ID=86120703
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211430089.6A Pending CN116051788A (en) | 2022-11-15 | 2022-11-15 | Method and device for realizing virtual reality |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116051788A (en) |
-
2022
- 2022-11-15 CN CN202211430089.6A patent/CN116051788A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10586369B1 (en) | Using dialog and contextual data of a virtual reality environment to create metadata to drive avatar animation | |
| US10832674B2 (en) | Voice data processing method and electronic device supporting the same | |
| US20210065716A1 (en) | Voice processing method and electronic device supporting the same | |
| EP3997554A1 (en) | Semantically tagged virtual and physical objects | |
| US8484219B2 (en) | Developing a knowledge base associated with a user that facilitates evolution of an intelligent user interface | |
| US11548147B2 (en) | Method and device for robot interactions | |
| JP2018190413A (en) | Method and system for processing user command to adjust and provide operation of device and content provision range by grasping presentation method of user speech | |
| US11314548B2 (en) | Electronic device and server for processing data received from electronic device | |
| CN111919248B (en) | System for processing user utterance and control method thereof | |
| US8954356B2 (en) | Evolution of a user interface based on learned idiosyncrasies and collected data of a user | |
| WO2019124647A1 (en) | Method and computer apparatus for automatically building or updating hierarchical conversation flow management model for interactive ai agent system, and computer-readable recording medium | |
| CN112221139B (en) | Information interaction method and device for game and computer readable storage medium | |
| US20230259540A1 (en) | Conversational ai platform with extractive question answering | |
| US20230306666A1 (en) | Sound Based Modification Of A Virtual Environment | |
| JP2019220168A (en) | Systems and methods for designing haptics using speech commands | |
| CN108055617A (en) | Microphone awakening method and device, terminal equipment and storage medium | |
| KR20210042523A (en) | An electronic apparatus and Method for controlling the electronic apparatus thereof | |
| US20230143177A1 (en) | Contextual suppression of assistant command(s) | |
| CN111176435A (en) | User behavior-based man-machine interaction method and sound box | |
| WO2016206645A1 (en) | Method and apparatus for loading control data into machine device | |
| US20200402498A1 (en) | Information processing apparatus, information processing method, and program | |
| KR20200080389A (en) | Electronic apparatus and method for controlling the electronicy apparatus | |
| CN116051788A (en) | Method and device for realizing virtual reality | |
| JP7230803B2 (en) | Information processing device and information processing method | |
| CN116841391A (en) | Digital human interaction control method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |