WO2024169207A1

WO2024169207A1 - Method of displaying performance content of virtual character and related device

Info

Publication number: WO2024169207A1
Application number: PCT/CN2023/124424
Authority: WO
Inventors: 张鹏飞; 殷国君; 秦瑞; 李江伟
Original assignee: 华为云计算技术有限公司
Priority date: 2023-02-16
Filing date: 2023-10-13
Publication date: 2024-08-22
Also published as: CN118537452A

Abstract

Disclosed in the present application is a method of displaying performance content of a virtual character. The method comprises: acquiring a first streaming audio clip; on the basis of feature information of the first streaming audio clip and action feature information generated according to a second streaming audio clip, obtaining first fusion feature information, the first streaming audio clip and the second streaming audio clip being included in a same streaming audio, and the playing time of the second audio clip being before the playing time of the first audio clip; and on the basis of the first fusion feature information, generating facial expression information and body movement information of a virtual character. Therefore, the present application can directly output the performance content of the virtual character corresponding to the input streaming audio clip without the need of inputting a complete music piece, thus shortening the response time of generating facial expression information and body movement information of virtual characters.

Description

A method for displaying performance content of a virtual character and related equipment

This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office of China on February 16, 2023, with application number CN202310126607.3 and invention name “Method and system for generating virtual character dance movements based on cloud technology”, and the priority of the Chinese patent application filed with the State Intellectual Property Office of China on May 15, 2023, with application number CN202310544883.1 and invention name “A method for displaying performance content of a virtual character and related equipment”, all of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of computer technology, and in particular to a method for displaying performance content of a virtual character and related equipment.

Background Art

With the continuous development of computer technology, the technology of driving virtual characters to perform based on music has emerged, and now has very wide applications in the fields of virtual reality, human-computer interaction, etc. For example, the generated performance content can be applied to the virtual character based on a piece of music provided by the user.

In the related art, the performance content of the virtual character can be generated based on the algorithm. The solution of generating dance movements based on the algorithm is efficient and generalizable, and only needs to input the music information to generate the matching performance content. However, this method generally adopts an offline model, that is, it needs to input the complete music and extract the characteristic information of the music from the complete music to generate the corresponding performance content, and the response time is relatively long.

Summary of the invention

The present application provides a method and related equipment for displaying the performance content of a virtual character, which can directly output the performance content of the virtual character corresponding to the input streaming audio clip without inputting the complete music, thereby reducing the response time of generating the facial expression information and body movement information of the virtual character.

In a first aspect, the present application provides a method for displaying performance content of a virtual character, which can be applied in the field of computer technology and can be implemented by a target model. The method includes:

First, a first streaming audio segment is obtained; then, based on feature information of the first streaming audio segment and action feature information generated according to the second streaming audio segment, first fused feature information is obtained, wherein the first streaming audio segment and the second streaming audio segment are included in the same streaming audio, and the playback time of the second audio segment is before the playback time of the first audio segment; finally, facial expression information and body movement information of the virtual character are generated based on the first fused feature information.

In view of the problem that the offline model currently mainly adopts, the complete music needs to be input to generate the corresponding performance content, and the response time is long. In this application, on the one hand, the performance content of the virtual character is output in real time based on the input streaming audio clip, which reduces the response time of generating the performance content. On the second hand, when generating the facial expression information and body movement information of the virtual character, in addition to the feature information based on the first streaming audio clip, the action feature information generated by the second streaming audio clip is also referred to, so as to achieve the front and back connection of different audio clips of the same streaming audio in the performance content. On the third hand, considering that the amplitude and movement speed of the body movement corresponding to the virtual character have a certain influence on the facial expression, the action feature information is added when generating the first fusion feature information to enhance the display effect of the performance content of the virtual character. On the fourth hand, by synchronously outputting the body movement information and performance information of the virtual character, the effect of the virtual character singing and dancing is achieved.

In a possible implementation manner of the first aspect, the method further includes:

Get the third streaming audio segment;

Based on feature information of a third streaming audio segment and action feature information generated according to the first streaming audio segment, second fusion feature information is obtained, wherein the third streaming audio segment and the first streaming audio segment are included in the same streaming audio, and a playback time of the first streaming audio segment is before a playback time of the third streaming audio segment;

Facial expression information and body movement information of the virtual character are generated based on the second fused feature information.

In this possible implementation, for the same streaming audio, since the playback time of the third streaming audio segment is after the playback time of the first streaming audio segment, after generating the facial expression information and body movement information of the virtual character based on the first streaming audio segment, the facial expression information and body movement information of the virtual character corresponding to the third streaming audio segment can be generated based on the third streaming audio segment, thereby achieving the effect of online input of streaming audio segments and real-time output of the corresponding facial expression information and body movement information of the virtual character. In addition, in the process of generating the facial expression information and body movement information of the virtual character by the third streaming audio segment with a later playback time, the facial expression information and body movement information of the virtual character will be referenced. The action feature information generated by the second streaming audio segment that is played earlier is considered, thereby achieving the connection between the actions of the virtual character on the same audio segment and improving the user experience.

In a possible implementation manner of the first aspect, the feature information of the first streaming audio segment includes text feature information, and generating facial expression information of the virtual character based on the first fused feature information includes:

Extract phonemes based on text feature information to obtain phoneme feature information;

Acquire first associated feature information on expressions of multiple virtual characters, where the first associated feature information is generated based on the second streaming audio segment;

Based on the first fusion feature information, the phoneme feature information and the first associated feature information, obtaining weight information of a basic expression corresponding to at least one virtual character;

Based on the weight information of the basic expression corresponding to the at least one virtual character, the basic expression corresponding to the at least one virtual character is adjusted to obtain the facial expression information corresponding to the at least one virtual character.

In this possible implementation, on the one hand, by extracting phoneme feature information from the feature information of the first streaming audio clip, the correlation between facial expression and mouth shape is strengthened when generating the facial expression of the virtual character, which not only increases the display dimension of facial expression information, enhances the display effect of facial expression information, but also improves the authenticity of facial expression. On the other hand, by referring to the first associated features of the expressions of multiple virtual characters, while generating the facial expression information corresponding to each virtual character, the facial expression information of other virtual characters is referred to, thereby achieving effective collaboration and cooperation of multiple virtual characters in facial expressions.

Extracting expression feature information corresponding to the plurality of virtual characters based on the facial expression information corresponding to the plurality of virtual characters;

Based on the expression feature information corresponding to the multiple virtual characters, the expression similarity between the multiple virtual characters is calculated;

The expression similarity between multiple virtual characters is used as a weight to adjust the expression feature information corresponding to each virtual character to obtain second associated feature information on the expressions of the multiple virtual characters. The second associated feature information is used to generate facial expression information corresponding to at least one virtual character corresponding to the third streaming audio segment.

In this possible implementation, the expression similarity between each virtual character and other virtual characters is calculated based on the expression feature information corresponding to different virtual characters, so that when generating the facial expression information of each virtual character, not only the facial expression information of the virtual character is considered, but also the facial expression information of other virtual characters is considered, so as to achieve effective collaboration and cooperation of multiple virtual characters in facial expressions. In addition, the second associated feature information generated before the playback time of the third streaming audio segment is used as the input of the third streaming audio segment to generate facial expression information, so as to achieve effective connection in facial expressions between streaming audio segments with different playback times, maintain the coherence of the facial expression information of the virtual characters, and further utilize the temporal correlation of learning facial expressions, which is conducive to maintaining the temporal coherence of facial expressions and avoiding sudden changes and jitters in facial expressions.

In a possible implementation manner of the first aspect, generating body movement information of the virtual character based on the first fused feature information includes:

Performing detail adjustment on the first fused feature information to obtain adjusted first fused feature information;

Acquire third associated feature information on actions of the plurality of virtual characters, where the third associated feature information is generated based on the second streaming audio segment;

Obtaining motion feature information generated by the first streaming audio segment;

The action feature information generated by the first streaming audio segment is adjusted using the adjusted first fusion feature information and the third associated feature information as offsets to obtain corresponding body motion information of at least one virtual character.

In this possible implementation, compared with directly classifying related body movements as the same body movement information, the generated body movement information of the virtual character lacks movement details. In this implementation, by making a detailed adjustment to the first fusion feature information, the action feature information generated by the first streaming audio segment is adjusted based on the adjusted first fusion feature information, so that the body movement information generated by each virtual character contains more action details. In addition, by adjusting the action feature information generated by the first streaming audio segment based on the third correlation feature in the actions between multiple virtual characters, each virtual character can take into account the body movement information of other virtual characters and achieve collaboration and cooperation with other virtual characters in body movements.

In a possible implementation manner of the first aspect, obtaining motion feature information generated by the first streaming audio segment includes:

Based on the first fusion feature information, the third association feature information, and the action feature information generated by the second streaming audio segment, action coding information corresponding to the plurality of virtual characters is obtained;

Based on the action coding information, action feature information corresponding to the action coding information of multiple virtual characters is obtained.

In this possible implementation, compared with directly extracting effective action feature information from the first fused feature information, the third associated feature information, and the second streaming audio segment, the action coding information corresponding to the plurality of virtual characters is obtained by performing action coding on the feature information. Then, based on the correspondence between the action coding information and the action feature information, the action feature information corresponding to the action coding information of multiple virtual characters is obtained, thereby reducing the amount of information for generating the action feature information and improving the operation efficiency.

Based on the action feature information generated by the first streaming audio segment, calculating the action similarity between the multiple virtual characters;

Using the action similarity between multiple virtual characters as a weight, the corresponding action feature information of each virtual character is adjusted to obtain fourth associated feature information on the actions of the multiple virtual characters. The fourth associated feature information is used to generate body movement information corresponding to at least one virtual character corresponding to the third streaming audio segment.

In this possible implementation, the expression similarity between each virtual character and other virtual characters is calculated based on the action feature information generated by the first streaming audio segment, so that when generating the body movement information of each virtual character, not only the body movement information of the virtual character but also the body movement information of other virtual characters are considered, so as to achieve effective cooperation and coordination of multiple virtual characters in body movement. In addition, the fourth associated feature information generated before the playback time of the third streaming audio segment is used as the input of the facial expression information generated by the third streaming audio segment, so as to achieve effective connection in body movement between streaming audio segments with different playback times, maintain the coherence of the facial expression information of the virtual character, and further utilize the temporal correlation of learning facial expressions, which is conducive to maintaining the temporal coherence of facial expressions and avoiding sudden changes and jitters in facial expressions.

Acquire tag information, where the tag information is used to indicate facial expression information and/or body movement information of the virtual character;

The facial expression information and/or body movement information corresponding to each virtual character is adjusted based on the tag information to obtain the adjusted facial expression information and/or body movement information corresponding to each virtual character.

In this possible implementation, after generating facial expression information and/or body movement information corresponding to multiple virtual characters, the facial expression information and/or body movement information can be adjusted based on the tag information to generate performance content of different styles to adapt to different types of virtual characters, so that a user-controllable style can be generated according to the stylized tags specified by the user, thereby improving interactivity with the user.

Extract phonemes based on text features to obtain phoneme feature information;

Obtaining facial expression information generated by a second streaming audio segment;

Obtaining weight information of a basic expression corresponding to the virtual character based on the first fused feature information, the phoneme feature information, and the facial expression information generated by the second streaming audio segment;

Based on the weight information of the basic expression corresponding to the virtual character, the basic expression of the virtual character is adjusted to obtain the facial expression information of the virtual character.

In this possible implementation, for the scenario of generating facial expression information of a single virtual character, on the one hand, by extracting phoneme feature information from the feature information of the first streaming audio segment, the correlation between facial expression and mouth shape is strengthened when generating the facial expression of the virtual character, which not only increases the display dimension of facial expression information, enhances the display effect of facial expression information, but also improves the authenticity of facial expression. On the other hand, when generating the facial expression information of the virtual character corresponding to the first streaming audio segment, by referring to the facial expression information of the second streaming audio segment with an earlier playback time, the streaming audio segments with different playback times are effectively connected in facial expression, thereby enhancing the display effect of the facial expression of the virtual character and improving the user experience.

Adjusting the first fused feature information to obtain adjusted first fused feature information;

The action feature information generated by the first streaming audio segment is adjusted using the adjusted first fusion feature as an offset to obtain body action information corresponding to the virtual character.

In this possible implementation, compared with directly classifying related body movements as the same body movement information, the generated body movement information of the virtual character lacks movement details. In this implementation, by making detail adjustments to the first fused feature information, the movement feature information generated by the first streaming audio segment is adjusted based on the adjusted first fused feature information, so that the body movement information generated by each virtual character contains more movement details.

In a second aspect, the present application provides a virtual character performance content display device, which can be used in the field of computer technology. The device includes a streaming audio segment acquisition module, a fusion feature information generation module and a performance content generation module. Among them,

A streaming audio segment acquisition module, used to acquire a first streaming audio segment;

a fusion feature information generating module, configured to obtain first fusion feature information based on feature information of a first streaming audio segment and action feature information generated according to a second streaming audio segment, wherein the first streaming audio segment and the second streaming audio segment are included in the same streaming audio, and a playback time of the second audio segment is before a playback time of the first audio segment;

The performance content generation module is used to generate facial expression information and body movement information of the virtual character based on the first fused feature information.

In a possible implementation manner of the second aspect, the method further includes:

The streaming audio segment acquisition module is further used to acquire a third streaming audio segment;

The fusion feature information generating module is further used to obtain second fusion feature information based on feature information of a third streaming audio segment and action feature information generated according to the first streaming audio segment, wherein the third streaming audio segment and the first streaming audio segment are included in the same streaming audio, and the playback time of the first streaming audio segment is before the playback time of the third streaming audio segment;

The performance content generation module is also used to generate facial expression information and body movement information of the virtual character based on the second fused feature information.

In a possible implementation manner of the second aspect, the feature information of the first streaming audio segment includes text feature information, and the performance content generation module is further configured to:

In a possible implementation manner of the second aspect, the performance content generation module is further configured to:

In a possible implementation manner of the second aspect, obtaining motion feature information generated by the first streaming audio segment includes:

Extract phonemes based on text features to obtain phoneme feature information;

In the second aspect of the present application, the various modules included in the performance content display device of the virtual character can also be used to implement the steps in the various possible implementation methods of the first aspect. For the specific implementation methods of the second aspect of the embodiment of the present application and the various possible implementation methods of the second aspect, as well as the beneficial effects brought about by each possible implementation method, you can refer to the description of the various possible implementation methods in the first aspect, and will not be repeated here one by one.

In a third aspect, an embodiment of the present application provides a model training method, including:

Acquire a first streaming audio segment; obtain first fused feature information based on feature information of the first streaming audio segment and action feature information generated according to the second streaming audio segment, wherein the first streaming audio segment and the second streaming audio segment are included in the same streaming audio, and the playback time of the second audio segment is before the playback time of the first audio segment; generate facial expression information and body movement information of a virtual character based on the first fused feature information; obtain a target loss based on the facial expression information and body movement information of the virtual character and the real facial expression information and body movement information of the virtual character, the target loss being used to indicate the difference between the facial expression information and body movement information and the real facial expression information and body movement information; based on the target loss, update the parameters of the model to be trained until the model training conditions are met, and obtain the target model.

In the third aspect of the present application, it can be understood that the target model can be used to execute the steps in the aforementioned first aspect or various possible implementation methods of the first aspect, which will not be described one by one here.

In a fourth aspect, an embodiment of the present application provides a model training device, comprising:

A performance content generation module, used to generate facial expression information and body movement information of the virtual character based on the first fused feature information;

A target loss acquisition module is used to obtain a target loss based on the facial expression information and body movement information of the virtual character of the virtual character and the real facial expression information and body movement information of the virtual character, wherein the target loss is used to indicate the difference between the facial expression information and body movement information and the real facial expression information and body movement information;

The target model training module is used to update the parameters of the model to be trained based on the target loss until the model training conditions are met to obtain the target model.

In a fifth aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory;

The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

In a sixth aspect, an embodiment of the present application provides a training device, including a processor and a memory, wherein the processor is coupled to the memory. The memory is used to store a program. The processor is used to execute the program in the memory, so that the execution device executes the method of the third aspect above.

In the seventh aspect, an embodiment of the present application provides a computer storage medium, characterized in that it includes instructions, which, when executed on a computing device, enable the computing device to execute the method in the above-mentioned first aspect or any possible implementation of the first aspect, or the method in the above-mentioned third aspect.

In an eighth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, enables The computing device cluster executes the method in the first aspect or any possible implementation manner of the first aspect, or the method in the third aspect.

In a ninth aspect, an embodiment of the present application provides a circuit system, the circuit system including a processing circuit, the processing circuit being configured to execute the method described in the first aspect or any possible implementation of the first aspect, or the method of the third aspect.

In a tenth aspect, the present application provides a chip system, including a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method as described in the first aspect or any possible implementation of the first aspect, or the method of the third aspect. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

The solutions of the second to tenth aspects mentioned above are used to implement or cooperate with the method of the first aspect or any possible implementation method thereof, and therefore can achieve the same or corresponding beneficial effects as the first aspect, and will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a structure of an artificial intelligence main framework used in an embodiment of the present application;

FIG2 is a system architecture diagram provided in an embodiment of the present application;

FIG3 is a flow chart of a method for displaying performance content of a virtual character provided in an embodiment of the present application;

FIG4 is another system architecture diagram provided in an embodiment of the present application;

FIG5 is another schematic flow chart of a method for displaying performance content of a virtual character provided in an embodiment of the present application;

FIG6 is a schematic diagram of a process for generating first fusion feature information provided in an embodiment of the present application;

FIG7 is a schematic diagram of a structure of a stylization processing module provided in an embodiment of the present application;

FIG8 is a schematic diagram of a process for generating facial expression information of a virtual character provided in an embodiment of the present application;

FIG9 is a schematic diagram of a structure of a multi-person action decoder provided in an embodiment of the present application;

FIG10 is a schematic diagram of a flow chart of generating body movement information of a virtual character provided in an embodiment of the present application;

FIG11 is another schematic flow chart of a method for displaying performance content of a virtual character provided in an embodiment of the present application;

FIG12 is another schematic diagram of a process for generating facial expression information of a virtual character provided in an embodiment of the present application;

FIG13 is another schematic diagram of a flow chart of generating body movement information of a virtual character provided in an embodiment of the present application;

FIG14 is a flow chart of a model training method provided in an embodiment of the present application;

FIG15 is a schematic diagram of a structure of a device for displaying performance content of a virtual character provided in an embodiment of the present application;

FIG16 is a schematic diagram of a structure of a model training device provided in an embodiment of the present application;

FIG17 is a schematic diagram of a structure of a computing device provided in an embodiment of the present application;

FIG18 is a schematic diagram of a structure of a computer device cluster provided in an embodiment of the present application;

FIG19 is a schematic diagram of a structure of a training device provided in an embodiment of the present application;

FIG. 20 is a schematic diagram of the structure of a chip provided in an embodiment of the present application.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices.

The term "and/or" in this application can be a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character "/" in this application generally indicates that the associated objects before and after are in an "or" relationship.

It should also be noted that, in some alternative implementations, the functions/acts noted may occur out of the order of the drawings. For example, two figures shown in succession may in fact occur substantially simultaneously or may sometimes be performed in the reverse order, depending on the functions/acts involved.

In the embodiments of the present application, unless otherwise specified, the meaning of "at least one" refers to one or more, and the meaning of "plurality" refers to two or more. It is understood that in the present application, "when", "if" and "if" all refer to the device making corresponding processing under certain objective circumstances, and do not limit the time, nor do they require that there must be a judgment action when the device is implemented, nor do they mean that there are other limitations. In addition, the special word "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

The embodiments of the present application are described below in conjunction with the accompanying drawings. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

The performance content display method of the virtual character provided in this application can be applied to the field of computer technology, especially in the field of artificial intelligence (AI). AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

First, the overall workflow of the artificial intelligence system is described. Please refer to Figure 1, which is a structural diagram of the artificial intelligence main framework applied in the embodiment of the present application. The above artificial intelligence theme framework is explained from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data undergoes a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

(1) Infrastructure:

The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and is supported by the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips, such as central processing units (CPU), neural-network processing units (NPU), graphics processing units (GPU), application specific integrated circuits (ASIC) or field programmable gate arrays (FPGA) and other hardware acceleration chips; the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

(2) Data

The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, video, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

(3) Data processing

Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

(4) General capabilities

After the data is processed as mentioned above, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing (such as image recognition, object detection) etc.), speech recognition, etc.

(5) Smart products and industry applications

Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, smart terminals, etc.

The embodiments of the present application involve a large number of neural network-related applications. In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts of the neural network that may be involved in the embodiments of the present application are first introduced below.

(1) Neural Network

A neural network may be composed of neural units. For example, a neural unit may refer to an operation unit that takes x _s as input, and the output of the operation unit may be as follows:

Where s=1, 2, ...n, n is a natural number greater than 1, _Ws is the weight of _xs , and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

(2) Recurrent Neural Network

Recurrent neural network (RNN) is used to process sequence data. In the traditional neural network model, from the input layer to the hidden layer and then to the output layer, the layers are fully connected, and the nodes within each layer are disconnected. Although this ordinary neural network has solved many difficult problems, it is still powerless for many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layer are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment.

(3) Transformer structure

The Transformer structure is a feature extraction network (similar to a convolutional neural network) that includes an encoder and a decoder.

Encoder: Performs feature learning in the global receptive field through self-attention, such as pixel features.

Decoder: Learn the features of the required modules, such as the features of the output box, through self-attention and cross-attention.

(4) Multi-Layer Perceptron (MLP)

Multilayer Perceptron, also known as Multilayer Perceptron, is a feedforward artificial neural network model. MLP is an artificial neural network (ANN) based on a fully connected (FC) forward structure, which contains artificial neurons (AN, hereinafter referred to as neurons) ranging from a dozen to hundreds. MLP organizes neurons into a multi-layer structure, and uses a full connection method between layers to form an ANN with multi-weighted connection layers connected layer by layer. Generally speaking, MLP contains an input layer (this layer does not actually contain operations), one or more hidden layers, and an output layer.

(5) Loss function

The loss function can also be called the cost function, which is a metric that compares the difference between the predicted output of a machine learning model for a sample and the true value of the sample (also called the supervised value).

The loss function can usually include mean square error, cross entropy, logarithm, exponential and other loss functions. For example, the mean square error can be used as the loss function, defined as The specific loss function can be selected according to the actual application scenario.

The present application can be applied to application scenarios in which facial expression information and body movement information of virtual characters are generated based on music in the above-mentioned application fields. Specifically, as an example, when it is necessary to generate facial expression information and body movement information of virtual characters corresponding to music in scenes such as movies, games, large-scale evening parties, virtual live broadcasts, and in-car large-screen entertainment, facial expression information and body movement information of virtual characters corresponding to different scenes can be generated based on the soundtrack in the movie, the music in the game, the music in the large-scale evening party or humming, etc. The application scenarios of the embodiments of the present application are not exhaustively listed here.

Before introducing in detail the method for displaying the performance content of a virtual character provided in the embodiment of the present application, the system architecture provided in the embodiment of the present application is first introduced in conjunction with Figure 2.

Please refer to Figure 2, which is a system architecture diagram provided in an embodiment of the present application. In the embodiment shown in Figure 2, the system architecture 200 includes a database 230 and a client device 240. The data acquisition device 260 is used to collect data and store it in the database 230, and the training module 202 generates a target model/rule 201 based on the data maintained in the database 230. The following will describe in more detail how the training module 202 obtains the target model/rule 201 based on the data. The target model/rule 201 is the target model mentioned in the following implementation of the present application. Please refer to the relevant descriptions in the following Figures 5-14.

The calculation module 211 may include a training module 202, and the target model/rule obtained by the training module 202 may be applied to different systems or devices. In FIG2 , the execution device 210 is configured with a transceiver 212, which may be a wireless transceiver, an optical transceiver, or a wired interface (such as an I/O interface), etc., to interact with external devices for data, and a "user" may input data to the transceiver 212 through a client device 240. For example, the client device 240 may send a target task to the execution device 210, request the execution device to train a neural network, and send a database for training to the execution device 210.

The execution device 210 can call data, codes, etc. in the data storage system 250 , and can also store data, instructions, etc. in the data storage system 250 .

The calculation module 211 uses the target model/rule 201 to process the input data. Specifically, in a possible implementation, please refer to Figure 3, which is a flow chart of a method for displaying the performance content of a virtual character provided in an embodiment of the present application. The method can be executed by the calculation module 211 as shown in Figure 2, specifically, by the execution device 210. The execution device 210 is used to: S1, obtain a first streaming audio segment; S2, based on the feature information of the first streaming audio segment and the action feature information generated according to the second streaming audio segment, obtain first fusion feature information, the first streaming audio segment and the second streaming audio segment are included in the same streaming audio, and the playback time of the second audio segment is before the playback time of the first audio segment; S3, generate facial expression information and body movement information of the virtual character based on the first fusion feature information.

Finally, the transceiver 212 returns the output result of the target model to the client device 240. For example, the user can input an audio clip through the client device 240, and the facial expression information and body movement information of the virtual character are output through the target model and fed back to the client device 240.

More deeply, the training module 202 can obtain corresponding target models/rules 201 for different tasks based on different data to provide users with better results.

In the case shown in FIG. 2 , the data input into the execution device 210 can be determined based on the user's input data. For example, the user can operate in the interface provided by the transceiver 212. In another case, the client device 240 can automatically input data into the transceiver 212 and obtain the result. If the automatic data input of the client device 240 requires the user's authorization, the user can set the corresponding authority in the client device 240. The user can view the result output by the execution device 210 in the client device 240. The specific presentation form can be specific forms such as expressions and actions. The client device 240 can also serve as a data collection terminal to store the collected data associated with the target task into the database 230.

The training or updating process mentioned in the present application can be performed by the training module 202. It is understandable that the training process of the neural network is to learn the way to control the spatial transformation, more specifically, to learn the weight matrix. The purpose of training the neural network is to make the output of the neural network as close to the expected value as possible. Therefore, the weight vector of each layer of the neural network in the neural network can be updated according to the difference between the predicted value and the expected value of the current network by comparing the predicted value and the expected value of the current network (of course, the weight vector can usually be initialized before the first update, that is, the parameters of each layer in the deep neural network are pre-configured). For example, if the predicted value of the network is too high, the value of the weight in the weight matrix is adjusted to reduce the predicted value, and after continuous adjustment, the value output by the neural network is close to or equal to the expected value. Specifically, the difference between the predicted value and the expected value of the neural network can be measured by a loss function or an objective function. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. The training of the neural network can be understood as the process of minimizing the loss as much as possible. The process of updating the weight of the starting network and training the serial network in the following embodiments of the present application can refer to this process, which will not be repeated below.

As shown in FIG. 2 , a target model/rule 201 is obtained by training the training module 202. The target model/rule 201 is implemented in the present application. Examples may include deep convolutional neural networks (DCNN), recurrent neural networks (RNNS), etc. The neural networks mentioned in this application may include various types, such as deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN) or residual networks and other neural networks.

In the training phase, the database 230 can be used to store sample sets for training. The execution device 210 generates a target model/rule 201 for processing samples, and iteratively trains the target model/rule 201 using the sample set in the database to obtain a mature target model/rule 201, which is specifically represented by a neural network. The neural network obtained by the execution device 210 can be applied to different systems or devices.

In the inference stage, the execution device 210 can call the data, code, etc. in the data storage system 250, or store the data, instructions, etc. in the data storage system 250. The data storage system 250 can be placed in the execution device 210, or the data storage system 250 can be an external memory relative to the execution device 210. The calculation module 211 can process the samples obtained by the execution device 210 through the neural network to obtain the prediction result. The specific expression form of the prediction result is related to the function of the neural network.

It should be noted that FIG2 is only an exemplary schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG2, the data storage system 250 is an external memory relative to the execution device 210. In other scenarios, the data storage system 250 can also be placed in the execution device 210.

The target model/rule 201 trained by the training module 202 can be applied to different systems or devices, such as mobile phones, tablet computers, laptops, augmented reality (AR)/virtual reality (VR), vehicle terminals, etc., and can also be servers or cloud devices.

The target model/rule 201 in the embodiment of the present application may be a model for executing the performance content display method of the virtual character provided in the present application, that is, the target model/rule 201 may be a neural network provided in the present application for generating facial expression information and body movement information of the virtual character. Specifically, the target model provided in the embodiment of the present application may include one or more networks such as CNN, deep convolutional neural networks (DCNN), recurrent neural networks (RNN), etc.

Please refer to Figure 4, which is another system architecture diagram provided by an embodiment of the present application. In the system architecture 400, the execution device 210 is implemented by one or more servers, and optionally cooperates with other computing devices, such as data storage, routers, load balancers and other devices; the execution device 210 can be arranged at one physical site, or distributed at multiple physical sites. The execution device 210 can use the data in the data storage system 250, or call the program code in the data storage system 250 to implement the steps of the training method for the computing device corresponding to the following Figure 14 of the present application.

Users can operate their respective user devices (e.g., local device 401 and local device 402) to interact with execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

The local device of each user can interact with the execution device 210 through a communication network of any communication mechanism/communication standard, and the communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof. Specifically, the communication network may include a wireless network, a wired network, or a combination of a wireless network and a wired network, etc. The wireless network includes but is not limited to: a fifth-generation mobile communication technology (5th-Generation, 5G) system, a long-term evolution (long term evolution, LTE) system, a global system for mobile communication (global system for mobile communication, GSM) or a code division multiple access (code division multiple access, CDMA) network, a wideband code division multiple access (wideband code division multiple access, WCDMA) network, wireless fidelity (wireless fidelity, WiFi), Bluetooth (bluetooth), Zigbee protocol (Zigbee), radio frequency identification technology (radio frequency identification, RFID), long-range (Lora) wireless communication, and near-field wireless communication (NFC) Any one or more combinations. The wired network may include an optical fiber communication network or a network composed of coaxial cables, etc.

In another implementation, one or more aspects of the execution device 210 may be implemented by each local device, for example, the local device 401 may provide local data or feedback calculation results to the execution device 210. The local device may also be referred to as a computing device.

It should be noted that all functions of the execution device 210 can also be implemented by the local device. For example, the local device 401 implements the functions of the execution device 210 and provides services to its own user, or provides services to the user of the local device 402.

In order to better understand the technical solution provided by the embodiments of the present application, the currently used method for displaying the performance content of a virtual character is first introduced below.

1. Real-time Motion Capture

The real-time motion capture solution mainly includes three steps: wearable devices, real-time motion capture, and character driving. First, the dancers put on the motion capture suits in advance and complete the calibration; then, the staff starts dancing and uses the motion capture software to collect the corresponding body movement sequences in real time; finally, the collected body movement sequences are transferred to the virtual character to be driven through motion redirection, driving the virtual character to show the body movements corresponding to the music.

Real-time motion capture uses a real-person driven approach, which allows staff to drive virtual characters in real time, and the body movements are realistic and rich.

However, real-time motion capture is expensive and requires a high level of professionalism from the staff. In addition, the real-time motion capture method is not generalizable and requires the staff's body movements to be re-collected each time to drive the virtual character, which is not suitable for application scenarios with large demands.

2. Generate body movements based on algorithms

Generating body movements based on algorithms can greatly improve the efficiency of body movement generation. Compared with simply generating body movements, more attention is currently paid to generating body movements based on music. That is, input complete music and use algorithms to generate body movements that match the input music in terms of rhythm, style, and other dimensions.

In some solutions, the algorithm-based solution for generating body movements is efficient and generalizable, but it is mainly an offline model, that is, it is necessary to input complete music to generate corresponding body movements. This requires extracting information such as the rhythm of the music from the complete music. However, the offline model has a long response time, which limits the potential application scenarios of music-driven body movement generation.

In other solutions, the dance movements of a single virtual character are mainly generated, and the ability to collaborate between multiple characters is not available. In addition, the faces and limbs of multiple virtual characters cannot be driven synchronously.

In other solutions, the generated dance moves are of fixed style and cannot be well adapted to the styles of different types of characters.

Compared with the above scheme, in the embodiment of the present application, on the one hand, the performance content of the virtual character is output in real time based on the input streaming audio clip, which reduces the response time of generating the performance content. On the other hand, by referring to the associated features of multiple virtual characters in facial expressions and body movements, the effective collaboration and coordination of multiple virtual characters in facial expressions and body movements is achieved. On the third hand, by adding stylized tags, the styles of different types of virtual characters are effectively adapted. On the fourth hand, by synchronously outputting the body movement information and performance information of the virtual character, the effect of the virtual character singing and dancing is achieved.

In order to solve the above problems, the embodiment of the present application provides a method for displaying the performance content of a virtual character. The following describes the specific implementation process of the reasoning phase and the training phase of the method for displaying the performance content of a virtual character provided by the embodiment of the present application in combination with the accompanying drawings and application scenarios.

1. Reasoning Stage

For ease of explanation, the following describes multiple virtual character scenarios and a single virtual character scenario respectively.

In an optional implementation, the corresponding implementation method can be executed according to the number of virtual characters selected by the user to generate performance content information of the virtual characters controllable by the user. This allows the execution device to generate facial expression information and body movement information of multiple virtual characters according to parameters such as the number of virtual characters set by the user, thereby improving user interactivity.

(I) Multiple virtual character scenes

In the embodiment of the present application, the reasoning stage describes the process of how the execution device 210 uses the target model/rule 201 to process the collected information data to generate a prediction result. Please refer to Figure 5 for details. Figure 5 is another flow chart of the performance content display method of the virtual character provided in the embodiment of the present application. The method can be completed by the target model. The method can be implemented by the target model, including steps 501 to 517.

Step 501: An execution device obtains a first streaming audio segment.

In the embodiment of the present application, the execution device obtains the first streaming audio segment in real time.

It is understandable that the response time is long when inputting complete music to generate corresponding performance content. Streaming audio means that audio can be input in real time in a streaming manner without the need to input complete audio, thereby achieving the effect of driving the character while playing music. It can be applied to online scenarios and reduce response time.

In an optional implementation manner, the execution device obtains audio information of the first streaming audio segment.

In this implementation, in scenarios such as humming, the execution device obtains audio information of the first stream audio segment, and needs to extract corresponding text information from the audio information of the first stream audio segment.

Optionally, the execution device may use automatic speech recognition technology (ASR) to parse corresponding text information from the audio information.

It is understandable that, in actual application, the execution device may use a variety of speech recognition technologies to parse the corresponding text information from the audio information, which is not limited here.

In another optional implementation, the execution device obtains the audio information and text information of the first streaming audio segment.

In this implementation, the execution device can directly obtain the text information of the first stream audio segment. For example, the execution device can directly obtain the first stream audio segment with lyrics text, directly obtain the audio information of the first stream audio segment and the text information corresponding to the first stream audio segment, and no longer need to parse the text information from the audio information of the first stream audio segment.

Step 502: The execution device extracts feature information of the first streaming audio segment from the first streaming audio segment.

In the embodiment of the present application, after acquiring the audio information and text information of the first stream audio segment, the execution device extracts the feature information of the first stream audio segment from the first stream audio segment, wherein the feature information of the first stream audio segment includes the first audio feature information and the text feature information.

Optionally, the execution device uses open source libraries such as librosa and madmom to extract first audio feature information such as Onset and Chromatogram from the audio information.

Optionally, the execution device uses a text encoder module in a Contrastive Language-Image Pre-Training (CLIP) model to extract text feature information.

On the basis that the execution device extracts the first audio feature information and the text feature information from the first streaming audio segment, in an optional implementation manner, the feature information of the first streaming audio segment further includes rhythm feature information.

Optionally, the execution device uses an MLP network to predict the rhythm feature information from the first audio feature information.

It is understandable that in order to better generate facial expression information and body movement information corresponding to the audio clip, the matching of rhythm is crucial. However, for audio clips of shorter length, such as audio clips of less than 1s, it is difficult to extract effective rhythm feature information or the accuracy of extraction is low. The embodiment of the present application uses an MLP network to achieve the matching of audio clips with facial expression information and body movement information in rhythm, and continuously optimizes the network through training and learning.

In another optional implementation, the feature information of the first streaming audio segment further includes phoneme feature information.

Optionally, the execution device extracts phoneme feature information from text feature information using an open source library and a text alignment tool.

For example, the execution device can use open source libraries such as Phonemizer to extract phoneme information, and use alignment tools such as MFA (Montreal Forced Aligner) to extract timestamp information to achieve alignment of audio and text in the time dimension. Phoneme represents a basic unit of sound production. Phoneme feature information includes multiple phonemes decomposed from the text and the duration of each phoneme.

It is understandable that phoneme features play an important role in the lip-driven effect of facial expressions. The embodiment of the present application uses an open source library to extract phonemes from audio text, and uses an MFA tool to align text and audio to obtain the timestamp information corresponding to each phoneme.

Step 503: The execution device obtains motion feature information generated by the second streaming audio segment.

In the embodiment of the present application, considering that the amplitude and speed of the virtual character's body movements have a certain influence on the facial expression, the execution device adds the action feature information when generating the first fusion feature information. The execution device obtains the action feature information generated by the second streaming audio segment to achieve the connection between different audio segments of the same streaming audio in terms of performance content.

It is understandable that the second stream audio segment can be any audio segment before the first stream audio segment is played, for example, the previous audio segment of the first stream audio segment, etc., and can be set according to actual needs and is not limited here. In addition, when the first stream audio segment is the first audio segment of the current audio, the action feature vector can be set randomly, by user-specified method or by historical value.

Step 504: The execution device obtains first fusion feature information based on the feature information of the first streaming audio segment and the action feature information generated according to the second streaming audio segment.

In the embodiment of the present application, the execution device fuses the feature information of the first streaming audio segment and the feature information of the action generated by the second streaming audio segment to obtain first fused feature information.

In an optional implementation, a specific method for the execution device to obtain the first fusion feature information includes:

The execution device performs feature fusion on the first audio feature information and the rhythm feature information of the first streaming audio segment to obtain second audio feature information;

The execution device concatenates the text feature information, the second audio feature information and the action feature information, and inputs the concatenated information into the first neural network to obtain the first fusion feature information.

In this implementation, for example, please refer to FIG. 6, which is a flow chart of generating the first fused feature information provided in an embodiment of the present application. The execution device performs feature fusion on the first audio feature information and the predicted rhythm feature information to obtain the second audio feature information. Then, the execution device splices the text feature information, the second audio feature information and the action feature information, and inputs them into the first neural network to obtain the first fused feature information.

Optionally, the execution device concatenates the text feature information, the first feature information, and the action feature information, and inputs the concatenated information into a multi-layer Transformer network to output the first fused feature information.

It should be understood that in actual application, the type of the first neural network can be set according to actual needs, which is only used as an example and not limited here.

After executing steps 501 to 504, there may be multiple implementations based on different generated contents. The following takes the execution device generating the facial expression information and body movement information of the virtual character as an example to illustrate the following steps. It should be understood that in actual application, the execution device synchronously outputs the facial expression information and body movement information of the virtual character to drive the virtual character to achieve the effect of singing and dancing. The execution device executes steps 505-510 and steps 511-517 in parallel.

(a) Generate facial expression information of virtual characters

Step 505: The execution device obtains first associated feature information on expressions of multiple virtual characters, where the first associated feature information is generated based on the second streaming audio segment.

In the embodiment of the present application, the execution device obtains the first associated feature information on the expressions of multiple virtual characters generated based on the second streaming audio segment, and generates the facial expression information corresponding to the second streaming audio segment based on the first associated feature information. Thus, while generating the facial expression information corresponding to each virtual character, the facial expression information of other virtual characters is referenced, so as to achieve effective collaboration and cooperation of the multiple virtual characters in facial expressions.

It is understandable that when the first streaming audio segment is the first audio segment obtained or the method is executed for the first time, the first associated feature information defaults to the feature information corresponding to the initial expression. The initial expression can be set according to actual needs, such as the initial default expressionless, which is limited here.

Step 506: The execution device obtains weight information of a basic expression corresponding to at least one virtual character based on the first fusion feature information, the phoneme feature information and the first associated feature information.

In an embodiment of the present application, the execution device generates weight information of a basic expression corresponding to at least one virtual character based on the first fused feature information, the phoneme feature information extracted from the text feature information, and the first associated feature information generated from the second streaming audio segment.

In an optional implementation, the execution device inputs the first fusion feature information, the phoneme feature information and the first association feature information into a multi-layer AcLSTM network, and outputs weight information of a basic expression corresponding to at least one virtual character by autoregression.

In this implementation, the execution device can output the weight information of the basic expressions corresponding to multiple virtual characters in an autoregressive manner. That is, when the execution device outputs the weight information of the basic expressions corresponding to multiple virtual characters corresponding to the first streaming audio segment, the second streaming audio segment is used to generate the first associated feature information. The second streaming audio segment is the previous audio segment or the previous audio segment of the first streaming audio segment. Correspondingly, when the execution device outputs the weight information of the basic expressions corresponding to multiple virtual characters corresponding to the third streaming audio segment, the first streaming audio segment is used to generate the second associated feature information. The third streaming audio segment is the next audio segment or the subsequent audio segment of the first streaming audio segment. As a result, the execution device uses the associated feature information output before the playback time of the current streaming audio segment as input for re-iteration, so as to further utilize the temporal correlation of the learned facial expressions, maintain the temporal continuity of the facial expressions, and avoid sudden changes and jitters in facial expressions.

It is understandable that compared with the LSTM network, the AcLSTM (Auto-Conditioned LSTM) network can better handle the problem of generating long sequences, that is, it can avoid or minimize the situation where the generated facial expressions no longer change or change less as the processing time increases when generating long facial expression sequences.

Step 507: The execution device adjusts the basic expression corresponding to the at least one virtual character based on the weight information of the basic expression corresponding to the at least one virtual character to obtain facial expression information corresponding to the at least one virtual character.

In the embodiment of the present application, the execution device performs a weighting operation on at least one virtual character based on the weighting information of the basic expression corresponding to at least one virtual character. The basic expression corresponding to the color is adjusted to make detailed adjustments to the basic expression corresponding to the virtual character to obtain facial expression information matching the first streaming audio clip.

In an optional implementation, the execution device obtains basic expression information corresponding to at least one virtual character, and obtains facial expression information corresponding to the at least one virtual character by weighted summation.

In this implementation, the basic expression information corresponding to at least one virtual character can be understood as a set of basic expression bases for each virtual character. For example, the left eye rolling up, the right eye rolling up, etc. can correspond to an expression base, and each basic expression base can represent the position of the vertex of the facial mesh of the virtual character in a specific state, and each vertex can be a three-dimensional coordinate.

Exemplarily, the formula for the execution device to generate facial expression information corresponding to at least one virtual character may be:

Among them, _Fi represents the facial expression information corresponding to the virtual character i, N represents the number or dimension of the expression base, b _l represents the basic expression information corresponding to the expression base l, e _l represents the weight information of the basic expression corresponding to the expression base l, and _b0 represents the basic expression, such as the default expressionless face.

It is understandable that although the execution device uses feature information between multiple virtual characters in the process of outputting facial expression information corresponding to the virtual characters, when finally outputting the facial expression information corresponding to the virtual characters, the number of virtual characters finally displayed can be set according to actual needs to display facial expression information corresponding to one or more virtual characters.

Furthermore, after the execution device obtains the facial expression information corresponding to at least one virtual character, before driving the facial expression of the virtual character based on redirection or other methods, in an optional implementation, the execution device may obtain tag information and adjust the facial expression information corresponding to each virtual character based on the tag information to obtain the adjusted facial expression information corresponding to each virtual character. The tag information is used to indicate the facial expression information of the virtual character.

In this implementation, the execution device may use a stylization processing module to perform stylization processing on the facial expression information of the generated virtual character according to the tag information specified by the user, and generate facial expression information of multiple styles to adapt to different types of virtual characters.

Optionally, the stylization processing module can adopt the ERD (Encoder-RNN-Decoder) architecture, and use the Encoder network to de-stylize the facial expression information of multiple virtual characters input, and only retain the expression content part. Then, by combining the RNN network of the label information of a specific style with the output of the first RNN network, the facial expression features with a specific style are obtained, and finally the Decoder network is used to decode the stylized facial expression information.

Exemplarily, please refer to FIG. 7, which is a schematic diagram of the structure of a stylized processing module provided in an embodiment of the present application. The stylized processing module includes branches corresponding to multiple stylized labels, such as pride, happiness, etc., and at least one first RNN network (used to represent neutral motion sequence features without style). The execution device obtains facial expression information with a specific style by adding the output of the RNN network branch of the specific stylized label to the output of the first RNN network branch. Assuming that a style of happiness plus pride is generated, the execution device can add the output of the RNN network branch of the stylized label corresponding to happiness to the output of the first RNN network branch, and then add the output of the RNN network branch of the stylized label corresponding to pride to the output of the first RNN network branch. Finally, the two added results are averaged to obtain facial expression information with styles corresponding to happiness and pride.

It can be understood that the first RNN network can be understood as a default RNN network, or a basic RNN network without style.

Step 508: The execution device extracts expression feature information corresponding to the multiple virtual characters based on the facial expression information corresponding to the multiple virtual characters.

In an embodiment of the present application, after acquiring the facial expression information corresponding to multiple virtual characters corresponding to the first streaming audio segment, the execution device uses the facial expression information corresponding to the multiple virtual characters as input in an autoregressive manner and outputs expression feature information corresponding to the multiple virtual characters.

In an optional implementation, the execution device inputs facial expression information corresponding to multiple virtual characters into the MLP network, and extracts expression feature information corresponding to each virtual character.

It is understandable that in actual application, a specific network model can be selected according to actual needs to extract expression feature information, such as a Transformer network, etc. This is only an example and not limited to one example.

Step 509: The execution device calculates the similarity of expressions between the multiple virtual characters based on the expression feature information corresponding to the multiple virtual characters.

In the embodiment of the present application, after the execution device extracts the expression feature information corresponding to the multiple virtual characters based on the facial expression information corresponding to the multiple virtual characters, it can calculate the similarity in expression between each virtual character and other virtual characters in the multiple virtual characters. Obtain the associated feature information of expressions between each virtual character and other virtual characters.

In an optional implementation, the execution device calculates the cosine similarity between the expression feature information of each virtual character and the expression feature information of other virtual characters as the similarity in expression between the multiple virtual characters.

Step 510: The execution device uses the similarity of expressions between the multiple virtual characters as a weight to adjust the expression feature information corresponding to each virtual character to obtain second associated feature information on the expressions of the multiple virtual characters.

In the embodiment of the present application, after calculating the cosine similarity between the expression feature information k _i of each virtual character and the expression feature information k _j of other virtual characters j, the execution device uses the cosine similarity as a weight coefficient and performs weighted summation with the expression feature information k _j of other virtual characters j to obtain a cross feature. Finally, the expression feature information k _i of the current virtual character i and the cross feature are bitwise added to obtain the second associated feature information _fi of the virtual character i in expression.

In an optional implementation, the execution device may obtain the second associated feature information _fi of the facial expression of the virtual character i according to the following calculation formula.

Among them, _fi represents the second associated feature information of the expressions of virtual character i and other virtual characters j, k _i represents the expression feature information corresponding to virtual character i, n represents the number of virtual characters, k _j represents the expression feature information corresponding to virtual character j, and cosine_sim() is used to calculate the cosine similarity of the expressions of virtual characters i and j.

For the convenience of explaining the process of generating facial expression information of a virtual character, please refer to FIG8 for example, which is a flow chart of generating facial expression information of a virtual character provided by an embodiment of the present application. Please refer to steps 505 to 510 for specific execution steps.

It can be understood that the structural diagram shown in FIG. 8 is only an example structural description of one embodiment of the present application when generating facial expression information of a virtual character, and is not limited here.

(b) Generate virtual character’s body movement information

Step 511: The execution device obtains third associated feature information and action feature information generated by the second streaming audio segment.

In an embodiment of the present application, the execution device obtains third-association feature information and action feature information on the actions of multiple virtual characters generated based on the second streaming audio segment, and generates body movement information corresponding to the first streaming audio segment based on the third-association feature information and action feature information, so that the execution device can refer to the body movement information of other virtual characters while generating the body movement information corresponding to each virtual character, thereby achieving effective collaboration and coordination of multiple virtual characters in body movements.

It is understandable that when the second streaming audio segment is the first audio segment or the method is executed for the first time, the third associated characteristic information defaults to the characteristic information corresponding to the initial action. The initial action can be set according to actual needs and is limited here.

Step 512: The execution device obtains action coding information corresponding to the multiple virtual characters based on the first fusion feature information, the third association feature information and the action feature information.

In an embodiment of the present application, the execution device obtains action coding information corresponding to multiple virtual characters based on the first fusion feature information, the third associated feature information generated by the second streaming audio segment, and the action feature information generated by the second streaming audio segment.

It is understandable that the execution device can output the action coding information corresponding to multiple virtual characters in an autoregressive manner. That is, when the execution device outputs the action coding information corresponding to multiple virtual characters corresponding to the first streaming audio segment, the second streaming audio segment is used to generate the third associated feature information and action feature information. The second streaming audio segment is the previous audio segment or the previous audio segment of the first streaming audio segment. Correspondingly, when the execution device outputs the action coding information corresponding to multiple virtual characters corresponding to the third streaming audio segment, the first streaming audio segment is used to generate the fourth associated feature information and action feature information. The third streaming audio segment is the next audio segment or the subsequent audio segment of the first streaming audio segment. As a result, the execution device uses the associated feature information output before the playback time of the current streaming audio segment as input for re-iteration, so as to further utilize the temporal correlation of the learned body movements, which is conducive to maintaining the temporal continuity of the body movements and reducing the incoherence of the body movements.

Optionally, the execution device inputs the first fused feature information, the third associated feature information, and the action feature information generated by the second streaming audio segment into a multi-layer AcLSTM network, and outputs action encoding information corresponding to multiple virtual characters by autoregression.

It can be understood that compared with the LSTM network, the AcLSTM (Auto-Conditioned LSTM) network can better handle the problem of generating long sequences, that is, it can avoid or minimize the situation where the generated limb movements no longer change or change less as the processing time increases when generating long limb movement sequences.

Step 513: The execution device obtains action feature information corresponding to the action coding information of the plurality of virtual characters based on the action coding information.

In the embodiment of the present application, the execution device obtains the action feature information corresponding to the action coding information of multiple virtual characters based on the correspondence between the action coding information and the action feature information.

It can be understood that each action encoding information corresponds to an action feature information in the code book (Code Book), which is used to represent a multi-frame full-body action sequence. The execution device can directly obtain the corresponding action feature information of multiple virtual characters from the Code Book based on the action encoding information of multiple virtual characters input.

Compared with directly inputting the posture information of each virtual character, including the position coordinates, rotation angle and speed of the joint points, to generate the action feature information corresponding to each virtual character, the amount of input information is large and the operation efficiency is low. The execution device can pre-encode the position coordinates, rotation angle and speed of each joint point to obtain the corresponding action coding information. In actual operation, the execution device only needs to obtain the action coding information corresponding to each virtual character, and based on the correspondence between the action coding information and the action feature information, the action feature information corresponding to each virtual character can be obtained, thereby reducing the amount of information processing and improving the operation efficiency.

Step 514: The execution device calculates the action similarity between the multiple virtual characters based on the action feature information generated by the first streaming audio segment.

In an embodiment of the present application, after the execution device extracts the action feature information corresponding to multiple virtual characters based on the action coding information corresponding to the multiple virtual characters, it can obtain the associated feature information on the actions between each virtual character and other virtual characters by calculating the similarity in actions between each virtual character and other virtual characters in the multiple virtual characters.

In an optional implementation, the execution device calculates the cosine similarity between the action feature information of each virtual character and the action feature information of other virtual characters as the action similarity between the multiple virtual characters.

Step 515: The execution device uses the action similarities between the multiple virtual characters as weights to adjust the corresponding action feature information of each virtual character to obtain fourth associated feature information on the actions of the multiple virtual characters.

In the embodiment of the present application, after calculating the cosine similarity between the action feature information p _i of each virtual character and the action feature information p _j of other virtual characters j, the execution device uses the cosine similarity as a weight coefficient and performs weighted summation with the action feature information p _j of other virtual characters j to obtain a cross feature. Finally, the action feature information p _i of the current virtual character i and the cross feature are bitwise added to obtain the second associated feature information _fi of the virtual character i in action.

In an optional implementation, the execution device may obtain the fourth associated feature information f′ _i of the action of the virtual character i according to the following calculation formula.

Among them, f′ _i represents the fourth associated feature information of the action between virtual character i and other virtual characters, _pi represents the action feature information corresponding to virtual character i, n represents the number of virtual characters, _pj represents the action feature information corresponding to virtual character j, and cosine_sim() is used to calculate the cosine similarity between virtual character i and virtual character j in action.

Step 516: The execution device performs detail adjustment on the first fused feature information to obtain adjusted first fused feature information.

In an embodiment of the present application, since the first fusion feature is obtained based on the feature information of the first streaming audio segment and the action feature information generated by the second streaming audio segment, the execution device obtains the adjusted first fusion feature information by making detail adjustments to the action feature included in the first fusion feature.

Compared with the Code Book obtained based on model training in the related art, there is a clustering effect problem, that is, similar actions will be classified into the same action encoding information, which will cause each action encoding information to encode the average action of a group of similar actions. This will cause each feature vector in the action encoding information to lack action details. The embodiment of the present application generates the offset of the limb action by using a single action fine-tuning network (such as an MLP network) to generate more action details.

In an optional implementation, the execution device uses an MLP network to perform detail adjustments on the first fused feature information.

It is understandable that, in actual application, the execution device may also use other network models to make detailed adjustments to the first fusion feature information. This is only an example and is not limited to this.

Step 517: The execution device adjusts the action feature information generated by the first streaming audio segment using the adjusted first fusion feature information and the third associated feature information as offsets to obtain body action information corresponding to at least one virtual character.

In the embodiment of the present application, after the execution device obtains the action feature information generated by the first streaming audio segment from the Code Book, one party On the one hand, for the acquired action feature information, the execution device calculates the fourth associated feature of each virtual character in a manner similar to the second associated feature extraction, and uses it as input information for generating action encoding information corresponding to multiple virtual characters; on the other hand, the execution device uses the adjusted first fusion feature information and the third associated feature information as offsets to adjust the action feature information generated by the first streaming audio segment, so as to obtain corresponding body motion information of at least one virtual character.

It is understandable that after obtaining the first fused feature information, the execution device will, on the one hand, make detailed adjustments to the first fused feature information, and on the other hand, obtain the action coding information corresponding to the multiple virtual characters based on the action feature information generated based on the first fused feature information, the third associated feature information and the second streaming audio segment. The action feature information generated by the first streaming audio segment here is obtained based on the action coding information corresponding to the multiple virtual characters.

In an optional implementation, the execution device inputs the adjusted first fusion feature information, the third associated feature information, and the action feature information generated by the first streaming audio segment into a multi-person action decoder respectively, and uses the adjusted first fusion feature information and the third associated feature information as offsets to adjust the action feature information generated by the first streaming audio segment to obtain the body movement information corresponding to at least one virtual character.

In this implementation, the execution device inputs the first fused feature information, the third associated feature information, and the action feature information generated by the first streaming audio clip into the multi-person action decoder, respectively, so that the generated body movement information can not only contain more details but also take into account the body movements of other virtual characters. Exemplarily, the multi-person action decoder is mainly composed of three Decoders. Please refer to Figure 9, which is a structural diagram of a multi-person action decoder provided in an embodiment of the present application. The multi-person action decoder outputs the body movement information of multiple virtual characters, that is, the joint point posture information of multiple virtual characters (including the position, rotation angle, speed and other information of the joint point). On the basis of generating the body movement information of multiple virtual characters, the body movement information corresponding to at least one virtual character can be displayed according to actual needs.

It is understandable that the multi-person action decoder can adopt neural networks such as MLP networks and convolutional networks, which can be set according to actual needs and are not limited here.

In another optional implementation, the execution device obtains tag information, and adjusts the body movement information corresponding to each virtual character based on the tag information to obtain the adjusted body movement information corresponding to each virtual character. The tag information is used to indicate the body movement information of the virtual character.

In this implementation, the execution device may use a stylization processing module to perform stylization processing on the generated virtual character's body movement information according to the tag information specified by the user, and generate body movement information of various styles to adapt to different types of virtual characters.

Optionally, the stylized processing module can adopt the ERD (Encoder-RNN-Decoder) architecture, and only retain the action content part of the body movement information of multiple virtual characters input through the Encoder network. Then, by combining the RNN network of the label information of a specific style with the output of the first RNN network, a motion sequence feature with a specific style is obtained, and finally the stylized body movement information is decoded using the Decoder network. The stylized processing module includes branches corresponding to multiple stylized labels, such as the elderly, young people, children, etc., and at least one first RNN network (used to represent neutral motion sequence features without style). The execution device obtains the body movement information with a specific style by adding the output of the RNN network branch of the specific stylized label to the output of the first RNN network branch. Among them, the specific structure diagram of the stylized processing module can refer to Figure 7.

It should be noted that there is no requirement for the order of step 514 to step 515 and step 516 to step 517.

For the convenience of explaining the process of generating the body movement information of the virtual character, please refer to FIG. 10 for example, which is a flow chart of generating the body movement information of the virtual character provided by the embodiment of the present application. Please refer to steps 511 to 517 for the specific execution steps.

It can be understood that the structural diagram shown in FIG. 10 is only an example structural description of one embodiment of the present application when generating the body movement information of a virtual character, and is not limited here.

(II) Single virtual character scene

In the embodiment of the present application, the reasoning stage describes the process of how the execution device 210 uses the target model/rule 201 to process the collected information data to generate a prediction result. Please refer to Figure 11 for details. Figure 11 is another flow chart of the method for displaying the performance content of a virtual character provided in the embodiment of the present application. The method may include steps 1101 to 1112.

Step 1101: The execution device obtains a first streaming audio segment.

Step 1102: The execution device extracts feature information of the first streaming audio segment from the first streaming audio segment.

Step 1103: The execution device obtains motion feature information generated by the second streaming audio segment.

Step 1104: The execution device obtains first fusion feature information based on the feature information of the first streaming audio segment and the action feature information generated according to the second streaming audio segment.

For the specific description of step 1101 to step 1104 , reference may be made to the description of step 501 to step 504 shown in FIG. 5 in the above embodiment, and will not be repeated here.

After executing steps 1101 to 1104, there may be multiple implementations based on different generated contents. The following takes the example of the execution device generating the facial expression information and body movement information of the virtual character respectively as an example to illustrate the following steps. It should be understood that in actual application, the execution device synchronously outputs the facial expression information and body movement information of the virtual character, and the execution device executes steps 1105-1107 and steps 1108-1112 in parallel.

(a) Generate facial expression information of virtual characters

Step 1105: The execution device obtains facial expression information generated by the second streaming audio segment.

In an embodiment of the present application, the execution device generates facial expression information corresponding to the first streaming audio segment by acquiring facial expression information generated by the second streaming audio segment.

Step 1106: The execution device obtains weight information of a basic expression corresponding to the virtual character based on the first fusion feature information, the phoneme feature information, and the facial expression information generated by the second streaming audio segment.

In the embodiment of the present application, compared with generating facial expression information corresponding to multiple virtual characters, there is no need to consider the associated feature information between the multiple virtual characters. Therefore, when generating weight information of the basic expression corresponding to the virtual character, the execution device directly inputs the facial expression information generated by the second streaming audio segment.

Step 1107: The execution device adjusts the basic expression of the virtual character based on the weight information of the basic expression corresponding to the virtual character to obtain facial expression information of the virtual character.

In an embodiment of the present application, the execution device adjusts the basic expression corresponding to the virtual character based on the weight information of the basic expression corresponding to the virtual character, so as to make detailed adjustments to the basic expression corresponding to the virtual character and obtain the facial expression information of the virtual character that matches the first streaming audio clip.

For the specific description of steps 1105 to 1107 , reference may be made to the description of steps 505 to 510 shown in FIG. 5 in the above embodiment, and will not be repeated here.

For the convenience of explaining the process of generating facial expression information of a virtual character, please refer to FIG. 12 for example, which is another flow chart of generating facial expression information of a virtual character provided by an embodiment of the present application. Please refer to steps 1105 to 1107 for specific execution steps.

It can be understood that the structural diagram shown in FIG. 12 is only an example structural description of one embodiment of the present application when generating facial expression information of a virtual character, and is not limited here.

(b) Generate virtual character’s body movement information

Step 1108: The execution device obtains motion feature information of the second streaming audio segment.

Step 1109: The execution device obtains action coding information corresponding to the virtual character based on the first fusion feature information and the second streaming audio segment action feature information.

In an embodiment of the present application, compared to generating action coding information corresponding to multiple virtual characters, the execution device directly inputs the first fusion feature information and the second streaming audio segment action feature information when generating action coding information corresponding to the virtual characters, without considering the associated feature information between the multiple virtual characters.

Step 1110: The execution device obtains action feature information corresponding to the action coding information of the virtual character based on the action coding information.

Step 1111: The execution device performs detail adjustment on the first fused feature information to obtain adjusted first fused feature information.

Step 1112: The execution device uses the adjusted first fusion feature information as an offset to adjust the action feature information generated by the first streaming audio segment to obtain body action information corresponding to the virtual character.

In the embodiment of the present application, considering that a single virtual character does not need to consider the associated feature information between multiple virtual characters, only the action feature information of a single virtual character is extracted when extracting the body motion information, and there is no need to extract the associated feature information.

In an optional implementation, the execution device inputs the adjusted first fusion feature information and the action feature information generated by the first streaming audio segment into a single-person action decoder respectively, and uses the adjusted first fusion feature information as an offset to adjust the action feature information generated by the first streaming audio segment to obtain the body movement information corresponding to a single virtual character.

In this implementation, there is no need to consider the associated feature information between multiple virtual characters. The single-player action decoder mainly consists of two The single-person action decoder outputs the body action information of a single virtual character, that is, the joint point posture information of a single virtual character (including the position, rotation angle, speed, etc. of the joint point).

It is understandable that the single-person action decoder can adopt neural networks such as MLP networks and convolutional networks, which can be set according to actual needs and are not limited here.

For the specific description of steps 1108 to 1112, reference may be made to the description of steps 511 to 517 shown in FIG. 5 in the above embodiment, and will not be repeated here.

For the convenience of explaining the process of generating the body movement information of the virtual character, please refer to FIG. 13 for example, which is another flow chart of generating the body movement information of the virtual character provided by the embodiment of the present application. Please refer to steps 1108 to 1112 for the specific execution steps.

It can be understood that the structural diagram shown in FIG. 13 is only an example structural description of one embodiment of the present application when generating the body movement information of a virtual character, and is not limited here.

2. Training Phase

In the embodiment of the present application, the training phase describes the process of how the training module 202 generates a mature neural network using the data set in the database 230. Specifically, please refer to FIG. 14, which is a flow chart of a model training method provided in the embodiment of the present application. The model training method provided in the embodiment of the present application may include:

Step 1401: The training device obtains a first streaming audio segment.

Step 1402: The training device obtains first fusion feature information based on the feature information of the first streaming audio segment and the action feature information generated according to the second streaming audio segment.

Step 1403: The training device generates facial expression information and body movement information of the virtual character based on the first fused feature information.

For the specific description of step 1401 to step 1403 , reference may be made to the description of step 501 to step 517 shown in FIG. 5 in the above embodiment, and will not be repeated here.

Step 1404: The training device obtains a target loss based on the facial expression information and body movement information of the virtual character and the real facial expression information and body movement information of the virtual character.

In the embodiment of the present application, the training device obtains a target loss based on the generated facial expression information of the virtual character and the real facial expression information, and the generated body movement information of the virtual character and the real body movement information. The target loss is used to indicate the difference between the facial expression information and the real facial expression information, and the body movement information and the real body movement information.

In an optional implementation, the training device may obtain the target loss based on the following method, including:

The training device obtains the relative positions, facial orientations and limb orientations of the virtual characters based on the limb motion information of the virtual characters;

The training device obtains a first loss based on the body movement information corresponding to each virtual character and the real body movement information;

The training device obtains a second loss based on the relative positions of the virtual characters and the real relative positions;

The training device obtains a third loss based on the facial orientation and body orientation of each virtual character and the real facial orientation and body orientation;

The training device obtains a target loss according to the first loss, the second loss, and the third loss.

In this implementation, the training device can obtain the target loss based on the body movement information of the virtual character, the relative positions between the virtual characters, and the facial orientation and body orientation of each virtual character.

It can be understood that the limb movement information corresponding to each virtual character refers to the limb movement information output by the target model, and the real limb movement information refers to the limb movement information actually collected through real-time motion capture and other methods.

Optionally, the training device obtains the first loss L ₁ based on the body movement information corresponding to each virtual character and the real body movement information, which is specifically:

Each virtual character learns the real body movement information based on the first loss, and realizes the coordination between characters in hand and body movements. For example, two virtual characters perform a heart-shaped gesture together, and by letting each virtual character learn its own action, the two virtual characters can coordinate in action and complete the heart-shaped gesture. The real body movement information can be obtained through dynamic capture.

Optionally, the training device obtains a formula for the second loss _L2 based on the relative positions of the virtual characters and the real relative positions:

Among them, i and j represent different virtual characters, and Δh _ij represent the relative position and the real relative position between virtual character i and virtual character j, respectively.

It is understandable that the relative position between virtual character i and virtual character j can be calculated by the limb motion information output by the target model, for example, the position information (x, y, z) of the joints contained in the limb motion information. The training device calculates the relative distance between the position information of the joints of virtual character i and the position information of the joints of virtual character j, that is, the relative position between virtual character i and virtual character j can be obtained. If it is a plane, it is calculated based on the plane coordinates (x, y). The real relative position between virtual character i and virtual character j can be collected by real-time motion capture and other methods. The training device realizes the cooperation of different virtual characters in the process of movement by adding a second loss _L2 , generates a motion trajectory with a collaborative effect, and realizes the alignment of the motion trajectory by constraining the relative position of any two virtual characters to be consistent with the real relative position.

Optionally, the training device obtains a formula for the third loss _L3 based on the facial orientation and limb orientation of each virtual character and the real facial orientation and limb orientation:

Among them, i and j represent different virtual characters, and Δdr _ij represent the facial orientation between virtual character i and virtual character j and the real facial orientation, respectively. and Δdf _ij represent the limb orientation between virtual character i and virtual character j and the real limb orientation, respectively.

It is understandable that the facial orientation and limb orientation between virtual character i and virtual character j can be calculated through the limb motion information output by the target model. For example, the rotation angle contained in the limb motion information, the training device can obtain the rotation angle of the joint point of virtual character i and the rotation angle of virtual character j. Among them, the limb orientation of the virtual character mainly refers to the orientation of the root node. The training device realizes the collaboration of different virtual characters in the process of movement by adding a third loss _L3 , produces the "cooperation effect of two virtual characters staring at each other", and learns the orientation of different characters on the face and root node to achieve orientation alignment. In order to achieve effective coordination of multiple virtual characters in dimensions such as limb motion, motion trajectory and body orientation.

Optionally, the training device obtains a target loss according to the first loss, the second loss, and the third loss, specifically:

The training device sets different weights r ₁ , r ₂ , r ₃ for the first loss L ₁ , the second loss L ₂ and the third loss L ₃ respectively. The larger the weight, the greater the proportion. The final target loss is obtained by multiplying the weight by the corresponding loss and adding them, i.e., r ₁ *L ₁ +r ₂ *L ₂ +r ₃ *L ₃ .

It is understandable that in actual operation, the calculation method of the target loss can be set according to actual needs. This is only an example and not a limitation.

Step 1405: The training device updates the parameters of the model to be trained based on the target loss until the model training conditions are met to obtain the target model.

On the basis of the embodiments corresponding to FIG. 1 to FIG. 14 , in order to better implement the above-mentioned scheme of the embodiment of the present application, the following also provides related devices for implementing the above-mentioned scheme. Please refer to FIG. 15 , which is a structural schematic diagram of a performance content display device for a virtual character provided in the embodiment of the present application. The performance content display device 1500 for a virtual character includes a streaming audio segment acquisition module 1501, a fusion feature information generation module 1502, and a performance content generation module 1503. Among them:

The streaming audio segment acquisition module 1501 is used to acquire a first streaming audio segment;

The fusion feature information generating module 1502 is used to obtain first fusion feature information based on feature information of a first streaming audio segment and action feature information generated according to a second streaming audio segment, wherein the first streaming audio segment and the second streaming audio segment are included in the same streaming audio, and a playback time of the second audio segment is before a playback time of the first audio segment;

The performance content generating module 1503 is used to generate facial expression information and body movement information of the virtual character based on the first fused feature information.

A possible implementation also includes:

The streaming audio segment acquisition module 1501 is further used to acquire a third streaming audio segment;

The fusion feature information generating module 1502 is further used to obtain second fusion feature information based on feature information of a third streaming audio segment and action feature information generated according to the first streaming audio segment, wherein the third streaming audio segment and the first streaming audio segment are included in the same streaming audio, and the playback time of the first streaming audio segment is before the playback time of the third streaming audio segment;

The performance content generating module 1503 is further used to generate facial expression information and body movement information of the virtual character based on the second fused feature information.

In a possible implementation, the feature information of the first streamed audio segment includes text feature information, and the performance content generation module 1503 is further configured to:

In a possible implementation, the performance content generating module 1503 is further used to:

In a possible implementation, obtaining motion feature information generated by the first streaming audio segment includes:

A possible implementation also includes:

Extract phonemes based on text features to obtain phoneme feature information;

It should be noted that the information interaction, execution process, etc. between the modules/units in the virtual character's performance content display device 1500 are based on the same concept as the various method embodiments corresponding to Figure 5 in the present application. The specific contents can be found in the description of the method embodiments shown in the previous part of the present application, and will not be repeated here.

The present application also provides a model training device. Please refer to FIG. 16 . FIG. 16 is a schematic diagram of a structure of a model training device provided in the present application. The model training device 1600 may include:

The streaming audio segment acquisition module 1601 is used to acquire a first streaming audio segment;

The fusion feature information generating module 1602 is used to obtain first fusion feature information based on feature information of the first streaming audio segment and action feature information generated according to the second streaming audio segment, wherein the first streaming audio segment and the second streaming audio segment are included in the same streaming audio, and the playing time of the second audio segment is before the playing time of the first audio segment;

A performance content generating module 1603, configured to generate facial expression information and body movement information of a virtual character based on the first fused feature information;

A target loss acquisition module 1604 is used to obtain a target loss based on the facial expression information and body movement information of the virtual character of the virtual character and the real facial expression information and body movement information of the virtual character, wherein the target loss is used to indicate the difference between the facial expression information and body movement information and the real facial expression information and body movement information;

The target model training module 1605 is used to update the parameters of the model to be trained based on the target loss until the model training conditions are met to obtain the target model.

A possible implementation also includes:

The streaming audio segment acquisition module 1601 is further used to acquire a third streaming audio segment;

The fusion feature information generating module 1602 is further used to obtain second fusion feature information based on feature information of a third streaming audio segment and action feature information generated according to the first streaming audio segment, wherein the third streaming audio segment and the first streaming audio segment are included in the same streaming audio, and the playback time of the first streaming audio segment is before the playback time of the third streaming audio segment;

The performance content generating module 1603 is further used to generate facial expression information and body movement information of the virtual character based on the second fused feature information.

In a possible implementation, the feature information of the first streamed audio segment includes text feature information, and the performance content generation module 1603 is further configured to:

In a possible implementation, the performance content generating module 1603 is further used to:

A possible implementation also includes:

Extract phonemes based on text features to obtain phoneme feature information;

It should be noted that the information interaction, execution process, etc. between the modules/units in the model training device 1600 are based on the same concept as the various method embodiments corresponding to Figure 14 in the present application. The specific contents can be found in the description of the method embodiments shown in the previous part of the present application, and will not be repeated here.

The present application also provides a computing device 1700. As shown in FIG. 17, FIG. 17 is a schematic diagram of a structure of a computing device provided in an embodiment of the present application, and the computing device 1700 includes: a bus 1702, a processor 1704, a memory 1706, and a communication interface 1708. The processor 1704, the memory 1706, and the communication interface 1708 communicate through the bus 1702. The computing device 1700 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 1700.

The bus 1702 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, FIG. 17 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 1704 may include a path for transmitting information between various components of the computing device 1700 (e.g., the memory 1706, the processor 1704, and the communication interface 1708).

Processor 1704 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

The memory 1706 may include a volatile memory, such as a random access memory (RAM). The processor 1704 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

The memory 1706 stores executable program codes, and the processor 1704 executes the executable program codes to respectively implement the functions of the aforementioned streaming audio segment acquisition module 1501, the fusion feature information generation module 1502, and the performance content generation module 1503, thereby implementing the performance content display method of the virtual character. That is, the memory 1706 stores instructions for executing the performance content display method of the virtual character.

The communication interface 1708 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1700 and other devices or communication networks.

The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

As shown in FIG18 , FIG18 is a schematic diagram of a structure of a computer device cluster provided in an embodiment of the present application, and the computing device cluster includes at least one computing device 1700. The memory 1706 in one or more computing devices 1700 in the computing device cluster may store the same instructions for executing the performance content display method of the virtual character.

In some possible implementations, the memory 1706 of one or more computing devices 1700 in the computing device cluster may also store partial instructions for executing the method for displaying the performance content of the virtual character. In other words, the combination of one or more computing devices 1700 may jointly execute the instructions for executing the method for displaying the performance content of the virtual character.

It should be noted that the memory 1706 in different computing devices 1700 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the performance content display device of the virtual character. That is, the instructions stored in the memory 1706 in different computing devices 1700 can realize the functions of one or more modules among the streaming audio segment acquisition module 1501, the fusion feature information generation module 1502 and the performance content generation module 1503.

In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network.

The embodiment of the present application also provides a training device, please refer to Figure 19, which is a structural diagram of a training device provided by the embodiment of the present application. Specifically, the training device 1900 is implemented by one or more servers, and the training device 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processor 1922 can be configured to communicate with the storage medium 1930 to execute a series of instruction operations in the storage medium 1930 on the training device 1900.

The training device 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, and/or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

In the embodiment of the present application, the central processing unit 1922 is used to execute the model training method executed by the training device in the embodiment corresponding to Figure 14. It should be noted that the specific manner in which the central processing unit 1922 executes the aforementioned steps is based on the same concept as the various method embodiments corresponding to Figure 17 in the present application, and the technical effects brought about are the same as the various method embodiments corresponding to Figure 14 in the present application. For specific contents, please refer to the description in the method embodiments shown in the preceding embodiment of the present application, which will not be repeated here.

The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the method of any of the above embodiments.

The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the method of any of the above embodiments.

The performance content display device, model training device, computing device, and training device for a virtual character provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, a pin, or a circuit, etc. The processing unit may execute computer execution instructions stored in the storage unit so that the chip executes the performance content display method for a virtual character described in the above-mentioned various method embodiments, or so that the chip executes the model training method described in the embodiment shown in FIG. 14 above. Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip within the wireless access device end, such as a read-only memory (read-only memory, ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), etc.

Specifically, please refer to FIG. 20 , which is a schematic diagram of a structure of a chip provided in an embodiment of the present application, wherein the chip may be a neural network processor NPU 2000, which is mounted on a host CPU (Host CPU) as a coprocessor and is assigned tasks by the Host CPU. The core part of the NPU is an operation circuit 2003, which is controlled by a controller 2004 to extract matrix data from a memory and perform multiplication operations.

In some implementations, the operation circuit 2003 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 2003 is a two-dimensional systolic array. The operation circuit 2003 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 2003 is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 2002 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 2001 and performs matrix operation with matrix B. The partial result or final result of the matrix is stored in the accumulator 2008.

The unified memory 2006 is used to store input data and output data. The weight data is directly transferred to the weight memory 2002 through the direct memory access controller (DMAC) 2005. The input data is also transferred to the unified memory 2006 through the DMAC.

BIU stands for Bus Interface Unit 2010, which is used for the interaction between AXI bus and DMAC and instruction fetch buffer (IFB) 2009.

The bus interface unit 2010 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 2009 to obtain instructions from the external memory, and is also used for the storage unit access controller 2005 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2006 or to transfer weight data to the weight memory 2002 or to transfer input data to the input memory 2001.

The vector calculation unit 2007 includes multiple operation processing units, which can further process the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional/fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

In some implementations, the vector calculation unit 2007 can store the processed output vector to the unified memory 2006. For example, the vector calculation unit 2007 can apply a linear function and/or a nonlinear function to the output of the operation circuit 2003, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values to generate an activation value. In some implementations, the vector calculation unit 2007 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2003, for example, for use in a subsequent layer in a neural network.

An instruction fetch buffer 2009 connected to the controller 2004, for storing instructions used by the controller 2004;

The unified memory 2006, the input memory 2001, the weight memory 2002 and the instruction fetch memory 2009 are all on-chip memories. The external memory is private to the NPU hardware architecture.

Among them, the operations of each layer in the above-mentioned target model can be performed by the operation circuit 2003 or the vector calculation unit 2007.

The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.

It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

Through the description of the above implementation methods, technicians in the relevant field can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware. Of course, it can also be implemented by special-purpose hardware including special-purpose integrated circuits, special-purpose CPUs, special-purpose memories, special-purpose components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially In other words, the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in the various embodiments of the present application.

In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

A method for displaying performance content of a virtual character, characterized by comprising:

Get the first streaming audio segment;

obtaining first fusion feature information based on feature information of the first streaming audio segment and action feature information generated according to the second streaming audio segment, wherein the first streaming audio segment and the second streaming audio segment are included in the same streaming audio, and a playback time of the second audio segment is before a playback time of the first audio segment;

Facial expression information and body movement information of the virtual character are generated based on the first fused feature information.
The method according to claim 1, further comprising:

Get the third streaming audio segment;

obtaining second fusion feature information based on feature information of the third streaming audio segment and action feature information generated according to the first streaming audio segment, wherein the third streaming audio segment and the first streaming audio segment are included in the same streaming audio, and a playback time of the first streaming audio segment is before a playback time of the third streaming audio segment;

Facial expression information and body movement information of the virtual character are generated based on the second fused feature information.
The method according to claim 1 or 2 is characterized in that the feature information of the first streaming audio segment includes text feature information, and the generating of the facial expression information of the virtual character based on the first fused feature information comprises:

Perform phoneme extraction based on the text feature information to obtain phoneme feature information;

Acquire first associated feature information on expressions of multiple virtual characters, where the first associated feature information is generated based on the second streaming audio segment;

Based on the first fusion feature information, the phoneme feature information and the first association feature information, obtaining weight information of a basic expression corresponding to at least one virtual character;

Based on the weight information of the basic expression corresponding to the at least one virtual character, the basic expression corresponding to the at least one virtual character is adjusted to obtain the facial expression information corresponding to the at least one virtual character.
The method according to claim 3, further comprising:

Extracting facial expression feature information corresponding to the plurality of virtual characters based on facial expression information corresponding to the plurality of virtual characters;

Based on the expression feature information corresponding to the multiple virtual characters, calculating the expression similarity between the multiple virtual characters;

The expression similarity between the multiple virtual characters is used as a weight to adjust the expression feature information corresponding to each virtual character to obtain second associated feature information on the expressions of the multiple virtual characters. The second associated feature information is used to generate facial expression information corresponding to at least one virtual character corresponding to the third streaming audio segment.
The method according to any one of claims 1 to 4, characterized in that the step of generating the virtual character's body movement information based on the first fused feature information comprises:

Performing detail adjustment on the first fused feature information to obtain adjusted first fused feature information;

Acquire third associated feature information on the actions of the plurality of virtual characters, wherein the third associated feature information is generated based on the second streaming audio segment;

Obtaining motion feature information generated by the first streaming audio segment;

The action feature information generated by the first streaming audio segment is adjusted using the adjusted first fusion feature information and the third associated feature information as offsets to obtain corresponding body motion information of the at least one virtual character.
The method according to claim 5, characterized in that the step of obtaining the motion feature information generated by the first streaming audio segment comprises:

Obtaining action coding information corresponding to a plurality of virtual characters based on the first fused feature information, the third associated feature information, and the action feature information generated by the second streaming audio segment;

Based on the action coding information, action feature information corresponding to the action coding information of the multiple virtual characters is obtained.
The method according to claim 5 or 6, characterized in that it also includes:

Based on the action feature information generated by the first streaming audio segment, calculating the action similarity between the multiple virtual characters;

The action feature information corresponding to each of the virtual characters is adjusted by taking the action similarity between the multiple virtual characters as a weight to obtain fourth associated feature information on the actions of the multiple virtual characters, and the fourth associated feature information is used to generate the The body movement information corresponding to the at least one virtual character corresponding to the third streaming audio segment.
The method according to any one of claims 5 to 7, further comprising:

Acquire tag information, where the tag information is used to indicate facial expression information and/or body movement information of the virtual character;

The facial expression information and/or body movement information corresponding to each virtual character is adjusted based on the tag information to obtain adjusted facial expression information and/or body movement information corresponding to each virtual character.
The method according to claim 1, characterized in that the feature information of the first streaming audio segment includes text feature information, and the generating of the facial expression information of the virtual character based on the first fused feature information comprises:

Extract phonemes based on the text features to obtain phoneme feature information;

Obtaining facial expression information generated by the second streaming audio segment;

Obtaining weight information of a basic expression corresponding to the virtual character based on the first fused feature information, the phoneme feature information, and the facial expression information generated by the second streaming audio segment;

Based on the weight information of the basic expression corresponding to the virtual character, the basic expression of the virtual character is adjusted to obtain the facial expression information of the virtual character.
The method according to claim 1 or 9, characterized in that generating the body movement information of the virtual character based on the first fusion feature information comprises:

Adjusting the first fused feature information to obtain adjusted first fused feature information;

Obtaining motion feature information generated by the first streaming audio segment;

The action feature information generated by the first streaming audio segment is adjusted using the adjusted first fusion feature as an offset to obtain the body movement information corresponding to the virtual character.
A device for displaying performance content of a virtual character, characterized by comprising:

A streaming audio segment acquisition module, used to acquire a first streaming audio segment;

a fusion feature information generating module, configured to obtain first fusion feature information based on feature information of the first streaming audio segment and action feature information generated according to the second streaming audio segment, wherein the first streaming audio segment and the second streaming audio segment are included in the same streaming audio, and a playback time of the second audio segment is before a playback time of the first audio segment;

A performance content generation module is used to generate facial expression information and body movement information of the virtual character based on the first fusion feature information.
The device according to claim 11, further comprising:

The streaming audio segment acquisition module is further used to acquire a third streaming audio segment;

The fusion feature information generating module is further used to obtain second fusion feature information based on feature information of the third streaming audio segment and action feature information generated according to the first streaming audio segment, wherein the third streaming audio segment and the first streaming audio segment are included in the same streaming audio, and the playback time of the first streaming audio segment is before the playback time of the third streaming audio segment;

The performance content generation module is further used to generate facial expression information and body movement information of the virtual character based on the second fused feature information.
The device according to claim 11 or 12, characterized in that the feature information of the first streaming audio segment includes text feature information, and the performance content generation module is further used to:

Perform phoneme extraction based on the text feature information to obtain phoneme feature information;

Acquire first associated feature information on expressions of multiple virtual characters, where the first associated feature information is generated based on the second streaming audio segment;

Based on the first fusion feature information, the phoneme feature information and the first association feature information, obtaining weight information of a basic expression corresponding to at least one virtual character;

Based on the weight information of the basic expression corresponding to the at least one virtual character, the basic expression corresponding to the at least one virtual character is adjusted to obtain the facial expression information corresponding to the at least one virtual character.
The device according to claim 13, characterized in that the performance content generation module is further used to:

Extracting facial expression feature information corresponding to the plurality of virtual characters based on facial expression information corresponding to the plurality of virtual characters;

Based on the expression feature information corresponding to the multiple virtual characters, calculating the expression similarity between the multiple virtual characters;

The expression feature information corresponding to each virtual character is adjusted by using the expression similarity between the multiple virtual characters as a weight to obtain second associated feature information on the expressions of the multiple virtual characters, and the second associated feature information is used to generate the third streaming audio The video clip may include facial expression information corresponding to at least one virtual character corresponding to the video clip.
The device according to any one of claims 11 to 14, characterized in that the performance content generation module is further used to:

Performing detail adjustment on the first fused feature information to obtain adjusted first fused feature information;

Acquire third associated feature information on the actions of the plurality of virtual characters, wherein the third associated feature information is generated based on the second streaming audio segment;

Obtaining motion feature information generated by the first streaming audio segment;

The action feature information generated by the first streaming audio segment is adjusted using the adjusted first fusion feature information and the third associated feature information as offsets to obtain corresponding body motion information of the at least one virtual character.
The device according to claim 15, wherein obtaining the motion feature information generated by the first streaming audio segment comprises:

Obtaining action coding information corresponding to a plurality of virtual characters based on the first fused feature information, the third associated feature information, and the action feature information generated by the second streaming audio segment;

Based on the action coding information, action feature information corresponding to the action coding information of the multiple virtual characters is obtained.
The device according to claim 15 or 16, characterized in that the performance content generation module is further used to:

Based on the action feature information generated by the first streaming audio segment, calculating the action similarity between the multiple virtual characters;

Using the action similarity between the multiple virtual characters as a weight, the corresponding action feature information of each virtual character is adjusted to obtain fourth associated feature information on the actions of the multiple virtual characters. The fourth associated feature information is used to generate body movement information corresponding to the at least one virtual character corresponding to the third streaming audio segment.
The device according to any one of claims 15 to 17, characterized in that it also includes:

Acquire tag information, where the tag information is used to indicate facial expression information and/or body movement information of the virtual character;

The facial expression information and/or body movement information corresponding to each virtual character is adjusted based on the tag information to obtain adjusted facial expression information and/or body movement information corresponding to each virtual character.
The device according to claim 11, characterized in that the feature information of the first streaming audio segment includes text feature information, and the performance content generation module is further used to:

Extract phonemes based on the text features to obtain phoneme feature information;

Obtaining facial expression information generated by the second streaming audio segment;

Obtaining weight information of a basic expression corresponding to the virtual character based on the first fused feature information, the phoneme feature information, and the facial expression information generated by the second streaming audio segment;

Based on the weight information of the basic expression corresponding to the virtual character, the basic expression of the virtual character is adjusted to obtain the facial expression information of the virtual character.
The device according to claim 11 or 19, characterized in that the performance content generation module is further used to:

Adjusting the first fused feature information to obtain adjusted first fused feature information;

Obtaining motion feature information generated by the first streaming audio segment;

The action feature information generated by the first streaming audio segment is adjusted using the adjusted first fusion feature as an offset to obtain the body movement information corresponding to the virtual character.
A computing device cluster, characterized in that it includes at least one computing device, each computing device includes a processor and a memory;

The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 10.
A computer program product comprising instructions, wherein when the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 10.
A computer-readable storage medium, characterized in that it includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method as described in any one of claims 1 to 10.