CN118609536A - Audio generation method, device, equipment and storage medium - Google Patents

Abstract

The invention discloses an audio generation method, device, equipment and storage medium, and relates to the technical field of audio. The method comprises the following steps: responding to the received text information, acquiring historical audio information of a target user, and transmitting the text information by the target user through a target terminal; extracting tone features from the historical audio information; clustering the extracted tone features to determine tone features of a clustering center; determining the current tone characteristic of the target user according to the tone characteristic of the clustering center; the current timbre characteristics of the target user and the text information are input into a speech generation model to generate target audio having the current timbre characteristics of the target user. According to the technical scheme provided by the embodiment of the invention, personalized voice can be generated, so that user experience is improved.

Description

Audio generation method, device, equipment and storage medium

Technical Field

Embodiments of the present invention relate to the field of audio technology, and in particular to an audio generation method, apparatus, device and storage medium.

Background Art

In daily life and work, people often need to receive and send text messages. However, in certain specific scenarios, such as driving, sports or visual impairment, users may not be able to directly view or read text messages. In this case, if users can listen to the audio information corresponding to the text message content, it will greatly improve the convenience and security of information reception.

Although existing text-to-speech technologies can convert text into speech, they can usually only use preset or universal voice libraries to convert text into speech, which lacks personalization and authenticity, thus affecting the user experience.

Therefore, it is urgent to propose a new method to solve the above problems.

Summary of the invention

The present invention provides an audio generation method, device, equipment and storage medium, which can generate personalized voice, thereby improving user experience.

In a first aspect, an embodiment of the present invention provides an audio generation method, comprising:

In response to the received text message, acquiring historical audio information of a target user, wherein the text message is sent by the target user through a target terminal;

Extracting timbre features from the historical audio information;

Clustering the extracted timbre features to determine the timbre features of the cluster center;

Determine the current timbre feature of the target user according to the timbre feature of the cluster center;

The current timbre characteristics of the target user and the text information are input into a speech generation model to generate a target audio with the current timbre characteristics of the target user.

The technical solution of the present invention first responds to the received text information and obtains the historical audio information of the target user, the text information is sent by the target user through the target terminal; then the timbre features are extracted from the historical audio information; then the extracted timbre features are clustered to determine the timbre features of the cluster center; then the current timbre features of the target user are determined based on the timbre features of the cluster center; finally, the current timbre features of the target user and the text information are input into the speech generation model to generate the target audio with the current timbre features of the target user. The above technical solution provides a data basis for the subsequent extraction of timbre features by obtaining the historical audio information of the target user. Then the timbre features are extracted from the historical audio information, and the extracted timbre features are clustered, so that similar timbre features can be classified into one category to obtain the timbre features of the cluster center, which simplifies the data, thereby reducing the complexity of subsequent processing and improving work efficiency, and also provides a data basis for determining the current timbre features of the target user. Then, the current timbre features of the target user are determined by the representative features of the cluster, that is, the timbre features of the cluster center, so that the current timbre features of the target user finally determined can be more targeted and personalized, thereby providing the user with a more natural and realistic timbre experience and improving user satisfaction. Finally, the current timbre features and text information of the target user are input into the speech generation model to generate a target audio that meets the personalized timbre features of the target user, so that the target audio sounds like it is spoken by the target user himself, which greatly enhances the personalization and authenticity of the audio content, thereby improving the user experience. Compared with the prior art, although text can be converted into speech, usually only a preset or general voice library can be used to convert text into speech, which lacks personalization and authenticity, thus affecting the user experience. The present invention extracts timbre features from historical audio information, and clusters the extracted timbre features to obtain the timbre features of the cluster center; then determines the current timbre features of the target user based on the timbre features of the cluster center; finally, inputs the current timbre features of the target user and text information into a speech generation model to generate a target audio with the current timbre features of the target user, which can ensure that the generated audio is closer to the real voice of the target user in timbre, and brings a more natural and real timbre experience to the user. Therefore, the present invention can solve the problem that the audio generated by the existing text-to-speech technology lacks personalization and realism.

In a second aspect, an embodiment of the present invention further provides an audio generation device, the device comprising:

An acquisition module, configured to acquire historical audio information of a target user in response to a received text message, wherein the text message is sent by the target user via a target terminal;

An extraction module, used for extracting timbre features from the historical audio information;

A clustering module, used for clustering the extracted timbre features to determine the timbre features of the cluster center;

A determination module, used to determine the timbre characteristics of the target user according to the timbre characteristics of the cluster center;

A generation module is used to input the timbre characteristics of the target user and the text information into a speech generation model to generate a target audio with the timbre characteristics of the target user.

In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

at least one processor; and a memory communicatively coupled to the at least one processor;

The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can implement any audio generation method described in the first aspect.

In a fourth aspect, an embodiment of the present invention further provides a storage medium including computer executable instructions,

The computer executable instructions, when executed by a computer processor, implement any of the audio generation methods described in the first aspect.

It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium. The computer-readable storage medium may be packaged together with the processor of the audio generating device, or may be packaged separately from the processor of the audio generating device, and this application does not limit this.

The description of the second, third and fourth aspects in this application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third and fourth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

In this application, the name of the above-mentioned audio generating device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.

These and other aspects of the present application will become more apparent from the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG1 is a flow chart of an audio generation method provided by an embodiment of the present invention;

FIG2 is a flow chart of another audio generation method provided by an embodiment of the present invention;

FIG3 is a schematic diagram of the structure of an audio generating device provided by an embodiment of the present invention;

FIG. 4 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

The term "and/or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

In addition, the terms "including" and "having" and any variations thereof mentioned in the description of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

In the description of the present application, unless otherwise specified, “plurality” means two or more.

FIG1 is a flow chart of an audio generation method provided by an embodiment of the present invention. This embodiment is applicable to the case where a user receives a new text message and it is inconvenient for the user to operate the terminal by hand, wherein the terminal may be a smart phone or a tablet computer, etc. The method may be executed by an audio generation device, which may be implemented in software and/or hardware, and exemplarily, the device may be a computer. Referring to FIG1 , the audio generation method of this embodiment may specifically include the following steps:

Step 110: In response to the received text message, obtain historical audio information of the target user.

Specifically, the text message is sent by the target user through the target terminal. The target user is the user who sends the text message. The historical audio information of the target user refers to the audio information or records related to the target user stored in the device that receives the text message. The device that receives the text message can be a smart phone, a tablet computer or a laptop computer, etc., and the embodiment of the present invention does not limit this. For example, if user A sends a text message to user B's terminal B1 through terminal A1, the target user is A, the target terminal is A1, the user who receives the text message is B, and the device that receives the text message is B1.

In a specific implementation, before responding to the received text message, a monitoring mechanism needs to be set up to receive the text message. For example, the communication method to be used to receive the text message can be determined according to the actual situation or demand. The communication method can be a representational state transfer application programming interface, a network socket, a message queue, a real-time communication protocol, or a short message/mail service, etc. Then, according to the selected communication method, the corresponding monitoring service is deployed. If the selected communication method is a representational state transfer application programming interface, a Web service can be deployed; if the selected communication method is a network socket, a network socket service can be deployed; at the same time, in order to ensure the security of the monitoring service, a security mechanism can be configured, for example, the security mechanism can be identity authentication or authorization, etc. Then, the text message sent by the target user through the target terminal is received through the aforementioned established monitoring mechanism, and then the parsing algorithm can be determined according to the actual situation or demand, for example, the parsing algorithm can be a regular expression, an extensible markup language parser, or a template matching algorithm, etc. Then, the received text message is parsed according to the selected parsing algorithm to extract parameters related to obtaining the historical audio information of the target user, for example, the parameter can be an identifier of the target user. Then, to ensure that the extracted parameters are valid, the validity of the parameters can be verified, for example, verifying whether the user identifier exists. Then, the database is queried using the extracted parameters to obtain the historical audio information of the target user, for example, a database query statement is constructed using the extracted parameters, and the historical audio information of the target user is obtained from the database by executing the query statement. The obtained audio information can then be further processed according to business needs, for example, by storing or extracting timbre features.

In addition, before obtaining the historical audio information of the target user, in order to protect the user's privacy, it is necessary to ensure that the device currently performing the operation of obtaining the historical audio information of the target user has the authority to obtain the historical audio information of the target user.

In this embodiment, historical audio information of the target user is obtained to provide a data basis for subsequent extraction of timbre features.

Step 120: extract timbre features from historical audio information.

Specifically, just as different musical instruments have different pronunciation characteristics, different speakers also have different pronunciation characteristics, which will further make the sounds they produce full of recognition. That is, the speech signal contains characteristics that mark the speaker's pronunciation characteristics, which are called timbre features.

In a specific implementation, after obtaining the historical audio information of the target user, the historical audio information can be preprocessed, including but not limited to noise reduction, human voice separation, segmentation and other operations, so as to extract the human voice audio information. After preprocessing, a suitable timbre feature extractor can be selected according to the actual situation and application requirements to extract the timbre features. For example, an audio feature extractor can be used to extract audio features from the original audio, and then the timbre feature extractor and the timbre classifier can be jointly trained in combination with the timbre information. Finally, the audio feature extractor is used to extract the human voice audio features from the above human voice audio information, and then the trained timbre feature extractor is sent to obtain the timbre features in the historical audio information. When the computing power is tight, such as when deployed on the end-side device, audio feature extraction can use methods such as Fourier transform and Mel spectrum; when the computing power is sufficient, such as when deployed on the cloud-side device, audio feature extraction can use a pre-trained audio large model to further improve the effect of timbre feature extraction.

In addition, after the timbre features are extracted, the extracted timbre features can be post-processed, such as dimension reduction, normalization, etc., for subsequent use.

In this embodiment, the timbre features are extracted from the historical audio information.

Step 130: cluster the extracted timbre features to determine the timbre features at the cluster center.

Specifically, clustering is an unsupervised learning algorithm that groups or aggregates similar objects while making the objects in different groups as different as possible. In the present embodiment, clustering refers to aggregating similar timbre features among the extracted timbre features. The cluster center is a central point or representative point calculated by the clustering algorithm for each cluster or group. In cluster analysis, the cluster center is usually used to describe the overall characteristics or position of the cluster. In the present embodiment, the timbre features of the cluster center represent the typical or average characteristics of a certain type of timbre features.

In a specific implementation, after the timbre features are extracted, a suitable clustering algorithm can be selected according to the actual situation and application requirements to cluster the extracted timbre features, for example, the clustering algorithm can be a K-means clustering algorithm, hierarchical clustering, density-based noisy application space clustering, spectral clustering, etc. Then the selected clustering algorithm is applied to cluster the extracted timbre features. When the selected clustering algorithm is density-based noisy application space clustering, the extracted audio features can be first organized into a feature vector data set, each feature vector represents a timbre feature, and then the two parameters of the algorithm, i.e., the neighborhood size and the density threshold, are determined according to the actual situation or historical experience, and then for each point in the data set, the distance between it and all other points is calculated, that is, for each feature vector in the data set, the distance between it and all other feature vectors is calculated, for example, the calculated distance can be the Euclidean distance, and then the neighborhood of each point is determined according to the value of the aforementioned determined domain size, that is, the set of all points whose distance to the point is less than or equal to the domain size. Then traverse each point in the data set. If there are at least the density threshold points determined above in the neighborhood of the point, mark the point as a core point. If the point is not a core point, but its neighborhood contains at least one core point, mark the point as a boundary point. If the point is neither a core point nor a boundary point, mark it as a noise point. Starting from any unvisited core point, find all points that are reachable with its density, and these points form a cluster. Repeat this process until all core points have been visited and each cluster has been formed. Finally, the cluster center corresponding to each cluster can be represented by calculating the average eigenvector of all points in each cluster. The calculated average eigenvector is the timbre feature of the cluster center.

In this embodiment, the extracted timbre features are clustered so that similar timbre features can be classified into one category, that is, a large amount of timbre feature data can be simplified into several clustering centers, which greatly reduces the complexity of subsequent processing and improves work efficiency. It also provides a data basis for determining the current timbre features of the target user later.

Step 140: Determine the current timbre feature of the target user according to the timbre feature of the cluster center.

Specifically, the current timbre feature of the target user refers to a feature that can describe the timbre attribute of the target user, that is, it can represent the personalized or exclusive timbre of the target user.

In a specific implementation, if there are historical timbre features of the target user, the similarity or distance between the historical timbre features of the target user and the timbre features of each cluster center is calculated. Then, based on the calculated similarity or distance, the timbre features of the cluster center closest to the timbre features of the target user are found, and the timbre features are determined as the current candidate timbre features of the target user, and the timbre features of the target user are updated using the timbre features of the target user, and the updated historical timbre features are determined as the current timbre features of the target user. For example, the Euclidean distance between the historical timbre features of the target user and the timbre features of each cluster center can be calculated, and then the timbre features of the cluster center with the smallest distance are determined as the current candidate timbre features of the target user. If there are no historical timbre features of the target user, the current timbre features of the target user can be determined based on the number of cluster centers. When the number of cluster centers is one, the timbre features of the cluster center are directly determined as the current timbre features of the target user. When the number of cluster centers is not one, the timbre features of each cluster center can be prioritized. For example, the timbre features of each cluster center can be prioritized using multi-attribute decision analysis technology, and the timbre features of the cluster center with the highest priority can be determined as the current timbre features of the target user.

In this embodiment, the current timbre characteristics of the target user are determined by the representative characteristics of the cluster, that is, the timbre characteristics of the cluster center, so that the current timbre characteristics of the target user finally determined can be more targeted and personalized, thereby providing the user with a more natural and realistic timbre experience and improving user satisfaction.

Step 150: Input the current timbre characteristics and text information of the target user into a speech generation model to generate a target audio with the current timbre characteristics of the target user.

Specifically, the speech generation model is a machine learning model that can convert text information into speech signals. The target audio refers to the speech output generated by the speech generation model and has the current timbre characteristics of the target user. In this embodiment, the target audio sounds like the target user himself reading the text message he sent, which is highly personalized and natural.

In a specific implementation, before inputting the current timbre characteristics and text information of the target user into the speech generation model, a suitable speech generation model can be selected based on the actual situation or needs, and then the selected model can be trained using the data set of the text and its corresponding audio collected in advance, so as to obtain the speech generation model used in the end. After that, the current timbre characteristics and text information of the target user are input into the trained speech generation model, and the model will generate the corresponding audio waveform according to the input timbre characteristics and text information. After obtaining the audio, the generated audio can be post-processed, that is, the generated audio can be post-processed by noise elimination, volume equalization, dynamic compression, etc. to improve its quality and naturalness. Finally, the post-processed audio is played.

In this embodiment, by inputting the current timbre characteristics and text information of the target user into the speech generation model, a target audio that conforms to the personalized timbre characteristics of the target user can be generated, so that the target audio sounds as if it is spoken by the target user himself, greatly enhancing the personalization and authenticity of the audio content, thereby improving the user experience.

In the embodiment of the present invention, by obtaining the historical audio information of the target user, a data basis is provided for the subsequent extraction of timbre features. Then, the timbre features are extracted from the historical audio information, and the extracted timbre features are clustered, so that similar timbre features can be classified into one category to obtain the timbre features of the cluster center, simplifying the data, thereby reducing the complexity of subsequent processing, improving work efficiency, and also providing a data basis for determining the current timbre features of the target user later. Then, the current timbre features of the target user are determined by the representative features of the cluster, that is, the timbre features of the cluster center, so that the current timbre features of the target user finally determined can be more targeted and personalized, thereby providing the user with a more natural and real timbre experience and improving the user's satisfaction. Finally, the current timbre features and text information of the target user are input into the speech generation model to generate a target audio that meets the personalized timbre features of the target user, so that the target audio sounds like it is spoken by the target user himself, which greatly enhances the personalization and authenticity of the audio content, thereby improving the user experience. Compared with the prior art, although text can be converted into speech, usually only a preset or general voice library can be used to convert text into speech, which lacks personalization and authenticity, thereby affecting the user experience. The present invention obtains the historical audio information of the target user through the received text information, then extracts the timbre features from the historical audio information, and clusters the extracted timbre features to obtain the timbre features of the cluster center; then determines the current timbre features of the target user according to the timbre features of the cluster center; finally, inputs the current timbre features of the target user and the text information into the speech generation model to generate the target audio with the current timbre features of the target user, which can ensure that the generated audio is closer to the real voice of the target user in timbre, and brings a more natural and real timbre experience to the user. Therefore, the present invention can solve the problem that the audio generated by the existing text-to-speech technology lacks personalization and realism.

FIG2 is a flow chart of another audio generation method provided by an embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, the method may also include:

Step 211: Determine the user ID of the target user according to the received text information.

Specifically, the user identification refers to a mark or code used to uniquely identify the user. The user identification of the target user refers to a mark or code used to uniquely identify the target user.

In a specific implementation, the channel for receiving text information can be determined first, for example, the channel can be an instant messaging application or a chat window in a mobile application. Then, a listener or interceptor is set on the specified receiving channel to capture the text information sent by the target user, and then the text information is received through the specified channel. After that, the parsing algorithm can be determined according to the actual situation or needs, for example, the parsing algorithm can be a machine learning algorithm. Then, the received text information is parsed according to the selected parsing algorithm to extract the user ID of the target user. After the user ID of the target user is extracted, it can be verified and cleaned to ensure the accuracy and validity of the data. For example, the user ID can be verified by checking whether it conforms to the expected format and rules, and it can be cleaned by handling case sensitivity issues. The processed user ID of the target user is then stored for subsequent processing and use.

In this embodiment, the user identification of the target user is determined according to the received text information, which provides a data basis for obtaining the historical audio information of the target user later.

Step 212: Generate a request for obtaining the historical audio information of the target user based on the user identifier, and obtain the request result.

Specifically, the acquisition request is an instruction sent to the server to request access to or retrieval of a specific resource. In the present embodiment, the acquisition request is to request the target device to provide the historical audio information of the target user. The target device refers to the device that stores the historical audio information of the target user. For example, the target device can be a device that receives text messages or a server that stores information. The acquisition request may include a user ID, a time range, an audio type, etc. The request result is the server's response to the acquisition request. In the present embodiment, the request result is a response that agrees or refuses to obtain the historical audio information. If the request result is to agree to obtain, the historical audio information of the target user can be obtained based on the user ID of the target user.

In the specific implementation, in order to obtain the historical audio information of the target user, it is necessary to first determine a suitable interface, for example, the interface can be an application programming interface, and then use the client library of the hypertext transfer protocol or the secure hypertext transfer protocol to construct a request. For example, when using the client library of the hypertext transfer protocol to construct a request, first determine the request type, such as GET, POST, PUT, DELETE, etc., and then set the uniform resource location of the request, the uniform resource location includes the path parameter, and then set the request header, the request header includes the authentication information, content type, character encoding, etc. Then prepare the request body, the request body contains the data to be sent, such as the user ID of the target user, and finally call the function or method of the client library, pass in the uniform resource location, request method, request header and request body, and obtain the request. Then use the hypertext transfer protocol client library to send the constructed request to the target device, and then get a response returned by the target device, that is, the request result. Specifically, the specific request result can be determined according to the status code of the response. For example, when the response code is 200, it can represent that the request result is consent to obtain.

Step 213: When the request result is that the acquisition is approved, the historical audio information of the target user is acquired according to the user identifier.

In the specific implementation, if the request result is consent to obtain, the extracted user ID of the target user is used to construct a database query statement, and the historical audio information of the target user is obtained from the database storing the audio information by executing the query statement. After that, the obtained audio information can be further processed according to business needs, for example: storing or extracting timbre features.

On the other hand, when obtaining the historical audio information of the target user, in order to more effectively utilize computing resources and improve processing efficiency, timestamps can be used for efficient extraction. For example, each time the historical audio data of the target user is extracted, the end timestamp of the extracted audio data can be recorded. In the subsequent extraction process, this recorded timestamp can be used to locate the newly added audio data in the cache since the last extraction, and then only the newly added audio data is extracted. This can avoid repeated processing of the audio that has been analyzed, thereby saving computing power. The number of audio items extracted each time can also be limited for efficient extraction. For example, each time the historical audio data of the target user is extracted, only a certain number of the latest audio records are extracted, such as only the most recent 100 audio records. In this way, it can be ensured that the latest audio data is always paid attention to and processed, rather than processing outdated or no longer important data unnecessarily.

In this embodiment, in order to protect the privacy of the user, before obtaining the historical audio information of the target user, a request for obtaining the historical audio information of the target user is generated based on the user identifier, and a request result is obtained. If the request result is consent to obtain, the historical audio information of the target user is obtained according to the user identifier, ensuring that the device currently executing the operation of obtaining the historical audio information of the target user has the authority to obtain the historical audio information of the target user.

Step 214: filter out noise from the historical audio information to obtain initial human voice audio information.

Specifically, noise refers to unwanted sound elements in historical audio information, such as wind noise, traffic noise, mechanical noise of recording equipment, etc. The initial human voice audio information refers to the human voice part extracted from the historical audio information after filtering out the noise.

In a specific implementation, a noise reduction algorithm can be selected according to actual conditions or requirements. For example, the noise reduction algorithm can be a statistical filter, an adaptive filter, a deep learning algorithm, a spectral subtraction method, etc. Then, the selected noise reduction algorithm is applied to filter out noise from the historical audio information to obtain the initial human voice audio information. When the selected noise reduction algorithm is spectral subtraction, the audio signal in the historical audio information can be preprocessed first, and then the spectrum of the background noise can be obtained by performing a spectral analysis on the speech-free segment in the preprocessed audio signal. Then, the preprocessed audio signal is fast Fourier transformed to convert it from the time domain to the frequency domain to obtain the spectrum of the noisy speech. Then, the spectrum of the background noise obtained before is subtracted from the spectrum of the noisy speech to obtain an intermediate spectrum. Then, the intermediate spectrum is inversely fast Fourier transformed to convert it from the frequency domain back to the time domain to obtain the filtered audio signal, that is, the initial human voice audio information.

In this embodiment, initial human voice audio information is obtained by filtering out noise from historical audio information, which provides a data basis for obtaining target human voice audio information later.

Step 215: Separate and/or segment the initial human voice audio information according to the single person's voice to obtain the target human voice audio information.

Specifically, separation means that when there are multiple people speaking at the same time in an initial human voice audio information, the initial human voice audio information needs to be separated according to the single voice to obtain separated audio information, so as to ensure that each separated audio information has only one timbre feature. Segmentation means that when there are multiple people speaking successively in an initial human voice audio information, the initial human voice audio information needs to be segmented according to different human voices to obtain segmented audio information, so as to ensure that each segmented audio information has only one timbre feature. The target human voice audio information refers to the audio information obtained after separation and/or segmentation processing, in which there is only the sound of one person speaking, that is, there is only one timbre feature.

In a specific implementation, before separating and/or segmenting the initial human voice audio information according to a single person's voice, it is necessary to first determine whether there are multiple people speaking at the same time and/or multiple people speaking successively in the initial human voice audio information. If not, there is no need to separate and/or segment the initial human voice audio information according to a single person's voice. If not, the specific operation, i.e., separation and/or segmentation, is determined according to the specific type of existence, i.e., multiple people speaking at the same time and/or multiple people speaking successively. Specifically, it is possible to first determine whether there are multiple people speaking at the same time and/or multiple people speaking successively in each initial human voice audio information by using methods such as the energy variable of the audio signal, spectrum analysis technology or audio segmentation technology. For example, if the energy variable of the audio signal is used to determine whether there are multiple people speaking at the same time and/or multiple people speaking successively in each initial human voice audio information, then when the energy of the audio signal in an initial human voice audio information shows obvious fluctuations in time, and the fluctuation period matches the pronunciation period of the speech, it means that there are multiple people speaking successively in the audio information, and at this time, the audio information needs to be segmented. When the energy of the audio signal in an initial human voice audio information presents a continuous high energy state in time and there is no obvious fluctuation, it means that there are multiple people speaking at the same time in the audio information, and at this time, the audio information needs to be separated. When the energy of the audio signal in an initial human voice audio information is relatively stable in time and there is only one main energy peak, it means that there is only one person speaking in the audio information.

Then, each initial human voice audio information in which multiple people speak at the same time is separated according to the single person's voice, that is, the audio separation track technology can be used to separate the single person's voice for each initial human voice audio information. Specifically, in the audio information in which multiple people speak at the same time, the audio separation track technology can be used to identify and extract the track where the voice of each speaker is located, thereby separating it from other sounds and obtaining the target human voice audio information.

Then, the initial human voice audio information in which multiple people speak successively is segmented according to single-person speech, that is, the silent or low-energy areas in each initial human voice audio information can be identified respectively. These areas usually represent the intervals between different speakers. Then, the corresponding audio information is cut into multiple segments according to the silent or low-energy areas, so that each segment contains only one person's speech, thereby obtaining the target human voice audio information.

In this embodiment, the target human voice audio information is obtained by separating and/or segmenting the initial human voice audio information according to the single person's voice, which provides a data basis for subsequently extracting the timbre features of each target human voice audio information.

Step 216: extract the timbre features of each target human voice audio information.

In a specific implementation, after obtaining each target human voice audio information, the audio signal in each target human voice audio information can be preprocessed, including but not limited to noise reduction, human voice separation, segmentation and other operations, so as to extract the human voice audio information. After preprocessing, a suitable timbre feature extractor can be selected according to the actual situation and application requirements to extract timbre features. For example, an audio feature extractor can be used to extract audio features from the original audio, and then the timbre feature extractor and the timbre classifier can be jointly trained in combination with the timbre information. Finally, the audio feature extractor is used to extract the human voice audio features from the above human voice audio information, and then the trained timbre feature extractor is sent to obtain the timbre features in the historical audio information. When the computing power is tight, such as when deployed on the end-side device, audio feature extraction can use methods such as Fourier transform and Mel spectrum; when the computing power is sufficient, such as when deployed on the cloud-side device, audio feature extraction can use a pre-trained large audio model to further improve the effect of timbre feature extraction.

In this embodiment, by extracting the timbre features of each target human voice audio information, a data basis is provided for subsequently determining the current timbre features of the target user.

Step 217: cluster the extracted timbre features to determine the timbre features at the cluster center.

Furthermore, the extracted timbre features are clustered to determine the timbre features of the cluster center, including: clustering the extracted timbre features; and determining the average value of the timbre features in each cluster as the timbre features of the current cluster center.

In a specific implementation, after extracting the timbre features of each target human voice audio information, a suitable clustering algorithm can be selected to cluster it according to the actual situation and application requirements, for example, the clustering algorithm can be a K-means clustering algorithm, hierarchical clustering, density-based spatial clustering with noise, spectral clustering, etc. Then, the selected clustering algorithm is applied to cluster the extracted timbre features, and then the average value of the timbre features within each cluster is calculated respectively, and each average value obtained is determined as the timbre feature of the cluster center of the corresponding cluster.

In this embodiment, the extracted timbre features are clustered, and the average value of the timbre features in each cluster is determined as the timbre feature of the current cluster center, so that the overall characteristics of each cluster can be obtained more accurately, thereby improving the accuracy of timbre recognition.

Step 218: Determine whether there is a historical timbre feature of the target user.

If it exists, execute step 219; if it does not exist, execute step 220.

Specifically, the historical timbre features of the target user refer to the timbre features of the target user obtained in the past.

In a specific implementation, after obtaining the timbre features of the cluster center, it is determined whether there are historical timbre features of the target user. If so, the historical timbre features of the target user are matched with the extracted timbre features, or the historical timbre features of the target user are matched with the timbre features of each cluster center, and the current timbre features of the target user are determined according to the matching results; if not, the number of cluster centers is obtained to obtain the number of centers.

In one embodiment, if there are historical timbre features of the target user, the historical timbre features of the target user may be directly determined as the current timbre features of the target user.

In this embodiment, the needs in different scenarios are met by judging whether there are historical timbre features of the target user.

Step 219: Match the historical timbre features of the target user with the extracted timbre features, or match the historical timbre features of the target user with the timbre features of each cluster center, and determine the current timbre features of the target user according to the matching results.

In a specific implementation, after determining that there are historical timbre features of the target user, the historical timbre features of the target user can be matched with the extracted timbre features or the historical timbre features of the target user can be matched with the timbre features of each cluster center to obtain a corresponding matching result, that is, a suitable similarity measurement method can be selected according to the actual situation to perform matching, thereby obtaining a matching result. For example, if the historical timbre features of the target user are matched with the extracted timbre features, and the selected similarity measurement method is cosine similarity, then the degree of matching between the extracted timbre features and the historical timbre features of the target user is measured by calculating the cosine value between them. Specifically, for each extracted timbre feature, the similarity score between it and the historical timbre features of the target user can be calculated, and then the matching result can be determined according to the similarity score. For example, a similarity threshold can be set according to the actual situation. When the similarity score between the extracted timbre features and the historical timbre features of the target user is greater than or equal to the similarity threshold, the matching result can be determined to be a match, otherwise the matching result can be determined to be a mismatch. For each extracted timbre feature, the distance between it and the historical timbre features of the target user can also be calculated and compared with a preset threshold. The distance can be measured by cosine distance, Euclidean distance, etc., which are not limited here, as long as they are applicable to the distance between measuring vectors. If the distance is less than the threshold, the matching result can be determined to be a match, otherwise the matching result is determined to be a mismatch.

In this embodiment, the historical timbre features of the target user are matched with the extracted timbre features, or the historical timbre features of the target user are matched with the timbre features of each cluster center, and the current timbre features of the target user are determined based on the matching results. This can make the current timbre features of the determined target user more personalized, thereby providing users with more personalized services.

Furthermore, the current timbre features of the target user are determined based on the matching result, including: if the matching result is a mismatch, no operation is performed; if the matching result is a match, the historical timbre features of the target user are updated using the matched timbre features, and after completing the matching and updating of the timbre features, the historical timbre features of the target user are determined as the current timbre features of the target user.

In a specific implementation, after obtaining the matching result, the current timbre features of the target user can be determined according to the obtained matching result. Specifically, if the matching result is not a match, no operation is performed, that is, the historical timbre features of the target user can be directly determined as the current timbre features of the target user; if the matching result is a match, the timbre features of the target user can be updated by using the timbre features that match the historical timbre features of the target user among the extracted timbre features, or the timbre features of the timbre features of each cluster center that match the historical timbre features of the target user, and after completing the matching and updating of the timbre features, the historical timbre features of the target user are determined as the current timbre features of the target user. For example, the extracted timbre features or timbre features of each cluster center whose similarity scores with the historical timbre features of the target user are greater than or equal to the similarity threshold can be determined as matching timbre features. If the number of extracted timbre features or timbre features of each cluster center that are greater than or equal to the similarity threshold is greater than one, the timbre features with the highest similarity score among the extracted timbre features or timbre features of each cluster center that are greater than or equal to the similarity threshold can be determined as matching timbre features. Then, the historical timbre features are updated based on the matching timbre features, and the updated historical timbre features are determined as the current timbre features of the target user. The feature update method can be a moving average method, etc.

In this embodiment, through the above steps, various timbre feature matching situations can be handled more robustly. Even if mismatch occurs in some cases, the function can be kept running normally by selecting the historical timbre features, thereby avoiding erroneous operations caused by the inability to recognize the timbre. At the same time, this method reduces the need for users to manually adjust or confirm the timbre features, simplifying the user's operation process. During matching, by updating the historical timbre features, the timbre features are dynamically updated according to the actual pronunciation of the target user, avoiding sudden changes in timbre.

Step 220: Obtain the number of cluster centers to obtain the number of centers.

Specifically, the number of cluster centers refers to the number of cluster centers corresponding to each cluster obtained after clustering the extracted timbre features, that is, the number of centers.

In the specific implementation, the extracted timbre features are clustered using a clustering algorithm to obtain the timbre features of the cluster centers and the number of cluster centers, i.e., the number of centers, which is automatically calculated by the clustering algorithm according to the data distribution and the set clustering parameters during the clustering process.

Step 221: Determine whether the number of cluster centers is equal to one.

If they are equal, execute step 222; if they are not equal, execute step 223.

Step 222: Determine the timbre feature of the cluster center as the current timbre feature of the target user.

In the specific implementation, when the number of cluster centers is one, it means that all the extracted timbre features are tightly clustered around this center point during the clustering process, which means that these timbre features have high similarity, so the timbre features of the cluster center can be directly determined as the current timbre features of the target user.

In this embodiment, when the number of cluster centers is one, it means that these features form a stable and representative timbre category. Therefore, taking the timbre feature of this cluster center as the current timbre feature of the target user not only conforms to the actual data distribution, but also can accurately reflect the timbre characteristics of the target user.

Step 223: Determine whether there is a target cluster center.

If it exists, execute step 224; if it is not equal, execute step 225.

Specifically, the difference between the number of timbre features under the target cluster center and the number of timbre features under other cluster centers exceeds a preset quantity threshold. The preset quantity threshold refers to a boundary value set according to actual conditions or needs, which is used to determine whether the difference in the number of timbre features between different cluster centers is significant. For example: when the number of timbre features under each cluster center ranges from 100 to 500, the preset quantity threshold may be 200. The number of timbre features under a cluster center refers to the amount of data contained in each cluster after clustering, that is, the number of all timbre feature points belonging to the cluster.

In the specific implementation, when the number of cluster centers is greater than one, it means that the extracted timbre features form multiple obvious clusters in the timbre feature data set. Therefore, in order to determine the current timbre features of the target user, the number of timbre features under each cluster center can be calculated first, that is, the number of timbre feature points belonging to each cluster center can be counted to determine whether there is a target cluster center.

Step 224: Determine the timbre feature of the target cluster center as the current timbre feature of the target user.

In the specific implementation, after determining the existence of a target cluster center, it means that the number of timbre features under the target cluster center is significantly greater than that of other cluster centers, that is, the number of timbre features under the target cluster center is much larger than that of other cluster centers, indicating that these timbre features that occupy the majority are more likely to come from the target user. Then, the timbre features corresponding to the target cluster center with the largest number of timbre features can be determined as the current timbre features of the target user.

Step 225: determine the default timbre feature of the speech synthesis system as the current timbre feature of the target user.

In a specific implementation, after determining that there is no target cluster center, it means that the current timbre feature of the target user cannot be determined. At this time, the default timbre feature of the speech synthesis system is determined as the current timbre feature of the target user.

In this embodiment, when the number of cluster centers is greater than one, the number of timbre features under each cluster center is calculated to determine whether there is a target cluster center. If there is, it means that there is a timbre feature that is more consistent with the actual timbre of the target user, so the timbre feature of the target cluster center can be determined as the current timbre feature of the target user. If it does not exist, it means that the distribution of the timbre features is scattered and the current timbre feature of the target user cannot be determined, so it can be returned to the system default timbre feature for synthesis, avoiding the misunderstanding caused by using the wrong timbre feature for synthesis, and improving the user experience.

Step 226: Input the current timbre characteristics and text information of the target user into the speech generation model to generate target audio with the current timbre characteristics of the target user.

Therefore, the technical method of the present invention determines the user identification of the target user according to the received text information, and provides a data basis for obtaining the historical audio information of the target user. In order to protect the privacy of the user, before obtaining the historical audio information of the target user, a request for obtaining the historical audio information of the target user is generated based on the user identification, and the request result is obtained. When the request result is to agree to obtain, the historical audio information of the target user is obtained according to the user identification, which ensures that the device currently performing the operation of obtaining the historical audio information of the target user has the authority to obtain the historical audio information of the target user. Then, by filtering out noise from the historical audio information, the initial human voice audio information is obtained, which provides a data basis for obtaining the target human voice audio information. Then, the initial human voice audio information is separated and/or segmented according to the single voice to obtain the target human voice audio information, which provides a data basis for extracting the timbre features of each target human voice audio information. Then, by extracting the timbre features of each target human voice audio information, a data basis is provided for determining the current timbre features of the target user. Then, the extracted timbre features are clustered, and similar timbre features can be classified into one category to obtain the timbre features of the cluster center, which simplifies the data, thereby reducing the complexity of subsequent processing and improving work efficiency. Then, it is determined whether there are historical timbre features of the target user, so as to adapt to the needs in different scenarios. When the historical timbre features of the target user exist, the historical timbre features of the target user are matched with the extracted timbre features, or the historical timbre features of the target user are matched with the timbre features of each cluster center, and the current timbre features of the target user are determined according to the matching results, so that the current timbre features of the determined target user can be more personalized, thereby providing users with more personalized services. When the historical timbre features of the target user do not exist, the number of cluster centers is obtained to obtain the number of centers. It is determined whether the number of cluster centers is equal to one. When the number of cluster centers is one, the timbre features of the cluster center are determined as the current timbre features of the target user, accurately reflecting the timbre characteristics of the target user. When the number of cluster centers is greater than one, it is determined whether there is a target cluster center. If so, it means that there is a timbre feature that is more in line with the actual timbre of the target user, so the timbre features of the target cluster center can be determined as the current timbre features of the target user. If it does not exist, it means that the distribution of each timbre feature is scattered and the current timbre feature of the target user cannot be determined. Therefore, it can be reverted to the system default timbre feature for synthesis, avoiding the misunderstanding caused by using the wrong timbre feature for synthesis, and improving the user experience. Finally, the current timbre feature and text information of the target user are input into the speech generation model to generate a target audio that meets the personalized timbre feature of the target user, so that the target audio sounds like it is spoken by the target user himself, which greatly enhances the personalization and authenticity of the audio content, thereby improving the user experience. Compared with the prior art, although text can be converted into speech, usually only a preset or general voice library can be used to convert text into speech, which lacks personalization and authenticity, thereby affecting the user experience. The present invention obtains the historical audio information of the target user according to the received text information, and then extracts the timbre feature from the historical audio information, and then determines the current timbre feature of the target user according to the extracted timbre feature, and then inputs the current timbre feature and text information of the target user into the speech generation model to generate a target audio with the current timbre feature of the target user, which can ensure that the generated audio is closer to the real voice of the target user in timbre, and brings a more natural and real timbre experience to the user. Therefore, the present invention can solve the problem that the audio generated by the existing text-to-speech technology lacks personalization and realism.

In addition, after determining that the matching result is a match, the matching timbre features can be determined first, and then the historical timbre features can be updated based on the matching timbre features, and the updated historical timbre features can be determined as the current timbre features of the target user. This not only allows the timbre features to be dynamically updated according to the actual pronunciation of the target user, but also avoids sudden changes in timbre, thereby providing users with a more natural and realistic timbre experience and improving user satisfaction.

Figure 3 is a structural schematic diagram of an audio generating device provided in an embodiment of the present invention. The device and the audio generating methods of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the audio generating device, reference can be made to the embodiment of the above-mentioned audio generating method.

As shown in FIG3 , the device comprises:

The acquisition module 310 is used to acquire historical audio information of a target user in response to the received text information, wherein the text information is sent by the target user through a target terminal;

An extraction module 320, for extracting timbre features from the historical audio information;

A clustering module 330, for clustering the extracted timbre features to determine the timbre features of the cluster center;

A determination module 340, configured to determine the current timbre feature of the target user according to the timbre feature of the cluster center;

The generation module 350 is used to input the current timbre characteristics of the target user and the text information into a speech generation model to generate a target audio with the current timbre characteristics of the target user.

Based on the above embodiment, the acquisition module 310 is specifically used for:

Determining a user identifier of the target user according to the received text information;

Generate a request for obtaining the historical audio information of the target user based on the user identifier, and obtain a request result;

When the request result is that the acquisition is approved, the historical audio information of the target user is acquired according to the user identifier.

Based on the above embodiment, the extraction module 320 is specifically used for:

Filtering out noise from the historical audio information to obtain initial human voice audio information;

Separating and/or segmenting the initial human voice audio information according to the individual voices to obtain target human voice audio information;

The timbre features of each target human voice audio information are extracted.

Based on the above embodiment, the clustering module 330 is specifically used for:

Clustering the extracted timbre features;

The average value of the timbre features in each cluster is determined as the timbre feature of the current cluster center.

Based on the above embodiment, the device further includes:

A judgment module is used to judge whether there is a historical timbre feature of the target user before determining the current timbre feature of the target user based on the timbre feature of the cluster center; if the historical timbre feature of the target user does not exist, triggering the step of determining the current timbre feature of the target user based on the timbre feature of the cluster center.

Based on the above embodiment, the device further includes:

A matching module is used to match the historical timbre features of the target user with the extracted timbre features after determining whether the historical timbre features of the target user exist, or to match the historical timbre features of the target user with the timbre features of each cluster center, and determine the current timbre features of the target user according to the matching results.

Based on the above embodiment, the matching module determines the current timbre characteristics of the target user according to the matching result, including:

If the matching result is no match, no operation is performed;

If the matching result is a match, the historical timbre feature of the target user is updated using the matched timbre feature. After the timbre feature is matched and updated, the historical timbre feature of the target user is determined as the current timbre feature of the target user.

Based on the above embodiment, the determination module 340 is specifically used for:

Obtaining the number of cluster centers to obtain the number of centers;

When the number of centers is equal to one, determining the timbre feature of the cluster center as the current timbre feature of the target user;

When the number of centers is greater than one, determine whether there is a target cluster center, and the difference between the number of timbre features under the target cluster center and the number of timbre features under other cluster centers exceeds a preset quantity threshold; when the target cluster center exists, determine the timbre features of the target cluster center as the current timbre features of the target user; otherwise, the current timbre features of the target user cannot be determined, and the default timbre features of the speech synthesis system are determined as the current timbre features of the target user.

The audio generating device provided in the embodiment of the present invention can execute the audio generating method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

It is worth noting that in the embodiment of the above-mentioned audio generating device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

FIG4 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. FIG4 shows a block diagram of an exemplary electronic device 4 suitable for implementing an embodiment of the present invention. The electronic device 4 shown in FIG4 is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present invention.

As shown in Fig. 4, electronic device 4 is in the form of a general purpose computing electronic device. Components of electronic device 4 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

The electronic device 4 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 4, including volatile and non-volatile media, removable and non-removable media.

The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32. The electronic device 4 may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media (not shown in FIG. 4 , commonly referred to as a “hard drive”). Although not shown in FIG. 4 , a disk drive for reading and writing to a removable non-volatile disk (e.g., a “floppy disk”) and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data media interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.

A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the embodiments described herein.

The electronic device 4 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 4, and/or communicate with any device that enables the electronic device 4 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input/output (I/O) interface 22. Furthermore, the electronic device 4 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and/or public networks, such as the Internet) via a network adapter 20. As shown in FIG. 4 , the network adapter 20 communicates with other modules of the electronic device 4 via a bus 18. It should be understood that, although not shown in FIG. 4 , other hardware and/or software modules may be used in conjunction with the electronic device 4, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

The processing unit 16 executes various functional applications and page displays by running the program stored in the system memory 28, for example, implementing the audio generation method provided in the embodiment of the present invention, which includes:

In response to the received text message, acquiring historical audio information of a target user, wherein the text message is sent by the target user through a target terminal;

Extracting timbre features from the historical audio information;

Clustering the extracted timbre features to determine the timbre features of the cluster center;

Determine the current timbre feature of the target user according to the timbre feature of the cluster center;

The current timbre characteristics of the target user and the text information are input into a speech generation model to generate a target audio with the current timbre characteristics of the target user.

Of course, those skilled in the art can understand that the processor can also implement the technical solution of the audio generation method provided by any embodiment of the present invention.

An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, for example, an audio generation method provided by an embodiment of the present invention is implemented. The method includes:

In response to the received text message, acquiring historical audio information of a target user, wherein the text message is sent by the target user through a target terminal;

Extracting timbre features from the historical audio information;

Clustering the extracted timbre features to determine the timbre features of the cluster center;

Determine the current timbre feature of the target user according to the timbre feature of the cluster center;

The current timbre characteristics of the target user and the text information are input into a speech generation model to generate a target audio with the current timbre characteristics of the target user.

The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

It should be understood by those skilled in the art that the above modules or steps of the present invention can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

In addition, the acquisition, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of national laws and regulations.

Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (11)

1. A method of audio generation, the method comprising the steps of:

responding to received text information, acquiring historical audio information of a target user, wherein the text information is sent by the target user through a target terminal;

extracting tone features from the historical audio information;

clustering the extracted tone features to determine tone features of a clustering center;

determining the current tone characteristic of the target user according to the tone characteristic of the clustering center;

the current tone color characteristics of the target user and the text information are input into a speech generation model to generate target audio with the current tone color characteristics of the target user.

2. The audio generation method according to claim 1, wherein acquiring the historical audio information of the target user in response to the received text information, comprises:

determining a user identification of the target user according to the received text information;

Generating an acquisition request of the historical audio information of the target user based on the user identification, and obtaining a request result;

and under the condition that the request result is the consent to be acquired, acquiring the historical audio information of the target user according to the user identification.

3. The audio generation method of claim 1, wherein extracting timbre features from the historical audio information comprises:

filtering noise from the historical audio information to obtain initial voice audio information;

Separating and/or segmenting the initial voice audio information according to single voice to obtain target voice audio information;

and extracting tone characteristics of each target voice audio information.

4. The audio generation method of claim 1, wherein clustering the extracted timbre features to determine timbre features of a cluster center comprises:

Clustering the extracted tone features;

and determining the average value of the tone color features in each cluster as the tone color feature of the current cluster center.

5. The audio generation method of claim 1, further comprising, before determining the current timbre characteristic of the target user from the timbre characteristic of the cluster center:

judging whether the historical tone characteristics of the target user exist or not;

And if the historical tone characteristics of the target user do not exist, triggering and executing the step of determining the current tone characteristics of the target user according to the tone characteristics of the clustering center.

6. The audio generation method of claim 5, further comprising, after determining whether there is a history of timbre characteristics of the target user:

If the historical tone characteristics of the target user exist, matching the historical tone characteristics of the target user with the extracted tone characteristics, or matching the historical tone characteristics of the target user with the tone characteristics of each clustering center, and determining the current tone characteristics of the target user according to the matching result.

7. The audio generation method of claim 6, wherein determining the current timbre characteristic of the target user based on the matching result comprises:

if the matching result is not matching, not performing operation;

and if the matching result is that the target user is matched, updating the historical tone characteristics of the target user by using the matched tone characteristics, and determining the historical tone characteristics of the target user as the current tone characteristics of the target user after the matching and updating of the tone characteristics are completed.

8. The audio generation method of claim 1, wherein determining the current timbre characteristic of the target user from the timbre characteristic of the cluster center comprises:

Acquiring the number of the clustering centers to obtain the number of the centers;

when the number of the centers is equal to one, determining the tone characteristic of the clustering center as the current tone characteristic of the target user;

When the number of the centers is greater than one, judging whether a target clustering center exists, wherein the difference value between the number of tone color features under the target clustering center and the number of tone color features under other clustering centers exceeds a preset number threshold; when the target clustering center exists, determining that the tone characteristic of the target clustering center is the current tone characteristic of the target user; otherwise, the current tone characteristic of the target user cannot be determined, and the default tone characteristic of the voice synthesis system is determined as the current tone characteristic of the target user.

9. An audio generating apparatus, comprising:

The acquisition module is used for responding to the received text information and acquiring the historical audio information of the target user, wherein the text information is sent by the target user through the target terminal;

The extraction module is used for extracting tone characteristics from the historical audio information;

the clustering module is used for clustering the extracted tone color features to determine tone color features of a clustering center;

the determining module is used for determining the current tone characteristic of the target user according to the tone characteristic of the clustering center;

And the generation module is used for inputting the current tone color characteristics of the target user and the text information into a voice generation model so as to generate target audio with the current tone color characteristics of the target user.

10. An electronic device, the electronic device comprising:

At least one processor; and a memory communicatively coupled to the at least one processor;

wherein the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the audio generation method of any one of claims 1-8.

11. A storage medium containing computer executable instructions which when executed by a computer processor implement performing the audio generation method of any of claims 1-8.