KR102663162B1 - Method and system for synthesizing speech - Google Patents

Abstract

The present disclosure relates to a speech synthesis method performed by at least one processor. The speech synthesis method includes receiving input text, generating a text representation from the input text using a text encoder, and generating a self-supervised representation containing linguistic information from the text representation using a self-supervised representation generator. It includes the steps of generating acoustic features based on the self-supervised representation using an acoustic feature generator, and generating synthesized speech based on the acoustic features using a speech generator.

Description

Voice synthesis method and system {METHOD AND SYSTEM FOR SYNTHESIZING SPEECH}

The present disclosure relates to a speech synthesis method and system, and specifically, to generate a self-supervised representation containing linguistic information from a text representation, and to generate acoustic features based on the self-supervised representation, thereby preventing linguistic information from being lost. Disclosed is a method and system for generating synthetic speech.

Voice synthesis technology is a technology used to reproduce the required voice in applications that require human voice, such as announcements, narration, navigation, artificial intelligence assistants, etc., without recording the actual human voice in advance. Recently, artificial neural network-based voice synthesis methods have been actively researched, and voices synthesized according to this method reflect much more natural voice characteristics than existing methods.

Most artificial neural network-based voice synthesis methods can generate acoustic features from text expressions and generate synthesized voices based on them. However, according to this conventional method, linguistic information is lost as acoustic features are generated directly from text expressions, and pronunciation may be muffled or excessively smooth synthesized speech may be generated. In other words, synthesized speech generated according to conventional methods may not sufficiently reflect the linguistic characteristics of the text.

The present disclosure provides a voice synthesis method, a computer program stored in a recording medium, and a device (system) to solve the above problems.

The present disclosure may be implemented in various ways, including as a method, device (system), or computer program stored in a readable storage medium.

According to an embodiment of the present disclosure, a method of speech synthesis, performed by at least one processor, includes receiving input text, generating a text representation from the input text using a text encoder, and using a self-supervised representation generator. Using a step of generating a self-supervised representation including linguistic information from the text expression, using an acoustic feature generator to generate an acoustic feature based on the self-supervised expression, and Using a speech generator to generate an acoustic feature and generating a synthetic voice based on the synthetic speech.

According to an embodiment of the present disclosure, a speech synthesis method performed by at least one processor includes receiving a source speech, source speech information, and target speech information, using an acoustic feature encoder, the source speech and the source speech. Generating source acoustic features based on the information, using an acoustic feature remover, generating a self-supervised representation including linguistic information based on the source acoustic features and source speech information, using an acoustic feature generator, It includes generating a target sound feature based on the self-map expression and the target voice information, and using a voice generator to generate a converted voice based on the target sound feature and the target voice information.

A computer program stored in a computer-readable recording medium is provided to execute the method according to an embodiment of the present disclosure on a computer.

An information processing system according to an embodiment of the present disclosure includes a memory and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, and the at least one program includes an input Receive text, generate a text representation from the input text using a text encoder, generate a self-supervised representation containing linguistic information from the text representation using a self-supervised representation generator, and use an acoustic feature generator to generate a text representation from the input text. , Includes commands for generating acoustic features based on self-supervised representation and generating synthesized speech based on acoustic features using a speech generator.

An information processing system according to an embodiment of the present disclosure includes a memory and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, and the at least one program is a source Receive voice, source voice information and target voice information, use an acoustic feature encoder to generate source acoustic features based on the source voice and source voice information, and use an acoustic feature remover to generate source acoustic features and source voice information. Generate a self-supervised representation based on, use an acoustic feature generator to generate a target acoustic feature based on the self-supervised expression and target voice information, and use a voice generator to generate a target acoustic feature based on the target acoustic feature and target voice information. Contains commands for generating converted voices.

According to some embodiments of the present disclosure, a two-step structure is adopted to generate a self-supervised representation including linguistic information from a text expression and generate acoustic features based on it, and the linguistic features and acoustic features are hierarchically generated. By learning, a synthetic voice with accurate pronunciation that sufficiently reflects linguistic characteristics can be generated.

According to some embodiments of the present disclosure, by using an acoustic feature remover learned to remove only acoustic features while maintaining the linguistic information of the voice and an acoustic feature generator in which the learned acoustic feature remover is inversely converted, the linguistic information of the source voice is A converted voice can be generated in which only the acoustic characteristics are converted while maintaining the same sound characteristics.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clear to a person skilled in the art (referred to as “a person skilled in the art”) in the technical field to which this disclosure pertains from the description of the claims. It will be understandable.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals indicate like elements, but are not limited thereto.
1 is a diagram illustrating an example of a method for generating synthesized speech according to an embodiment of the present disclosure.
Figure 2 is a schematic diagram showing a configuration in which an information processing system according to an embodiment of the present disclosure is connected to communicate with a plurality of user terminals.
Figure 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to an embodiment of the present disclosure.
FIG. 4 is a diagram illustrating an example of a method for learning a speech synthesis model according to an embodiment of the present disclosure.
FIG. 5 is a diagram illustrating an example of a method for generating a synthesized voice using a learned voice synthesis model according to an embodiment of the present disclosure.
Figure 6 is a diagram illustrating an example of a method for generating converted speech according to an embodiment of the present disclosure.
FIG. 7 is a diagram illustrating an example of a method for transfer learning a speech synthesis model using training speech for a new speaker according to an embodiment of the present disclosure.
Figure 8 is a diagram illustrating an example of evaluating the performance of a speech synthesis model according to an embodiment of the present disclosure.
Figure 9 is a flowchart showing an example of a voice synthesis method according to an embodiment of the present disclosure.
Figure 10 is a flowchart showing an example of a voice synthesis method according to an embodiment of the present disclosure.

Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, detailed descriptions of well-known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the gist of the present disclosure.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. Additionally, in the description of the following embodiments, overlapping descriptions of identical or corresponding components may be omitted. However, even if descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments and methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the present disclosure is complete and that the present disclosure does not convey the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but this may vary depending on the intention or precedent of a technician working in the related field, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Accordingly, the terms used in this disclosure should be defined based on the meaning of the term and the overall content of the present disclosure, rather than simply the name of the term.

In this specification, singular expressions include plural expressions, unless the context clearly specifies the singular. Additionally, plural expressions include singular expressions, unless the context clearly specifies plurality. When it is said that a certain part includes a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

Additionally, the term 'module' or 'unit' used in the specification refers to a software or hardware component, and the 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium and may be configured to run on one or more processors. Thus, as an example, a 'module' or 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. Components and 'modules' or 'parts' may be combined into smaller components and 'modules' or 'parts' or further components and 'modules' or 'parts'. Could be further separated.

According to an embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some contexts, 'processor' may refer to an application-specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), etc. 'Processor' refers to a combination of processing devices, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such combination of configurations. You may. Additionally, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), May also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. A memory is said to be in electronic communication with a processor if the processor can read information from and/or write information to the memory. The memory integrated into the processor is in electronic communication with the processor.

In the present disclosure, 'system' may include at least one of a server device and a cloud device, but is not limited thereto. For example, a system may consist of one or more server devices. As another example, a system may consist of one or more cloud devices. As another example, the system may be operated with a server device and a cloud device configured together.

In this disclosure, 'machine learning model' may include any model used to infer an answer to a given input. According to one embodiment, the machine learning model may include an artificial neural network model including an input layer (layer), a plurality of hidden layers, and an output layer. Here, each layer may include multiple nodes. In this disclosure, a plurality of machine learning models such as a self-supervised expression generator, an acoustic feature generator, a voice generator, and a text predictor are described as separate machine learning models, but the present disclosure is not limited thereto, and some or all of the plurality of machine learning models are one. It can be implemented as a machine learning model. Additionally, a model described as a single machine learning model may be implemented as multiple machine learning models. In this disclosure, a machine learning model may refer to an artificial neural network model, and an artificial neural network model may refer to a machine learning model.

In the present disclosure, 'each of a plurality of A' or 'each of a plurality of A' may refer to each of all components included in a plurality of A, or may refer to each of some components included in a plurality of A. .

FIG. 1 is a diagram illustrating an example of a method for generating a synthesized voice 160 according to an embodiment of the present disclosure. According to one embodiment, the information processing system may receive input text 110 and generate synthesized speech 160 based on the input text 110 using a speech synthesis model. For example, the input text 110 may be a set of unit texts included in the text. As a specific example, the input text 110 may be a character string containing a plurality of characters or a phoneme string containing a plurality of phonemes. Additionally, the synthesized voice 160 may be a voice that sounds as if the text is being uttered by a speaker.

According to one embodiment, the speech synthesis model may include a text encoder 120, a self-supervised representation generator 130, an acoustic feature generator 140, and a speech generator 150.

The text encoder 120 may receive the input text 110 as an input and generate a text representation including text information from the input text 110. For example, the text representation may be an embedding vector containing text information of the input text 110.

The self-supervised representation generator 130 may receive a text representation generated by the text encoder 120 as input and generate a self-supervised representation containing linguistic information from the text representation.

The acoustic feature generator 140 may receive the self-map representation generated by the self-map representation generator 130 as an input and generate an acoustic feature based on the self-map expression. In one embodiment, when the speech synthesis model is a multi-speaker model, the acoustic feature generator 140 additionally receives target speech information 170 and generates the target speech information 170 based on the self-supervised representation and target speech information 170. Acoustic features can be created.

The voice generator 150 may receive the acoustic features generated by the acoustic feature generator 140 as input and generate the synthesized voice 160 based on the acoustic features. In one embodiment, when the voice synthesis model is a multi-speaker model, the voice generator 150 additionally receives target voice information 170 and reflects the target voice information based on the acoustic characteristics and target voice information 170. A synthetic voice 160 can be generated.

According to the conventional end-to-end speech synthesis method, acoustic features are generated from text expressions, and synthesized speech is generated based on them. According to this conventional method, as acoustic features are generated directly from the text expression, linguistic information is lost, and the generated synthetic voice does not sufficiently reflect the linguistic features of the text, such as resulting in muffled pronunciation or overly smooth synthesized voice. There was a problem that it couldn't be done. According to an embodiment of the present disclosure, a two-step structure is adopted for generating a self-supervised representation including linguistic information from a text expression and generating acoustic features based on it, and the linguistic features and acoustic features are hierarchically configured. By learning, it is possible to generate a synthetic voice 160 with accurate pronunciation that sufficiently reflects linguistic characteristics.

The voice synthesis method according to the present disclosure can be performed by any performing entity, such as an information processing system or a user terminal. However, for convenience of explanation, the following description will be made on the assumption that the voice synthesis method is performed by an information processing system (eg, at least one processor of the information processing system).

Figure 2 is a schematic diagram showing a configuration in which the information processing system 230 according to an embodiment of the present disclosure is connected to communicate with a plurality of user terminals 210_1, 210_2, and 210_3. As shown, a plurality of user terminals 210_1, 210_2, and 210_3 may be connected to an information processing system 230 that can provide a voice synthesis service and/or a voice conversion service through the network 220. Here, the plurality of user terminals 210_1, 210_2, and 210_3 may include user terminals to be provided with a voice synthesis service and/or a voice conversion service. In one embodiment, information processing system 230 is one or more server devices capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to speech synthesis services and/or speech conversion services, etc. and/or a database, or one or more distributed computing devices and/or distributed databases based on cloud computing services.

The voice synthesis service and/or voice conversion service provided by the information processing system 230 is a voice synthesis application, voice conversion application, mobile browser application, or web browser installed on each of the plurality of user terminals 210_1, 210_2, and 210_3. It may be provided to the user through, etc. For example, the information processing system 230 may provide information corresponding to a request for synthesized voice for text received from the user terminals 210_1, 210_2, and 210_3 through a voice synthesis application or perform corresponding processing. .

A plurality of user terminals 210_1, 210_2, and 210_3 may communicate with the information processing system 230 through the network 220. The network 220 may be configured to enable communication between a plurality of user terminals 210_1, 210_2, and 210_3 and the information processing system 230. Depending on the installation environment, the network 220 may be, for example, a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a mobile communication network, a wireless LAN (WLAN), It may consist of wireless networks such as Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and may include communication methods utilizing communication networks that the network 220 may include (e.g., mobile communication networks, wired Internet, wireless Internet, broadcasting networks, satellite networks, etc.) as well as user terminals (210_1, 210_2, 210_3). ) may also include short-range wireless communication between

In Figure 2, the mobile phone terminal (210_1), tablet terminal (210_2), and PC terminal (210_3) are shown as examples of user terminals, but they are not limited thereto, and the user terminals (210_1, 210_2, 210_3) use wired and/or wireless communication. This is possible and may be any computing device on which a speech synthesis application, a voice conversion application, a mobile browser application, or a web browser, etc. can be installed and executed. For example, user terminals include AI speakers, smartphones, mobile phones, navigation, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, It may include wearable devices, IoT (internet of things) devices, VR (virtual reality) devices, AR (augmented reality) devices, set-top boxes, etc. In addition, in Figure 2, three user terminals (210_1, 210_2, 210_3) are shown as communicating with the information processing system 230 through the network 220, but this is not limited to this, and a different number of user terminals are connected to the network ( It may be configured to communicate with the information processing system 230 through 220).

According to one embodiment, the information processing system 230 may receive data associated with the input text from a plurality of user terminals 210_1, 210_2, and 210_3. Then, the information processing system 230 may generate a synthesized voice based on data associated with the received input text and provide the synthesized voice to a plurality of user terminals 210_1, 210_2, and 210_3. Additionally or alternatively, the information processing system 230 may receive data associated with the source voice from a plurality of user terminals 210_1, 210_2, and 210_3. Then, the information processing system 230 may generate a converted voice based on data associated with the received source voice and provide the converted voice to a plurality of user terminals 210_1, 210_2, and 210_3.

Figure 3 is a block diagram showing the internal configuration of the user terminal 210 and the information processing system 230 according to an embodiment of the present disclosure. The user terminal 210 may refer to any computing device capable of executing a voice synthesis application, a voice conversion application, a mobile browser application, or a web browser and capable of wired/wireless communication, for example, the mobile phone terminal of FIG. 2. It may include (210_1), a tablet terminal (210_2), a PC terminal (210_3), etc. As shown, the user terminal 210 may include a memory 312, a processor 314, a communication module 316, and an input/output interface 318. Similarly, information processing system 230 may include memory 332, processor 334, communication module 336, and input/output interface 338. As shown in FIG. 3, the user terminal 210 and the information processing system 230 are configured to communicate information and/or data through the network 220 using respective communication modules 316 and 336. It can be. Additionally, the input/output device 320 may be configured to input information and/or data to the user terminal 210 through the input/output interface 318 or to output information and/or data generated from the user terminal 210.

Memories 312 and 332 may include any non-transitory computer-readable recording medium. According to one embodiment, the memories 312 and 332 are non-permanent mass storage devices such as read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. It can be included. As another example, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the user terminal 210 or the information processing system 230 as a separate persistent storage device that is distinct from memory. Additionally, the memories 312 and 332 may store an operating system and at least one program code (for example, code for a voice synthesis application installed and running on the user terminal 210).

These software components may be loaded from a computer-readable recording medium separate from the memories 312 and 332. This separate computer-readable recording medium may include a recording medium directly connectable to the user terminal 210 and the information processing system 230, for example, a floppy drive, disk, tape, DVD/CD- It may include computer-readable recording media such as ROM drives and memory cards. As another example, software components may be loaded into the memories 312 and 332 through a communication module rather than a computer-readable recording medium. For example, at least one program is loaded into memory 312, 332 based on a computer program installed by files provided over the network 220 by developers or a file distribution system that distributes installation files for applications. It can be.

The processors 314 and 334 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processors 314 and 334 by memories 312 and 332 or communication modules 316 and 336. For example, processors 314 and 334 may be configured to execute received instructions according to program codes stored in recording devices such as memories 312 and 332.

The communication modules 316 and 336 may provide a configuration or function for the user terminal 210 and the information processing system 230 to communicate with each other through the network 220, and may provide a configuration or function for the user terminal 210 and/or information processing. The system 230 may provide a configuration or function for communicating with other user terminals or other systems (for example, a separate cloud system, etc.). In one example, a request or data generated by the processor 314 of the user terminal 210 according to a program code stored in a recording device such as memory 312 (e.g., a synthetic voice request for input text, a request for a source voice) converted voice request, etc.) may be transmitted to the information processing system 230 through the network 220 under the control of the communication module 316. Conversely, a control signal or command provided under the control of the processor 334 of the information processing system 230 is transmitted through the communication module 316 of the user terminal 210 through the communication module 336 and the network 220. It may be received by the user terminal 210. For example, the user terminal 210 may receive a synthesized voice for the input text or a converted voice for the source voice from the information processing system 230 through the communication module 316.

The input/output interface 318 may be a means for interfacing with the input/output device 320. As an example, input devices may include devices such as cameras, keyboards, microphones, mice, etc., including audio sensors and/or image sensors, and output devices may include devices such as displays, speakers, haptic feedback devices, etc. You can. As another example, the input/output interface 318 may be a means for interfacing with a device that has components or functions for performing input and output, such as a touch screen, integrated into one. For example, the processor 314 of the user terminal 210 uses information and/or data provided by the information processing system 230 or another user terminal when processing instructions of a computer program loaded in the memory 312. A service screen, etc. configured as such may be displayed on the display through the input/output interface 318. In FIG. 3 , the input/output device 320 is shown not to be included in the user terminal 210, but the present invention is not limited to this and may be configured as a single device with the user terminal 210. Additionally, the input/output interface 338 of the information processing system 230 may be connected to the information processing system 230 or means for interfacing with a device (not shown) for input or output that the information processing system 230 may include. It can be. In FIG. 3 , the input/output interfaces 318 and 338 are shown as elements configured separately from the processors 314 and 334, but the present invention is not limited thereto, and the input/output interfaces 318 and 338 may be configured to be included in the processors 314 and 334. there is.

The user terminal 210 and information processing system 230 may include more components than those in FIG. 3 . However, there is no need to clearly show most prior art components. According to one embodiment, the user terminal 210 may be implemented to include at least some of the input/output devices 320 described above. Additionally, the user terminal 210 may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. For example, if the user terminal 210 is a smartphone, it may include components generally included in a smartphone, such as an acceleration sensor, a gyro sensor, a camera module, various physical buttons, and a touch screen. Various components such as buttons using a panel, input/output ports, and a vibrator for vibration may be implemented to be further included in the user terminal 210. According to one embodiment, the processor 314 of the user terminal 210 may be configured to operate an application that provides a voice synthesis service. At this time, code associated with the corresponding application and/or program may be loaded into the memory 312 of the user terminal 210.

While a program for an application providing a voice synthesis service is running, the processor 314 uses input devices such as a touch screen, a keyboard, a camera including an audio sensor and/or an image sensor, and a microphone connected to the input/output interface 318. It is possible to receive text, images, videos, voices, and/or actions input or selected through, and store the received text, images, videos, voices, and/or actions in the memory 312 or use the communication module 316 and It can be provided to the information processing system 230 through the network 220. For example, processor 314 may receive input text. It can be provided to the information processing system 230 through the communication module 316 and the network 220. As another example, processor 314 may receive source speech and provide it to information processing system 230 via communication module 316 and network 220.

The processor 314 of the user terminal 210 manages, processes, and/or stores information and/or data received from the input device 320, other user terminals, the information processing system 230, and/or a plurality of external systems. It can be configured to do so. Information and/or data processed by processor 314 may be provided to information processing system 230 via communication module 316 and network 220. The processor 314 of the user terminal 210 may transmit information and/or data to the input/output device 320 through the input/output interface 318 and output the information. For example, the processor 314 may display the received information and/or data on the screen of the user terminal.

The processor 334 of the information processing system 230 may be configured to manage, process, and/or store information and/or data received from a plurality of user terminals 210 and/or a plurality of external systems. Information and/or data processed by the processor 334 may be provided to the user terminal 210 through the communication module 336 and the network 220. In one embodiment, the processor 334 of the information processing system 230 generates self-supervised representations and acoustic features representing linguistic information based on data associated with input text received from a plurality of user terminals 210. And based on this, a synthetic voice can be generated.

The processor 334 of the information processing system 230 uses the output device 320, such as a display output capable device (e.g., touch screen, display, etc.), an audio output capable device (e.g., speaker), of the user terminal 210. It may be configured to output processed information and/or data. For example, the processor 334 of the information processing system 230 may provide the generated synthetic speech (or data associated with the synthetic speech) to the user terminal 210 via the communication module 336 and the network 220. , the generated synthesized voice may be configured to output through a voice output capable device of the user terminal 210, etc. As another example, processor 334 of information processing system 230 provides converted speech (or data associated with converted speech) to user terminal 210 via communication module 336 and network 220, and It may be configured to output through a voice output capable device, etc. (210).

FIG. 4 is a diagram illustrating an example of a method for learning a speech synthesis model according to an embodiment of the present disclosure. Speech synthesis model (e.g., acoustic feature encoder 420, speech generator 422, acoustic feature remover 430, self-supervised learning model 440, language feature encoder 442, self-supervised representation remover 450) , text encoder 462, etc.) may be implemented as a machine learning model (e.g., artificial neural network model) and is based on the learning voice 410 and/or the learning text 460 corresponding to the learning voice 410. This can be learned. For example, a speech synthesis model may be trained to minimize the total loss, and the total loss may be calculated based on at least one of the first to fourth losses. For example, the first to fourth losses can be calculated as follows.

The information processing system can generate learning acoustic features based on the learning voice 410 using the acoustic feature encoder 420, and the learning sound generated by the acoustic feature encoder 420 using the voice generator 422. A learning synthetic voice 424 can be generated based on the features. The first loss may be calculated based on the training synthetic voice 424 and the training voice 410 generated by the voice generator 422. For example, the first loss may be a reconstruction loss calculated based on the learning synthetic voice 424 and the learning voice 410 (GT).

The information processing system may use the acoustic feature remover 430 to remove acoustic features from the learned acoustic features generated by the acoustic feature encoder 420 to generate the first learning self-supervised representation. Additionally, the information processing system may generate a second learning self-supervised representation from the training speech 410 using the self-supervised learning model 440 and the language feature encoder 442. The second loss may be calculated based on the first learned self-supervised representation generated by the acoustic feature remover 430 and the second learned self-supervised representation generated by the language feature encoder 442. For example, the second loss may be a KL loss (KL-Divergence loss) calculated based on the first learned self-supervised expression and the second learned self-supervised expression.

According to one embodiment, the self-supervised learning model 440 may include at least one layer (eg, middle layer of wav2vec 2.0) of a pre-trained speech model. Here, at least one layer of the pre-trained speech model may be a layer that generates output containing a lot of linguistic information. During the learning process of the speech synthesis model, at least one layer of the pre-trained speech model may be frozen.

The information processing system may use a self-supervised representation remover 450 to generate a first learned text representation by removing the self-supervised representation from a second learned self-supervised representation generated by the language feature encoder 442. Additionally, the information processing system may generate a second training text representation from training text 460 using text encoder 462. The third loss may be calculated based on the first training text representation generated by the self-supervised representation remover 450 and the second training text representation generated by the text encoder 462. For example, the third loss may be a KL loss calculated based on the first learning text representation and the second learning text representation.

The information processing system may use the text predictor 444 to predict the unit text 446 from the second learning self-supervised representation generated by the language feature encoder 442. Here, the unit text 446 may be a unit text (eg, a phoneme, letter, syllable, or word) predicted to be included in the learning text 460. The fourth loss may be calculated based on the unit text 446 and the training text 460 predicted by the text predictor 444. For example, the fourth loss is based on the phoneme predicted by the text predictor 444 and the phoneme (GT) included in the training text 460 to predict the phoneme predicted to be included in the training text 460. It may be the calculated phoneme prediction loss.

The total loss may be calculated based on at least one of the above-described first to fourth losses. As a specific example, the total loss may be calculated as a weighted sum of the first to fourth losses. A speech synthesis model can be trained to minimize total loss.

FIG. 5 is a diagram illustrating an example of a method for generating a synthesized voice 520 using a learned voice synthesis model according to an embodiment of the present disclosure. According to one embodiment, an information processing system uses a speech synthesis model trained as described above with reference to FIG. 4 and/or learned as described below with reference to FIG. 7 to generate a synthesized speech for input text 510. (520) can be generated. For example, an information processing system may use a text encoder 462 to generate a textual representation from input text 510 and a self-supervised representation generator 452 to generate a self-generated representation containing linguistic information from the textual representation. A map representation can be created. The information processing system may then use the acoustic feature generator 432 to generate acoustic features based on the self-supervised representation, and use the speech generator 422 to generate a synthesized voice 520 based on the acoustic features. can be created.

According to one embodiment, the acoustic feature remover 430 and the self-supervised representation remover 450 may be a network capable of inverse transformation (eg, Normalizing Flow, Denoising Diffusion Model, etc.). In this case, the acoustic feature generator 432 used to generate acoustic features, the self-supervised representation generator 452 used to generate the self-supervised representation, the learned acoustic feature remover 430, and the self-supervised representation remover 450 perform inverse transformation, respectively. It may be a model. For example, the acoustic feature remover 430 or the acoustic feature generator 432 can be used by using a single network capable of inverse transformation in a different operation direction. Similarly, the self-supervised representation remover 450 or the self-supervised representation generator 452 can be used by using a single network capable of inverse transformation in a different operation direction.

FIG. 6 is a diagram illustrating an example of a method for generating converted voice 660 according to an embodiment of the present disclosure. According to one embodiment, speech synthesis may include speech transformation that generates converted speech 660 based on source speech 610. For example, voice conversion refers to generating a converted voice 660 by keeping the linguistic information such as speech text and pronunciation of the source voice 610 the same and converting only the acoustic features (e.g., voice). You can.

According to one embodiment, the information processing system may receive source voice 610, source voice information 612, and target voice information 642. For example, source voice information 612 may be received separately from source voice 610 or extracted from source voice 610. Target voice information 642 may be received separately or extracted from a target voice sample.

The information processing system may generate converted voice 660 based on source voice 610 using a voice conversion model. The speech conversion model may include an acoustic feature encoder 620, an acoustic feature remover 630, an acoustic feature generator 640, and a speech generator 650. According to one embodiment, the speech conversion model may be trained as described above with reference to FIG. 4 and/or as described below with reference to FIG. 7 .

The acoustic feature encoder 620 may receive the source voice 610 and the source voice information 612 as input, and generate source acoustic features based on the source voice 610 and the source voice information 612. For example, the source acoustic feature may be an embedding vector containing acoustic information of the source voice 610.

The acoustic feature remover 630 receives the source acoustic features and source speech information 612 generated by the acoustic feature encoder 620 as input, and creates a self-map representation based on the source acoustic features and source speech information 612. can be created. For example, the acoustic feature remover 630 may use the source speech information 612 to remove acoustic features from the source acoustic features and generate a self-supervised representation that includes linguistic information.

The acoustic feature generator 640 receives the self-supervised expression and target speech information 642 generated by the acoustic feature remover 630 as input, and generates target acoustic features based on the self-supervised expression and target speech information 642. can be created. For example, the acoustic feature generator 640 may use the target voice information 642 to generate a target acoustic feature including the acoustic characteristics of the target voice from the self-map representation. According to one embodiment, the acoustic feature generator 640 may be a model obtained by inverting the learned acoustic feature remover 630.

The voice generator 650 receives the target acoustic feature and target voice information 642 generated by the acoustic feature generator 640 as input, and generates a converted voice 660 based on the target acoustic feature and target voice information. You can.

In this way, using the acoustic feature remover 630, which is learned to remove only acoustic features while maintaining the linguistic information of the voice, and the acoustic feature generator 640 in which the learned acoustic feature remover 630 is inversely converted, the source voice 610 ) It is possible to generate a converted voice 660 in which only the acoustic features are converted while maintaining the linguistic information.

FIG. 7 is a diagram illustrating an example of a method for transfer learning (fine-tuning) a speech synthesis model using training speech for a new speaker according to an embodiment of the present disclosure. Additionally, a speech synthesis model (e.g., speech generator 722, acoustic feature remover 730, etc.) may be transfer learned based on the training speech 710 for the new speaker. In other words, the speech synthesis model can be adapted to a new speaker using only voice samples, without a training text corresponding to the learning voice 710 for the new speaker.

For example, a speech synthesis model can be transfer learned to minimize the total loss, and the total loss can be calculated based on at least one of the first loss or the second loss. Here, the first loss or the second loss may be calculated the same or similar to that described with reference to FIG. 4. For example, the first loss and the second loss can be calculated as follows.

The information processing system may generate learning acoustic features based on the learning voice 710 using the acoustic feature encoder 720, and the learning sound generated by the acoustic feature encoder 720 using the voice generator 722. A learning synthetic voice 724 can be generated based on the features. The first loss may be calculated based on the training synthetic voice 724 and the training voice 710 generated by the voice generator 722. For example, the first loss may be a restoration loss calculated based on the training synthesis voice 724 and the training voice 710 (GT).

The information processing system may use the acoustic feature remover 730 to remove acoustic features from the learned acoustic features generated by the acoustic feature encoder 720 to generate the first learned self-supervised representation. Additionally, the information processing system may generate a second learning self-supervised representation from the training speech 710 using the self-supervised learning model 740 and the language feature encoder 742. The second loss may be calculated based on the first learned self-supervised representation generated by the acoustic feature remover 730 and the second learned self-supervised representation generated by the language feature encoder 742. For example, the second loss may be a KL loss calculated based on the first learned self-supervised expression and the second learned self-supervised expression.

In this way, the speech synthesis model can be adapted to a new speaker using only voice samples, without a training text corresponding to the learning voice 710 for the new speaker. The information processing system may use a speech synthesis model transfer-learned based on the learning voice 710 for the new speaker to generate a synthesized voice that reflects the voice characteristics of the new speaker based on the input text.

Figure 8 is a diagram illustrating an example of evaluating the performance of a speech synthesis model according to an embodiment of the present disclosure. FIG. 8 shows a table of results evaluating the performance of various speech synthesis models, including a speech synthesis model according to an embodiment of the present disclosure, using various performance evaluation indicators.

Table 1 (810) is a table showing the results of text-to-speech synthesis performance evaluation using the VCTK data set. Looking at the performance evaluation results regarding MOS (Mean Opinion Score) (nMOS (related to background noise) and sMOS (related to foreground quality)), the speech synthesis model of this method (shown as This Method) compared to other speech synthesis models. It can be confirmed that the performance of ) is the best. In addition, in terms of ASR (Automatic Speech Recognition) evaluation, the speech synthesis model of this method synthesizes speech with more accurate pronunciation and shows the lowest PER (Phone Error Rate) and WER (Word Error Rate) compared to other models. This can be confirmed. In addition, it can be confirmed that the speech synthesis model of this method has the lowest error distance in terms of MCD (Mel-Cepstral Distortion) and RMSE (Root Mean Squared Error).

Table 2 (820) shows the speech synthesis model of this method as a transfer-trained model based on the learning voice for a new speaker and the learning text corresponding to the learning voice (shown as Method 1, hereinafter the first model) and without the learning text. This table shows the results of evaluating the text-speech synthesis performance of a model (shown as Method 2, hereinafter referred to as the second model) that was transfer-trained based only on the learning voice for a new speaker. In terms of MOS (nMOS and sMOS) evaluation, it can be confirmed that the second model has similar performance to the first model. Additionally, it can be confirmed that the second model showed lower PER and WER than the first model. This shows that even if the model of this method is transfer-trained using only the training voice for a new speaker without any training text, the model can be well adapted to the new speaker.

Table 3 (830) is a table showing the results of voice conversion performance evaluation using the VCTK data set. It can be confirmed that the speech conversion model of this method (shown as This Method) has the best performance compared to other models in nMOS and ASR evaluation indices. This may mean that the speech conversion model of this method can separate linguistic information and acoustic information with as little loss of linguistic information as possible.

FIG. 9 is a flowchart illustrating an example of a voice synthesis method 900 according to an embodiment of the present disclosure. According to one embodiment, the method 900 may be initiated by a processor (eg, at least one processor of an information processing system and/or at least one processor of a user terminal) receiving input text (S910). If the speech synthesis model is a multi-speaker model, the processor can additionally receive target speech information.

The processor may then generate a text representation from the input text using a text encoder (S920) and generate a self-supervised representation containing linguistic information from the text representation using a self-supervised representation generator (S930). ). In one embodiment, the self-supervised representation generator may be a model in which the learned self-supervised representation remover is inverted.

Then, the processor may use the acoustic feature generator to generate acoustic features based on the self-map representation (S940). Additionally or alternatively, when the speech synthesis model is a multi-speaker model, the processor may use an acoustic feature generator to generate acoustic features based on the self-supervised representation and target speech information. According to one embodiment, the acoustic feature generator may be a model in which the learned acoustic feature remover is inversely transformed.

Thereafter, the processor may use a voice generator to generate a synthesized voice based on the acoustic characteristics (S950). Additionally or alternatively, when the speech synthesis model is a multi-speaker model, the processor may use a speech generator to generate a synthesized speech based on acoustic characteristics and target speech information.

According to one embodiment, a speech synthesis model (e.g., at least one of a self-supervised representation remover, an acoustic feature remover, or a speech generator) is configured to combine at least one of the first training speech or the first training text corresponding to the first training speech. It can be learned based on For example, a speech synthesis model may be trained to minimize the total loss, and the total loss may be calculated based on at least one of the first to fourth losses.

As a specific example, the first loss may be calculated based on the learned synthesized voice and the first learned voice, and the learned synthesized voice may be generated based on the first learned voice using an acoustic feature encoder and a voice generator. Additionally, the second loss may be calculated based on the first learned self-supervised expression and the second learned self-supervised expression. Here, the first learning self-supervised expression may be generated based on the first learning speech using an acoustic feature encoder and an acoustic feature remover, and the second learning self-supervised expression may be generated using a self-supervised learning model and a language feature encoder. It may be generated from the first learning voice.

The third loss may be calculated based on the first learning text representation and the second learning text representation. Here, the first learning text representation may be generated from the first learning speech using a self-supervised learning model, a language feature encoder, and a self-supervised representation remover, and the second learning text representation may be generated from the first learning text using a text encoder. can be created from Additionally, the fourth loss may be calculated based on the predicted unit text and the first training text, and the predicted unit text may be predicted from the first training speech using a self-supervised learning model, language feature encoder, and text predictor. You can. In one embodiment, the unit text predicted by the text predictor may include phonemes, letters, syllables, or words predicted to be included in the first training text.

According to one embodiment, the self-supervised learning model may include at least one layer of a pre-trained speech model, and at least one layer of the pre-trained speech model may be frozen during the learning process.

Additionally, speech synthesis models (e.g., speech generator and acoustic feature remover) may be transfer learned based on a second training speech for a new speaker. In this case, the processor may generate acoustic features based on the self-supervised representation using an acoustic feature generator in which an acoustic feature remover transfer-learned based on the second learning voice is inversely converted. Then, the processor may use a speech generator transfer-trained based on the second training speech to generate a synthetic speech that reflects the speech characteristics of the new speaker based on the acoustic characteristics.

FIG. 10 is a flowchart illustrating an example of a voice synthesis method 1000 according to an embodiment of the present disclosure. Method 1000 may be initiated by a processor (e.g., at least one processor of an information processing system and/or at least one processor of a user terminal) receiving a source voice, source voice information, and target voice information (S1010 ).

Then, the processor uses an acoustic feature encoder to generate source acoustic features based on the source speech and source speech information (S1020), and uses an acoustic feature remover to generate linguistic features based on the source acoustic features and source speech information. A self-map representation containing information can be created (S1030).

Then, the processor may use the acoustic feature generator to generate a target acoustic feature based on the self-map expression and target voice information (S1040). According to one embodiment, the acoustic feature generator may be a model in which the learned acoustic feature remover is inversely transformed. Afterwards, the processor may use a voice generator to generate a converted voice based on the target sound characteristics and target voice information (S1050).

According to one embodiment, a speech conversion model (eg, at least one of an acoustic feature remover or a speech generator) may be learned based on the third training speech. For example, a speech conversion model may be trained to minimize the total loss, and the total loss may be calculated based on at least one of the fifth loss or the sixth loss.

As a specific example, the fifth loss may be calculated based on the learned synthesized voice and the third learned voice, and the learned synthesized voice may be generated based on the third learned voice using an acoustic feature encoder and a voice generator. Additionally, the sixth loss may be calculated based on the first learned self-supervised expression and the second learned self-supervised expression. Here, the first learning self-supervised expression may be generated based on the third learning speech using an acoustic feature encoder and an acoustic feature remover, and the second learning self-supervised expression may be generated using a self-supervised learning model and a language feature encoder. It may be generated from a third learning voice.

According to one embodiment, the self-supervised learning model may include at least one layer of a pre-trained speech model. Here, at least one layer of the pre-trained speech model may be frozen during the learning process.

9, 10 and the above description are only one example of the present disclosure, and the scope of the present disclosure is not limited thereto. For example, according to another embodiment, at least some steps of the above-described method may be performed by a different performing entity, the order of each step may be changed, or at least some steps may be added/changed/omitted.

The above-described method may be provided as a computer program stored in a computer-readable recording medium for execution on a computer. Media may be used to continuously store executable programs on a computer, or may be temporarily stored for execution or download. Additionally, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and There may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

The methods, operations, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or It may be implemented or performed as any combination of those designed to perform the functions described in. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

For firmware and/or software implementations, techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and PROM ( on computer-readable media such as programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. It may also be implemented as stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

When implemented in software, the techniques may be stored on or transmitted through a computer-readable medium as one or more instructions or code. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or the desired program code in the form of instructions or data structures. It can be used to transfer or store data and can include any other media that can be accessed by a computer. Any connection is also properly termed a computer-readable medium.

For example, if the Software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, , fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. As used herein, disk and disk include CD, laser disk, optical disk, digital versatile disc (DVD), floppy disk, and Blu-ray disk, where disks are usually magnetic. It reproduces data optically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known. An exemplary storage medium may be coupled to the processor such that the processor may read information from or write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and storage media may reside within an ASIC. ASIC may exist within the user terminal. Alternatively, the processor and storage medium may exist as separate components in the user terminal.

Although the above-described embodiments have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the disclosure is not limited thereto and may also be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Furthermore, aspects of the subject matter of this disclosure may be implemented in multiple processing chips or devices, and storage may be similarly effected across the multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure as can be understood by a person skilled in the art to which the invention pertains. Additionally, such modifications and changes should be considered to fall within the scope of the claims appended hereto.

110: input text 120: text encoder
130: Self-supervised representation generator 140: Acoustic feature generator
150: voice generator 160: synthetic voice
170: Target voice information

Claims (20)

In a speech synthesis method performed by at least one processor,
Receiving input text;
generating a text representation from the input text using a text encoder;
generating a self-supervised representation including linguistic information from the text representation using a self-supervised representation generator;
Using an acoustic feature generator, generating acoustic features based on the self-map representation; and
Generating a synthesized voice based on the acoustic characteristics using a voice generator.
Including,
The acoustic feature generator is a model in which the learned acoustic feature remover is inverted,
The self-supervised representation generator is a model in which the learned self-supervised representation remover is inverted,
A speech synthesis method, wherein at least one of the self-supervised representation remover, the acoustic feature remover, or the speech generator is trained based on at least one of a first training voice or a first training text corresponding to the first training voice.
According to paragraph 1,
Step of receiving target voice information
It further includes,
The step of generating the acoustic feature is,
Using the acoustic feature generator, the acoustic feature is generated based on the self-map representation and the target voice information,
The step of generating the synthesized voice is,
Generating the synthesized voice based on the acoustic characteristics and the target voice information using the voice generator,
Voice synthesis method.
delete According to paragraph 1,
at least one of the self-supervised representation remover, the acoustic feature remover, or the speech generator is trained based on a first loss,
The first loss is calculated based on the learning synthetic voice and the first learning voice,
The learning synthesized voice is generated based on the first learning voice using an acoustic feature encoder and the voice generator.
According to paragraph 1,
at least one of the self-supervised representation remover, the acoustic feature remover, or the speech generator is trained based on a second loss,
The second loss is calculated based on the first learned self-supervised representation and the second learned self-supervised representation,
The first learning self-supervised representation is generated based on the first learning speech using an acoustic feature encoder and the acoustic feature remover,
The second learning self-supervised representation is generated from the first learning speech using a self-supervised learning model and a language feature encoder.
According to clause 5,
The self-supervised learning model includes at least one layer of a pre-trained speech model,
A speech synthesis method, wherein at least one layer of the pre-trained speech model is frozen during the learning process.
According to paragraph 1,
at least one of the self-supervised representation remover, the acoustic feature remover, or the speech generator is trained based on a third loss,
The third loss is calculated based on the first learning text representation and the second learning text representation,
the first trained text representation is generated from the first trained speech using a self-supervised learning model, a language feature encoder, and the self-supervised representation remover,
The second training text representation is generated from the first training text using the text encoder.
In clause 7,
The self-supervised learning model includes at least one layer of a pre-trained speech model,
A speech synthesis method, wherein at least one layer of the pre-trained speech model is frozen during the learning process.
According to paragraph 1,
at least one of the self-supervised representation remover, the acoustic feature remover, or the speech generator is learned based on a fourth loss,
The fourth loss is calculated based on the predicted unit text and the first learning text,
The predicted unit text is predicted from the first training speech using a self-supervised learning model, a language feature encoder, and a text predictor.
According to clause 9,
The predicted unit text includes a phoneme, character, syllable, or word predicted to be included in the first learning text.
According to paragraph 1,
The speech synthesis method wherein the speech generator and the acoustic feature remover are transfer learned (fine-tuned) based on a second training speech for a new speaker.
According to clause 11,
The step of generating the acoustic feature is,
Generating the acoustic features based on the self-supervised representation using an acoustic feature generator in which the acoustic feature remover transfer-learned based on the second learning voice is inversely converted,
The step of generating the synthesized voice is,
A voice synthesis method that generates a synthesized voice that reflects the voice characteristics of the new speaker based on the acoustic features, using a voice generator transfer-learned based on the second learning voice.
In a speech synthesis method performed by at least one processor,
Receiving source voice, source voice information, and target voice information;
Using an acoustic feature encoder, generating a source acoustic feature based on the source voice and the source voice information;
generating a self-supervised representation including linguistic information by removing acoustic features from the source acoustic features based on the source speech information using an acoustic feature remover;
Using an acoustic feature generator, generating a target acoustic feature based on the self-map representation and the target voice information; and
Using a voice generator, generating a converted voice based on the target acoustic feature and the target voice information.
A voice synthesis method including.
According to clause 13,
The acoustic feature generator is a model in which the learned acoustic feature remover is inverted,
A speech synthesis method, wherein at least one of the acoustic feature remover or the speech generator is learned based on a third training speech.
According to clause 14,
at least one of the acoustic feature remover or the speech generator is trained based on a fifth loss,
The fifth loss is calculated based on the learning synthetic voice and the third learning voice,
The learning synthesized voice is generated based on the third learning voice using the acoustic feature encoder and the voice generator.
According to clause 14,
at least one of the acoustic feature remover or the speech generator is trained based on a sixth loss,
The sixth loss is calculated based on the first learned self-supervised representation and the second learned self-supervised representation,
The first learning self-supervised representation is generated based on the third learning speech using the acoustic feature encoder and the acoustic feature remover,
The second learning self-supervised representation is generated from the third learning speech using a self-supervised learning model and a language feature encoder.
According to clause 16,
The self-supervised learning model includes at least one layer of a pre-trained speech model,
A speech synthesis method, wherein at least one layer of the pre-trained speech model is frozen during the learning process.
A computer program stored in a computer-readable recording medium for executing the method according to any one of claims 1, 2, and 4 to 17 on a computer.
As an information processing system,
Memory; and
At least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory
Including,
The at least one program is,
Receive input text,
Using a text encoder, generate a text representation from the input text,
Using a self-supervised expression generator, generate a self-supervised expression containing linguistic information from the text expression,
Using an acoustic feature generator, generate acoustic features based on the self-map representation,
Contains instructions for generating a synthesized voice based on the acoustic characteristics using a voice generator,
The acoustic feature generator is a model in which the learned acoustic feature remover is inverted,
The self-supervised representation generator is a model in which the learned self-supervised representation remover is inverted,
An information processing system, wherein at least one of the self-supervised representation remover, the acoustic feature remover, or the speech generator is a model learned based on at least one of a first training speech or a first training text corresponding to the first training speech. .
As an information processing system,
Memory; and
At least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory
Including,
The at least one program is,
Receive source voice, source voice information, and target voice information,
Using an acoustic feature encoder, generate a source acoustic feature based on the source voice and the source voice information,
Using an acoustic feature remover, generate a self-supervised representation including linguistic information by removing acoustic features from the source acoustic features based on the source speech information,
Using an acoustic feature generator, generate a target acoustic feature based on the self-map expression and the target voice information,
An information processing system comprising instructions for generating converted speech based on the target acoustic feature and the target speech information using a speech generator.

Images