CN104217713A - Tibetan-Chinese speech synthesis method and device - Google Patents

Abstract

The present invention provides a Chinese-Tibetan bilingual speech synthesis method and device, which are used to synthesize the input Chinese or Tibetan sentences to be synthesized by using the pre-established Chinese and Tibetan mixed corpus. Using the method and device of the present invention, Can simultaneously synthesize Chinese or Tibetan speech. Compared with the traditional HMM-based speech synthesis system, this system adds a speaker adaptive training process in the training stage to obtain the average tone model of Chinese-Tibetan mixed speech. Through this process, the speaker's difference in the speech database can be reduced. The impact is to improve the quality of synthesized speech. On the basis of the average sound model, through the speaker adaptive transformation algorithm, only a small amount of Tibetan or Chinese corpus to be synthesized can be synthesized with good naturalness and fluency. Tibetan or Chinese pronunciation. The research of this system is of great significance to promoting communication with ethnic minorities and promoting the development of ethnic minority voice technology.

Description

Chinese-Tibetan bilingual speech synthesis method and device

technical field

The present invention relates to the technical field of multilingual speech synthesis, and more specifically, provides a Chinese-Tibetan cross-language bilingual speech synthesis method and device.

Background technique

In recent years, multilingual speech synthesis technology has become a research hotspot in the field of human-computer speech interaction. Using this technology can realize human-computer voice interaction in different languages in the same system, which has important application value for countries or regions that speak several languages. China is a country with a large number of minority languages and dialects, which makes the research of this technology important. For example, in Tibetan areas in my country, Mandarin, Tibetan and dialects are mainly spoken. A variety of speech synthesis will be of great significance to promoting communication with ethnic minorities and promoting the development of ethnic minority voice technology.

The techniques of multilingual speech synthesis research at home and abroad mainly include primitive selection and splicing synthesis methods and statistical parameter speech synthesis methods. The basic principle of the waveform splicing synthesis method is to analyze the input text to obtain the basic unit information, then select the appropriate unit from the pre-recorded and marked voice library, make a small amount of adjustment, and then splicing to finally obtain the synthesized voice. Since the unit phonemes of the final synthesized speech are directly extracted from the sound bank, it can maintain the sound quality of the original speaker. However, the waveform splicing synthesis system generally requires a large-scale speech database. The workload of corpus production is very large, time-consuming and labor-intensive, and the synthesis effect largely depends on the speech database, which is greatly affected by the environment and has low robustness. The basic idea of the statistical parametric speech synthesis method is to decompose the parameters of the input speech signal and establish a statistical model, predict the speech parameters of the text to be synthesized through the trained model, input the parameters into the parametric synthesizer, and finally obtain the synthesized speech. This method requires less data and less manual intervention when the system is constructed, and the synthesized speech is smooth and smooth with high robustness. However, the sound quality of the synthesized voice is not high, and the emotional rhythm is not rich enough and relatively flat.

Since the HMM-based statistical parametric speech synthesis method can synthesize the speech of different speakers through speaker adaptive transformation, it has become a research hotspot in cross-language multilingual speech synthesis. The multilingual speech synthesis system based on HMM adopts mixed language method, phoneme mapping method or state mapping method to realize multilingual speech synthesis. However, most of the existing research focuses on languages with large corpora and relatively mature speech synthesis technology, and there is a lack of research on dialects, ethnic languages, and languages whose speech resources are not easy to obtain. At present, in domestic and foreign research, there is no multilingual speech synthesis system for Mandarin/minority languages or Mandarin/dialects. At present, research on multilingual speech synthesis mainly focuses on mainstream languages, and mostly uses phoneme mapping or state mapping, but both methods require a large amount of bilingual speech data. For Tibetan, which lacks speech resources, it is difficult to apply the above method to Mandarin-Tibetan multilingual speech synthesis due to the lack of large-scale Chinese-Tibetan bilingual speech corpus.

Contents of the invention

The present invention aims to solve the problem that the multilingual speech synthesis system proposed in the background technology lacks research on dialects, ethnic languages, and voices that are not easy to obtain, such as Tibetan, which cannot realize the multilingual speech synthesis of Putonghua-Tibetan language, and provides a A Chinese-Tibetan bilingual speech synthesis method and device.

For solving the problems of the technologies described above, the technical solution adopted in the present invention is: a Chinese-Tibetan bilingual speech synthesis method, wherein, comprising steps:

A. Taking the International Phonetic Alphabet as a reference, obtain the International Phonetic Alphabet for the input Tibetan Pinyin letters, and then compare them with the International Phonetic Alphabet of the Chinese Pinyin. The same parts are directly marked with SAMPA-SC, and the different parts use unused keyboard symbols Marking, using the SAMPA-T-oriented word-to-sound conversion algorithm to complete the SAMPA-T automatic labeling of Tibetan text corpus;

B. According to the similarity between Tibetan and Mandarin, on the basis of the Mandarin phonetic system, design a common Chinese and Tibetan phonetic system and problem sets;

C. Using the speech data of Chinese and Tibetan speakers, based on the HMM model, through speaker adaptive training, the mixed language average sound model is trained;

D. Using the corpus of a small number of speakers of the target language Tibetan or Chinese to be synthesized, the speaker adaptive model is obtained through speaker adaptive transformation, and the adaptive model is corrected and updated;

E. Input the text to be synthesized, generate speech parameters, and synthesize Tibetan or Chinese speech.

Further, the word-to-voice conversion algorithm of SAMPA-T described in the step A comprises the steps:

First read the Tibetan sentence text, and then segment the sentence and syllable according to the single vertical symbol and the syllable symbol to obtain the Tibetan monad. For each syllable, separate the sound and the final vowel through the base character Ding positioning and base character Ding decomposition, and finally through Look up the initial consonant SAMPA-T table and the final consonant SAMPA-T table to obtain the SAMPA-T character string of the syllable, and the base letter decomposition is to split the table according to the base letter.

Further, the common phonetic notation system and question set of designing Chinese and Tibetan in the said step B include the following steps:

First of all, Tibetan consonants that are pronounced the same as Mandarin are marked with Chinese pinyin, and those that are inconsistent with Mandarin are marked with Tibetan pinyin;

Then, select all the consonants and finals of Mandarin and Tibetan as well as silence and pause as the synthetic primitives of the context-dependent MSD-HSMM to design a contextual annotation format, which is used to mark the consonant and final layers, syllable layers, and word layers of each synthetic primitive , context-dependent features of prosodic word layer, phrase layer and sentence layer;

Finally, a Chinese-Tibetan bilingual general question set is designed on the basis of a Mandarin context-sensitive question set. This question set expands the related questions of Tibetan-specific synthetic primitives to reflect the special pronunciation of Tibetan. The question set contains 3000 Multiple context-sensitive questions covering all features of context-sensitive labeling.

Further, through the speaker adaptive training in the step C, the training to obtain the mixed language average tone model includes the following steps:

a. Perform phonetic analysis on multi-speaker Chinese corpus and single-speaker Tibetan corpus data, and extract their acoustic parameters:

(1) Extract mel cepstrum coefficients, logarithmic fundamental frequency and aperiodic index,

(2) Calculate their first-order difference and second-order difference;

b. Combining the context attribute set, conduct HMM model training, and train the statistical model of its acoustic parameters:

(1) HMM model of training spectrum and fundamental frequency parameters,

(2) Multi-distribution semi-hidden Markov model (MSD-HSMM) for training state duration parameters;

c. Use a small number of single-speaker Chinese speech databases and single-speaker Tibetan speech databases to perform speaker adaptive training to obtain the average sound model:

(1) Using the Constrained Maximum Likelihood Linear Regression (CMML) algorithm, the difference between the speech data of the speaker in training and the average voice is represented by a linear regression function,

(2) Normalize the differences between training speakers with a set of linear regression equations for state output distributions and state duration distributions,

(3) training to obtain the mixed language average sound model of Chinese and Tibetan bilinguals, so as to obtain the context-dependent MSD-HSMM model;

d. Use the single-speaker adaptive data of Chinese and Tibetan to perform speaker adaptive transformation:

(1) Using the CMML algorithm to calculate the mean vector and covariance matrix of the speaker's state output probability distribution and state duration probability distribution,

(2) Transform the mean vector and covariance matrix of the average tone model into a Tibetan or Chinese target speaker model to be synthesized with a set of transformation matrices of state output distribution and state duration distribution,

(3) Carry out maximum likelihood estimation to normalized and converted frequency spectrum, fundamental frequency and duration parameter;

e. Correct and update the adaptive model:

(1) Using the maximum a posteriori (MAP) algorithm to calculate the MAP estimation parameters of the average tone model state output and duration distribution,

(2) Calculate the mean vector of the state output and state duration after the adaptive transformation,

(3) Calculate the weighted average MAP estimate of the adaptive mean vector;

f, input the text to be synthesized, carry out text analysis to it, obtain the HMM model of the sentence;

g. Predict the parameters of the sentence HMM, generate the speech parameters, and obtain the synthesized speech after the parameter synthesizer, the formula is as follows:

in, output the mean vector for the state of the training speaker s, is the mean vector of its state duration. W=[A, b] and X=[α, β] are the transformation matrices of the difference between the state output distribution and the state duration distribution between the training speaker s and the average sound model respectively, and o _i and d _i are the average observation vector and the average duration vector.

Further, as described in the step D, using the corpus of a small number of speakers in the target language Tibetan or Chinese to be synthesized, the speaker adaptive model is obtained through speaker adaptive transformation, and the adaptive model is corrected and updated, Including the following steps:

First, after speaker adaptive training, use the HSMM-based CMLLR adaptive algorithm to calculate the mean vector and covariance matrix of the speaker transition state output probability distribution and duration probability distribution, the feature vector o and the state duration d in state i The transformation equation of is:

b _i (o)=N(o; Au _i -b, A∑ _i A ^T )

＝|A ^-1 |N(Wξ; u _i , ∑ _i )

$\begin{array}{c}{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; ;</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; )</span>\\ <span\; class="notranslate">\; =</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&alpha;\&psi;</span>}<span\; class="notranslate">\; ;</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\end{array}$

Among them, ξ=[o ^T , 1], ψ=[d, 1] ^T , μ _i is the mean value of the state output distribution, _mi is the mean value of the time-length distribution, ∑ _i is the diagonal covariance matrix, is the variance, W=[A ^-1 b ^-1 ] is the linear transformation matrix of the target speaker state output probability density distribution, X=[α ^-1 , β ^-1 ] is the transformation matrix of the state duration probability density distribution;

Then, through the adaptive transformation algorithm based on HSMM, normalization and transformation can be carried out to the frequency spectrum, fundamental frequency and duration parameters of voice data, and for the adaptive data O whose length is T, transformation Λ=(W, X) Perform maximum likelihood estimation

$\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}<span\; class="notranslate">\; ,</span>\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}<span\; class="notranslate">\; )</span>$

Among them, λ is the parameter set of HSMM;

Finally, the maximum a posteriori (MAP) algorithm is used to modify and update the adaptive model of speech. For a given HSMMλ, if its forward probability and backward probability are: α _i (i) and β _i (i ), then its generation probability of continuous observation sequence o _t-d+1 ......o _t in state i for:

${\mathrm{<span\; class="notranslate">\; \&kappa;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; )</span>}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span>\underset{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; t</span>}}{\mathrm{<span\; class="notranslate">\; \&Pi;</span>}}}{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>$

The MAP estimation is described as follows:

in, and is the mean vector after linear regression transformation, ω and τ are the MAP estimation parameters of the state output and time-length distribution, respectively, and is the adaptive mean vector and The weighted average MAP estimate of .

Further, the input to be synthesized text described in the step E, generating speech parameters, synthesizing Tibetan or Chinese speech includes the following steps:

First, use text analysis tools to convert the given text into a pronunciation annotation sequence containing contextual description information, use the decision tree obtained during the training process to predict the context-related HMM model of each pronunciation, and connect it into a sentence HMM model;

Then, a parameter generation algorithm is used to generate a parameter sequence of spectrum, duration and fundamental frequency from the sentence HMM;

Finally, the speech is synthesized by using the Mel logarithmic spectrum approximation (MLSA) filter as a parametric synthesizer.

Further, the Chinese-Tibetan bilingual speech synthesis device includes: an HMM model training unit, which is used to establish an HMM model of speech data; a speaker adaptive unit, which is used to normalize and convert the characteristic parameters of the speaker in training to obtain an adaptive model ; Speech synthesis unit for synthesizing Tibetan or Chinese speech to be synthesized.

Further, the HMM model training unit includes: a speech analysis subunit, which extracts the acoustic parameters of the speech data in the speech library, mainly extracting the fundamental frequency, frequency spectrum and duration parameters; the target HMM model determination subunit, combined with the context labeling information of the sound library , training the statistical model of the acoustic model, determining fundamental frequency, frequency spectrum and duration parameters according to the context attribute set, the speech analysis subunit is connected to the target HMM model determination subunit.

Further, the speaker adaptation unit includes a speaker training subunit, an average tone model determination subunit, a speaker adaptive transformation subunit and an adaptive model determination subunit connected in sequence, and the target HMM model determination subunit connected with the speaker training subunit,

The speaker training subunit is used for the difference between the state output distribution and the state duration distribution between the speaker and the average tone model in normalization training;

The average sound model determination subunit adopts the maximum likelihood linear regression algorithm to determine the average sound model of Chinese-Tibetan bilingual mixed speech;

The speaker adaptive transformation subunit uses adaptive data to calculate the mean vector and covariance matrix of the speaker's state output probability distribution and duration probability distribution, and convert it to the target speaker model;

The adaptive model determining subunit is to establish an adaptive model of the target speaker's MSD-HSMM.

Further, the speech synthesis unit includes an adaptive model correction subunit and a synthesis subunit connected in sequence, the adaptive model determination subunit is connected to the adaptive model correction subunit,

The adaptive model correction subunit uses the MAP algorithm to correct and update the adaptive model of speech, reduces model deviation, and improves the synthesis quality;

The synthesis subunit uses the modified adaptive model to predict the speech parameters of the input text, extracts the parameters and finally synthesizes Chinese or Tibetan speech through a speech synthesizer.

The advantages and positive effects of the present invention are: the Chinese-Tibetan bilingual speech synthesis method and device, through the similarity in pronunciation between Chinese and Tibetan, use the adaptive training and adaptive conversion algorithm based on HMM to realize the use of the same system Synthesize Chinese and Tibetan voices with good naturalness and fluency with the device. Compared with the traditional HMM-based speech synthesis system, this system adds a speaker adaptive training process in the training stage to obtain the average tone model of Chinese-Tibetan mixed speech. Through this process, the speaker's difference in the speech database can be reduced. The impact is to improve the quality of synthesized speech. On the basis of the average sound model, through the speaker adaptive transformation algorithm, only a small amount of Tibetan or Chinese corpus to be synthesized can be synthesized with good naturalness and fluency. Tibetan or Chinese pronunciation. The research of this system is of great significance to promoting communication with ethnic minorities and promoting the development of ethnic minority voice technology.

Description of drawings

Fig. 1 is a block diagram of a Chinese-Tibetan bilingual speech synthesis method;

Fig. 2 is a flow chart of Tibetan text to SAMPA-T conversion;

Fig. 3 is a block diagram of a Chinese-Tibetan bilingual speaker adaptive speech synthesis process;

Fig. 4 is a schematic structural diagram of a Chinese-Tibetan bilingual speech synthesis device;

Fig. 5 is the flowchart of model training;

Fig. 6 is a flowchart of speech synthesis.

Detailed ways

The present invention proposes a method for Chinese-Tibetan bilingual speech synthesis, proposes a word-to-sound conversion algorithm oriented to the Tibetan phonetic symbol SAMPA-T, and realizes the automatic labeling of the SAMPA-T of the Tibetan text corpus, according to Tibetan and The similarity between Mandarin, designed the common Mandarin and Tibetan phonetic transcription system, annotation format and question set, using the corpus of multiple Mandarin and Tibetan speakers, through HMM-based speaker adaptive training and speaker Adaptive transformation algorithm to finally synthesize Chinese or Tibetan speech. Chinese-Tibetan bilingual speech synthesis method block diagram of the present invention as shown in Figure 1, concrete steps are:

(1) Design the SAMPA-T tagging scheme for the Tibetan Lhasa dialect, and use the SAMPA-T-oriented word-to-sound conversion algorithm to complete the SAMPA-T automatic tagging of the Tibetan text corpus.

SAMPA (Speech Assessment Methods Phonetic Alphabet) is a computer-readable phonetic alphabet system that can represent all symbols of the International Phonetic Alphabet with ASCII characters. At present, SAMPA is widely used in major European languages, as well as East Asian languages such as Japanese. Chinese, Cantonese and Taiwan's "Mandarin" in China have also proposed SAMPA programs.

Since Tibetan and Chinese belong to the Tibetan-Chinese family, the present invention designs a computer-readable phonetic system SAMPA-T (Tibetan) for Tibetan on the basis of the machine-readable phonetic symbol design scheme of Mandarin Chinese, and uses Tibetan Lhasa The design scheme is listed as an example and the transliteration from Tibetan to SAMPA-T is realized.

By comparing the International Phonetic Alphabets of Chinese and Tibetan, it is found that the International Phonetic Alphabets of Chinese and Tibetan are partly consistent. Therefore, with the International Phonetic Alphabet as a reference, the International Phonetic Alphabet for the input Tibetan Pinyin letters is obtained, and then compared with the International Phonetic Alphabet of Chinese Pinyin For comparison of phonetic symbols, the same parts are directly marked with SAMPA-SC, and the different parts are marked with unused keyboard symbols according to the principle of simplification.

Tibetan is a type of phonetic writing, which is spelled out from the Tibetan phonetic alphabet, and its basic unit is a syllable. According to the structural positions of letters in the syllable, the traditional Tibetan grammar divides these syllables in different positions into "pre-added characters", "basic characters", "upper-added characters", "lower-added characters", "post-added characters" and "Add characters later", in which the base character is the core of the entire Tibetan character. Wherein Tibetan initial consonant=pre-addition word+upper addition word+basic character+under-addition word composition Ding, Tibetan final=vowel+back addition word+after addition word again.

The transliteration of Tibetan texts into SAMPA-T mainly considers the segmentation of Tibetan sentences, the segmentation of Tibetan words, the positioning of base characters, the separation and transliteration of consonants and vowels, and the combination of SAMPA-T strings. Among them, the SAMPA-T conversion of the base character D positioning and the consonant and final is converted into the core module, and the base character D positioning is the recognition of base characters, vowels, etc., which is mainly realized through dictionary-oriented statistics and search methods. Alphabet conversion is mainly realized by searching the SAMPA-T transliteration support library for initials and finals. First read the Tibetan text, and then segment sentences and syllables according to the single vertical symbols and syllable symbols to obtain the Tibetan list. For each syllable, the initials and finals are separated by base syllable positioning and syllable decomposition, and then the SAMPA-T of the syllable is obtained by looking up the SAMPA-T tables of initials and finals. The flow chart of Tibetan text to SAMPA-T conversion is shown in Figure 2.

(2) According to the similarity between Tibetan and Mandarin, and on the basis of the Mandarin phonetic system, design a common Chinese and Tibetan phonetic system and question sets.

Tibetan and Chinese belong to the Sino-Tibetan language family, and there are many similarities and differences in pronunciation. Both Mandarin Chinese and Tibetan Lhasa dialect are languages composed of syllables, and each syllable is composed of an initial consonant and a final consonant. Mandarin has 22 initials and 39 finals, while Tibetan Lhasa dialect has 36 initials and 45 finals, and the two languages share 20 initials and 13 finals. At first among the present invention, the Tibetan phonetic vowels that are consistent with the Mandarin pronunciation are marked with the Chinese phonetic alphabet; those that are inconsistent with the Mandarin pronunciation are marked with the Tibetan phonetic alphabet.

Then, select all the consonants and finals of Mandarin and Tibetan as well as silence and pause as the synthetic primitives of the context-dependent MSD-HSMM to design a contextual annotation format, which is used to mark the consonant and final layers, syllable layers, and word layers of each synthetic primitive , prosodic word level, phrase level and sentence level context-dependent features.

Finally, a Chinese-Tibetan bilingual question set is designed on the basis of a Mandarin context-sensitive question set. This question focuses on expanding the related questions of Tibetan-specific synthetic primitives to reflect the special pronunciation of Tibetan. The question set contains more than 3000 context-sensitive questions, covering all features of context-sensitive annotations.

This system uses a hierarchical labeling method to label the Tibetan text corpus, and the content of the label includes the syllable layer, boundary information and SAMPA-T transcription results. Use the internationally accepted Praat phonetic software for annotation, and the system can also add necessary annotation information as needed. After marking, write a script program to write the marking information into the .TexGrid file, which contains four layers of marking information, mainly including pronunciation and syllable boundary information. The question set contains classification information of basic features, such as initial consonant, final type, whether the syllable is in the first two digits of the prosodic phrase, etc. These classification information are usually based on a set of basic information in the context information. By reclassifying the question set, more complex context classification information than the basic features can be obtained. In the HTS system, the designed question sets are placed in the .hed file, each line describes a question, and the described question is a true-false question, and each question starts with a QS command.

(3) Using the Chinese-Tibetan multi-speaker speech data, based on the HMM model, through speaker adaptive training, the mixed language average sound model is trained.

Compared with the traditional speech synthesis method based on HMM, the present invention adds a speaker self-adaptive training process in the training stage to obtain the average tone model of Chinese-Tibetan mixed speech. The impact is to improve the quality of synthesized speech. On the basis of the average sound model, through the speaker adaptive transformation algorithm, only a small amount of Tibetan or Chinese corpus to be synthesized can be synthesized with good naturalness and fluency. Tibetan or Chinese pronunciation. The block diagram of the adaptive speech synthesis process for Chinese-Tibetan bilingual speakers is shown in Figure 3:

1> Conduct phonetic analysis on multi-speaker Chinese corpus and single-speaker Tibetan corpus data, and extract their acoustic parameters:

(1) Extract mel cepstrum coefficients, logarithmic fundamental frequency and aperiodic index;

(2) Calculate their first-order difference and second-order difference.

2> Combining the context attribute set, conduct HMM model training, and train the statistical model of its acoustic parameters:

(1) train the HMM model of frequency spectrum and fundamental frequency parameter;

(2) Multi-distribution semi-hidden Markov model (MSD-HSMM) for training state duration parameters.

3> Use a small number of single-speaker Chinese speech databases and single-speaker Tibetan speech databases for speaker adaptive training to obtain the average sound model:

(1) Using the Constrained Maximum Likelihood Linear Regression (CMML) algorithm, the difference between the speaker's voice data and the average voice in training is represented by a linear regression function;

(2) Normalize the differences between training speakers with a set of linear regression equations for state output distributions and state duration distributions;

(3) Training to get the Chinese-Tibetan bilingual mixed language average sound model, so as to get the context-dependent MSD-HSMM model.

4> Use the single-speaker adaptive data of Chinese and Tibetan to perform speaker adaptive transformation:

(1) Using the CMML algorithm to calculate the mean vector and covariance matrix of the speaker's state output probability distribution and state duration probability distribution;

(2) Transform the mean value vector and covariance matrix of the average tone model into Tibetan or Chinese target speaker models to be synthesized with a group of transformation matrices of state output distribution and state duration distribution;

(3) Perform maximum likelihood estimation on the normalized and transformed spectrum, fundamental frequency and duration parameters.

5> Correct and update the adaptive model:

(1) Using the maximum a posteriori (MAP) algorithm to calculate the MAP estimation parameters of the average sound model state output and duration distribution;

(2) Calculate the mean value vector of the state output and state duration after the adaptive transformation;

(3) Calculate the weighted average MAP estimate of the adaptive mean vector.

6> Input the text to be synthesized, perform text analysis on it, and obtain the HMM model of the sentence.

7> Predict the parameters of the sentence HMM, generate speech parameters, and obtain synthesized speech after passing through the parameter synthesizer.

Figure 3 is a flowchart of Chinese-Tibetan bilingual speech synthesis. Using the mixed corpus of Mandarin and Tibetan, the constrained maximum likelihood linear regression (CMML) is used to train the mean phonetic model of Chinese-Tibetan bilingual mixed language, so as to obtain the context-dependent multi-distribution semi-hidden Markov model (MSD-HSMM). In speaker adaptive training, the difference between the training speech data and the average voice of the training speaker is represented by a linear regression function of the output state distribution and the mean vector of the state duration distribution, which can be expressed by a set of state output distribution and state duration distribution The linear regression equation normalizes the differences between the training speakers with the following formula:

in, output the mean vector for the state of the training speaker s, is the mean vector of its state duration. W=[A, b] and X=[α, β] are the transformation matrices of the difference between the state output distribution and the state duration distribution between the training speaker s and the average sound model respectively, and o _i and d _i are the average observation vector and the average duration vector.

(4) Using the corpus of a small number of speakers of the target language Tibetan or Chinese to be synthesized, the speaker adaptive model is obtained through speaker adaptive transformation, and the adaptive model is corrected and updated.

After speaker adaptive training, the mean vector and covariance matrix of the speaker transition state output probability distribution and duration probability distribution are calculated by using the HSMM-based CMLLR adaptive algorithm. The transformation equation of eigenvector o and state duration d in state i is:

b _i (o)=N(o; Au _i -b, A∑ _i A ^T )

＝|A ^-1 |N(Wξ; u _i , ∑ _i )

$\begin{array}{c}{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; ;</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; )</span>\\ <span\; class="notranslate">\; =</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&alpha;\&psi;</span>}<span\; class="notranslate">\; ;</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\end{array}$

Among them, ξ=[o ^T , 1], ψ=[d, 1] ^T , μ _i is the mean value of the state output distribution, _mi is the mean value of the time-length distribution, ∑ _i is the diagonal covariance matrix, is the variance. W=[A ^-1 b ^-1 ] is the linear transformation matrix of the target speaker's state output probability density distribution, and X=[α ^-1 , β ^-1 ] is the transformation matrix of the state duration probability density distribution.

Through the adaptive transformation algorithm based on HSMM, the frequency spectrum, fundamental frequency and duration parameters of speech data can be normalized and transformed. For the adaptive data O whose length is T, the maximum likelihood estimation can be performed on the transformation Λ=(W, X),

$\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}<span\; class="notranslate">\; ,</span>\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}<span\; class="notranslate">\; )</span>$

Among them, λ is the parameter set of HSMM.

Finally, the adaptive model of speech is revised and updated by using the Maximum A Posteriori (MAP) algorithm. For a given HSMMλ, if its forward probability and backward probability are: α _i (i) and β _i (i), then it continuously observes the sequence o _t-d+1 in state i  … The generation probability of .o _t for:

${\mathrm{<span\; class="notranslate">\; \&kappa;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; )</span>}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span>\underset{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; t</span>}}{\mathrm{<span\; class="notranslate">\; \&Pi;</span>}}}{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>$

The MAP estimation is described as follows:

in, and is the mean vector after linear regression transformation, ω and τ are the MAP estimation parameters of the state output and time-length distribution, respectively. and is the adaptive mean vector and The weighted average MAP estimate of .

The training phase mainly includes preprocessing and HMM training. In the preprocessing stage, first analyze the speech data in the sound bank, and extract the corresponding speech parameters (fundamental frequency and spectral parameters). According to the extracted speech parameters, the observation vector of HMM can be divided into two parts: spectrum and fundamental frequency. The spectral parameter part is modeled by continuous probability distribution HMM, while the fundamental frequency part is modeled by multi-spatial probability distribution HMM (MSD-HMM). Modeling. At the same time, the system uses Gaussian distribution or Gamma distribution to establish a state duration model to describe the time structure of speech. In addition, the HMM synthesis system also uses linguistic and prosodic features to describe the context. Before the model training, it is necessary to design the context attribute set and the problem set for decision tree clustering, that is, select some context attributes that have a certain influence on the acoustic parameters (spectrum, fundamental frequency, and duration) based on prior knowledge and design the corresponding Question set for context-sensitive model clustering.

During the training of the model, the EM algorithm is used to train the HMM model of the sequence of acoustic parameter vectors according to the ML criterion. Finally, the spectral parameter model, the fundamental frequency parameter model and the duration model were clustered separately using a contextual decision tree to obtain a predictive model for synthetic use. The entire model training process is shown in Figure 5.

(5) Input the text to be synthesized, generate speech parameters, and synthesize Tibetan or Chinese speech.

First, text analysis tools are used to convert the given text into a pronunciation annotation sequence containing contextual description information, and the decision tree obtained during the training process is used to predict the context-related HMM model of each pronunciation, and connect it into a sentence HMM model. Then, a parameter generation algorithm is used to generate parameter sequences of spectrum, duration and fundamental frequency from the sentence HMM. Finally, the speech is synthesized by using the Mel logarithmic spectrum approximation (MLSA) filter as a parametric synthesizer. The entire synthesis process is shown in Figure 6.

Corresponding to the above method, the present invention also provides a Chinese-Tibetan bilingual speech synthesis device, which is used to perform speech synthesis on the input Chinese or Tibetan speech to be synthesized by utilizing the pre-established Chinese-Tibetan bilingual speech library. In terms of implementation, The functions of the device can be realized by software, hardware or a combination of software and hardware. The schematic diagram of the internal structure of the device of the present invention is shown in FIG. 4 .

The internal structure of the device of the present invention includes an HMM model training unit, a speaker adaptive unit and a speech synthesis unit.

1>HMM model training unit, for setting up the HMM model of voice data:

(1) Speech analysis subunit extracts the acoustic parameters of the speech data in the speech database, mainly extracting fundamental frequency, frequency spectrum and duration parameters;

(2) The target HMM model determines the subunit, combines the contextual labeling information of the sound bank, trains the statistical model of the acoustic model, and determines the fundamental frequency, spectrum and duration parameters according to the context attribute set.

2> Speaker Adaptation Unit, used to normalize and convert the characteristic parameters of the speaker in training to obtain an adaptive model:

(1) speaker training subunit, the difference between the state output distribution and the state duration distribution between the speaker and the average tone model in the normalization training;

(2) the average tone model determines the subunit, adopting the maximum likelihood linear regression algorithm to determine the average tone model of Chinese-Tibetan bilingual mixed speech;

(3) The speaker adaptive transformation subunit uses adaptive data to calculate the mean vector and covariance matrix of the speaker's state output probability distribution and duration probability distribution, and convert it to the target speaker model;

(4) An adaptive model determination subunit, which establishes an adaptive model of the target speaker's MSD-HSMM.

3> Speech synthesis unit, used to synthesize Tibetan or Chinese speech to be synthesized:

(1) The adaptive model correction subunit uses the MAP algorithm to correct and update the adaptive model of speech, reduces the model deviation, and improves the synthesis quality:

(2) The synthesis subunit uses the modified adaptive model to predict the speech parameters of the input text, and extracts the parameters to finally synthesize Chinese or Tibetan speech through a speech synthesizer.

The process of the above-mentioned method can be completed by a program instructing related hardware, and the program can be stored in a readable storage medium, and the corresponding steps in the above-mentioned method can be executed when the program is executed.

In order to illustrate the superiority of the method adopted in the present invention and other methods, evaluate the Tibetan speech and Chinese speech quality of synthesis, train the model of 3 different MSD-HSMM, by comparing the quality of synthetic speech under these three kinds of models, can The advantages of the method disclosed in the present invention are illustrated.

The speech library is selected as the Mandarin speech of 7 female speakers (169 sentences for each speaker) and 800 speeches of 1 Tibetan female speaker as the training data. The Tibetan sentences are selected from Tibetan newspapers in recent years. All recordings are saved in Microsoft WAV file format (single channel, 16-bit quantization, 16 kHz sampling).

In the experiment, 100 sentences were randomly selected from 800 Tibetan sentences as test sentences. From the remaining 700 Tibetan sentences, 10 sentences, 100 sentences and 700 sentences were randomly selected to establish three Tibetan training sets. These training sets were used together with the training utterances of 7 female Mandarin speakers to train the Sino-Tibetan bilingual mixed language average voice model. The 3 Tibetan training sets and the first female Mandarin speaker training corpus were also used to obtain speaker-dependent acoustic models for Tibetan and Mandarin.

1) SD model: the speaker-related model of Tibetan Lhasa dialect trained by using three Tibetan training sets (10/100/700 Tibetan sentences).

2) SI model: a speaker-independent model trained using only the training sentences of 7 female Mandarin speakers.

3) SAT model: First, use 3 Tibetan training sets and all Mandarin training sentences of 7 Mandarin speakers to train and obtain 3 Chinese-Tibetan bilingual mixed language average phonetic models; then use 3 Tibetan training sets and 7 Mandarin speakers respectively. The first Mandarin speaker's training sentence, the obtained speaker-dependent model.

During the evaluation, the Tibetan Lhasa dialect test sentences synthesized by the SD model and the SAT model were randomly played to 8 Tibetan Lhasa dialect evaluators. A total of 120 test voice files (20 Tibetan test sentences × 3 Tibetan training sets × 2 models). The testers are required to listen carefully to the 120 sentences, and then rate the voice quality of each sentence on a 5-point scale. After the MOS evaluation, the evaluators were also asked to describe the overall intelligibility of the Tibetan speech synthesized from different Tibetan Lhasa dialect training sets. In the evaluation of Chinese MOS, the same method was used to randomly play 54 synthetic Mandarin sentences (18 Mandarin test sentences × 3 Tibetan training sets) to the Mandarin evaluators. Score on a 5-point scale.

The MOS evaluation results of speech synthesized under different Tibetan sentence training corpora show the average MOS score and its 95% confidence interval. For Tibetan synthesized speech, the SAT model outperforms the SD model under each Tibetan training set. For the 10-sentence Tibetan training speech, the MOS score of the speech synthesized by the SD model is only 1.99, while the MOS of the SAT model is relatively high at 2.4. When participating in the evaluation, the testers felt that the Tibetan language synthesized by the SD model was difficult to understand, while the Tibetan language synthesized by the SAT model was easy to understand. When the Tibetan training sentence is 100 sentences, the MOS score and intelligibility of the two models are improved, but the SAT model is still significantly better than the SD model. When the training sentence reaches 700 sentences, the MOS scores of the two models are basically the same. At the same time, reviewers found the synthesized voice to be easy to understand. Therefore, in the case of small corpus, the quality of speech synthesized by SAT model is better than that of SD model. When the Tibetan corpus increases, the quality of Tibetan speech synthesized by the two models will tend to be the same. Therefore, the method disclosed in the present invention is very suitable for synthesizing high-quality speech in the absence of a corpus.

For Mandarin speech synthesis, under each Tibetan training set, mixing Tibetan sentences into the training corpus has almost no effect on the synthesis results of Mandarin. The synthesized Mandarin MOS scores are all around 4.0, and the synthesis effect is good.

The DMOS method is used to evaluate the similarity of language. In the DMOS evaluation, all test sentences and their original recordings are used to participate in the evaluation. A total of 140 Tibetan speech files synthesized (20 Tibetan sentences × 3 Tibetan training sets × 2 models + 20 Tibetan sentences synthesized by the SI model). Each synthesized Tibetan sentence and its original recording are a set of audio files. Randomly play 140 groups of test files to Tibetan Lhasa dialect testers: first play the original Tibetan recording, and then play the synthesized Tibetan speech. The evaluators are required to carefully compare the two voice files to assess how similar the synthesized voice is to the original voice. A 5-point scale is used for evaluation, with 5 points indicating that the synthesized speech is basically the same as the original speech, and 1 point representing that the synthesized speech is very different from the original speech.

In the evaluation of Chinese DMOS, the same method was used to randomly play 54 groups of Mandarin speech (18 Mandarin test sentences × 3 Tibetan training sets) to the Mandarin evaluators. points system.

The results show the mean DMOS score and its 95% confidence interval. For the synthesized speech of Tibetan, the DMOS score of the SI model is 2.41 points, which is better than the SD model trained with 10 Tibetan sentences and close to the SAT model trained with 10 Tibetan sentences. When the evaluators subjectively evaluated the synthesized SI model, they felt that the Tibetan speech synthesized by the SI model was similar to non-Tibetan speakers speaking Tibetan. This is because Mandarin and Tibetan not only share 33 synthetic primitives, but also have the same syllable structure and prosodic structure. Therefore, Tibetan-like speech can be synthesized using only Mandarin models. When adding more Tibetan training sentences, the DMOS score of the Tibetan speech synthesized by the SAT model is better than that of the SD model. When the number of Tibetan training sentences increases to 700, the DMOS score of the SD model is very close to that of the SAT model. This shows that in the case of fewer Tibetan sentences, the Tibetan Lhasa dialect speech synthesized by the method disclosed in the present invention is better than the Tibetan Lhasa dialect speech synthesized based on the SD model method.

The embodiments of the present invention have been described in detail above, but the content described is only a preferred embodiment of the present invention, and cannot be considered as limiting the implementation scope of the present invention. All equivalent changes and improvements made according to the scope of the present invention should still belong to the scope of this patent.

Claims (10)

1. The method for synthesizing the bilingual speech in the Tibetan language is characterized in that: the method comprises the following steps:

A. obtaining the international phonetic symbols of the input Tibetan language pinyin letters by taking the international phonetic symbols as reference, then comparing the international phonetic symbols with the international phonetic symbols of Chinese pinyin, directly marking the same parts by SAMPA-SC, marking the different parts by unused keyboard symbols, and completing SAMPA-T automatic labeling of the Tibetan language text corpus by utilizing a SAMPA-T oriented character-sound conversion algorithm;

B. designing a Chinese and Tibetan universal phonetic system and a question set on the basis of a Mandarin phonetic system according to the similarity of Tibetan and Mandarin;

C. training to obtain a mixed language average voice model by utilizing voice data of a plurality of speakers in the Tibetan language and through speaker self-adaptive training based on an HMM model;

D. obtaining a speaker self-adaptive model by utilizing the corpus of a speaker with a small amount of Tibetan language or Chinese voice to be synthesized through speaker self-adaptive transformation, and correcting and updating the self-adaptive model;

E. inputting a text to be synthesized, generating voice parameters, and synthesizing Tibetan or Chinese voice.

2. The method of synthesizing bilingual hanzang speech according to claim 1, wherein: the character-to-sound conversion algorithm of SAMPA-T in the step A comprises the following steps:

firstly reading in a Tibetan language sentence text, then segmenting sentences and syllables according to single pendants and syllable characters to obtain Tibetan language sentences, separating out initial consonants and vowels through positioning of a primary character string and decomposition of the primary character string for each syllable, and finally obtaining SAMPA-T character strings of the syllables by searching an initial SAMPA-T list and a vowel SAMPA-T list, wherein the decomposition of the primary character string is realized according to the split list of the primary character string.

3. The method of synthesizing bilingual hanzang speech according to claim 1, wherein: the step B of designing a general phonetic transcription system and a general question set of Chinese and Tibetan comprises the following steps:

firstly, the Tibetan initials and finals which are consistent with the pronunciation of the Mandarin are marked by the Pinyin of Chinese, and the Tibetan initials and finals which are inconsistent with the pronunciation of the Mandarin are marked by the Pinyin of Tibetan;

then, selecting all initial and final consonants and mutes and pauses of the Mandarin and the Tibetan as context-related MSD-HSMM synthesis primitives to design a context labeling format for labeling the context-related characteristics of an initial and final layer, a syllable layer, a word layer, a prosodic word layer, a phrase layer and a sentence layer of each synthesis primitive;

finally, a problem set which is universal for the Chinese-Tibetan bilingual is designed on the basis of a context-related problem set of the Mandarin, the problem set expands related problems of synthesized primitives which are specific to the Tibetan to reflect the special pronunciation of the Tibetan, and the problem set comprises more than 3000 context-related problems and covers all the characteristics of context-related labels.

4. The method of synthesizing bilingual hanzang speech according to claim 1, wherein: the step C of obtaining the mixed language average voice model through the speaker self-adaptive training and the training comprises the following steps:

a. carrying out voice analysis on the Chinese language database of multiple speakers and the Tibetan language database data of single speaker, and extracting acoustic parameters:

(1) extracting mel cepstrum coefficients, logarithmic fundamental frequency and non-periodic indexes,

(2) calculating the first order difference and the second order difference;

b. and (3) carrying out HMM model training by combining the context attribute set, and training a statistical model of the acoustic parameters:

(1) training HMM models of the spectral and fundamental frequency parameters,

(2) training a multi-distribution semi-hidden Markov model (MSD-HSMM) of a state duration parameter;

c. using a small amount of single speaker Chinese speech library and a single speaker Tibetan speech library to perform speaker self-adaptive training, thereby obtaining an average sound model:

(1) using constrained maximum likelihood linear regression (CMML) algorithm, expressing the difference between the phonetic data and average voice of the speaker in training by linear regression function,

(2) the differences between training speakers are normalized using a set of linear regression equations for the state output distribution and the state duration distribution,

(3) training to obtain a mixed language average sound model of the Chinese-Tibetan bilingual so as to obtain a context-dependent MSD-HSMM model;

d. the speaker self-adaptive transformation is carried out by utilizing the single speaker self-adaptive data of Chinese and Tibetan:

(1) adopting CMML algorithm to calculate the mean vector and covariance matrix of the state output probability distribution and state duration probability distribution of the speaker,

(2) transforming the mean vector and covariance matrix of the mean tone model into a target speaker model of Tibetan or Chinese to be synthesized using a set of transformation matrices of state output distribution and state duration distribution,

(3) carrying out maximum likelihood estimation on the frequency spectrum, the fundamental frequency and the time length parameters after normalization and conversion;

e. and (3) modifying and updating the adaptive model:

(1) calculating MAP estimation parameters of average tone model state output and time length distribution by adopting a Maximum A Posteriori (MAP) algorithm,

(2) calculating the average vector of the state output and the state duration after the self-adaptive transformation,

(3) calculating a weighted average MAP estimation value of the adaptive mean vector;

f. inputting a text to be synthesized, and performing text analysis on the text to obtain an HMM model of a sentence;

g. performing parameter prediction on a sentence HMM, performing voice parameter generation, and obtaining synthetic voice through a parameter synthesizer, wherein the formula is as follows:

wherein,the mean vector is output for training the state of the speaker s,is its state-long mean vector, W ═ A, b]And X ═ α, β]A transformation matrix o for training the difference between the state output distribution and the state duration distribution between the speaker s and the average tone model_iAnd d_iThe average observation vector and the average time length vector.

5. The method of synthesizing bilingual hanzang speech according to claim 1, wherein: the step D of obtaining the speaker self-adaptive model by utilizing the corpus of the speaker with a small amount of Tibetan language or Chinese voice to be synthesized through speaker self-adaptive transformation and correcting and updating the self-adaptive model comprises the following steps:

firstly, after adaptive training of a speaker, calculating to obtain a mean vector and a covariance matrix of state output probability distribution and duration probability distribution of speaker conversion by using a CMLLR adaptive algorithm based on HSMM, wherein a transformation equation of a feature vector o and a state duration d under a state i is as follows:

b_i(o)＝N(o；Au_i-b，A∑_iA^T)

＝|A^-1|N(Wξ；u_i，Σ_i)

<math> <mfenced open='' close=''> <mtable> <mtr> <mtd> <msub> <mi>p</mi> <mi>i</mi> </msub> <mrow> <mo>(</mo> <mi>d</mi> <mo>)</mo> </mrow> <mo>=</mo> <mi>N</mi> <mrow> <mo>(</mo> <mi>d</mi> <mo>;</mo> <mi>&alpha;</mi> <msub> <mi>m</mi> <mi>i</mi> </msub> <mo>-</mo> <mi>&beta;</mi> <mo>,</mo> <mi>&alpha;</mi> <msubsup> <mi>&sigma;</mi> <mi>i</mi> <mn>2</mn> </msubsup> <mi>&alpha;</mi> <mo>)</mo> </mrow> </mtd> </mtr> <mtr> <mtd> <mo>=</mo> <mo>|</mo> <msup> <mi>&alpha;</mi> <mrow> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mo>|</mo> <mi>N</mi> <mrow> <mo>(</mo> <mi>&alpha;&psi;</mi> <mo>;</mo> <msub> <mi>m</mi> <mi>i</mi> </msub> <mo>,</mo> <msubsup> <mi>&sigma;</mi> <mi>i</mi> <mn>2</mn> </msubsup> <mo>)</mo> </mrow> </mtd> </mtr> </mtable> </mfenced> </math>

wherein xi is ═ o^T，1]，ψ＝[d，1]^T，μ_iIs the mean of the state output distribution, m_iIs the mean value, Σ, of the time length distribution_iIn the form of a diagonal covariance matrix,is the variance, W ═ A^-1 b^-1]Outputting a linear transformation matrix of probability density distribution for the target speaker state, X ═ alpha^-1，β^-1]A transformation matrix of state duration probability density distribution;

then, through the adaptive transformation algorithm based on HSMM, the frequency spectrum, fundamental frequency and duration parameters of the voice data can be normalized and transformed, and for the adaptive data O with the length T, the transformation Λ ═ W, X can be estimated with maximum likelihood,

<math> <mrow> <mover> <mi>&Lambda;</mi> <mo>~</mo> </mover> <mo>=</mo> <mrow> <mo>(</mo> <mover> <mi>W</mi> <mo>~</mo> </mover> <mo>,</mo> <mover> <mi>X</mi> <mo>~</mo> </mover> <mo>)</mo> </mrow> <mo>=</mo> <munder> <mrow> <mi>arg</mi> <mi>max</mi> </mrow> <mi>&Lambda;</mi> </munder> <mi>P</mi> <mrow> <mo>(</mo> <mi>O</mi> <mo>|</mo> <mi>&lambda;</mi> <mo>,</mo> <mi>&Lambda;</mi> <mo>)</mo> </mrow> </mrow> </math>

wherein λ is the parameter set of HSMM;

finally, the Maximum A Posteriori (MAP) algorithm is used to modify and update the adaptive model of the speech, and for a given HSMM λ, if the forward probability and the backward probability are: alpha is alpha_i(i) And beta_i(i) Then it continuously observes the sequence o in state i_t-d+1......o_tGeneration probability ofComprises the following steps:

<math> <mrow> <msubsup> <mi>&kappa;</mi> <mi>t</mi> <mi>d</mi> </msubsup> <mrow> <mo>(</mo> <mi>i</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfrac> <mn>1</mn> <mrow> <mi>P</mi> <mrow> <mo>(</mo> <mi>O</mi> <mo>|</mo> <mi>&lambda;</mi> <mo>)</mo> </mrow> </mrow> </mfrac> <munderover> <munder> <mi>&Sigma;</mi> <mrow> <mi>j</mi> <mo>=</mo> <mn>1</mn> </mrow> </munder> <mrow> <mi>j</mi> <mo>&NotEqual;</mo> <mi>i</mi> </mrow> <mi>N</mi> </munderover> <msub> <mi>&alpha;</mi> <mrow> <mi>t</mi> <mo>-</mo> <mi>d</mi> </mrow> </msub> <mrow> <mo>(</mo> <mi>j</mi> <mo>)</mo> </mrow> <mi>p</mi> <mrow> <mo>(</mo> <mi>d</mi> <mo>)</mo> </mrow> <munderover> <mi>&Pi;</mi> <mrow> <mi>s</mi> <mo>=</mo> <mi>t</mi> <mo>-</mo> <mi>d</mi> <mo>+</mo> <mn>1</mn> </mrow> <mi>t</mi> </munderover> <msub> <mi>b</mi> <mi>i</mi> </msub> <mrow> <mo>(</mo> <msub> <mi>O</mi> <mi>s</mi> </msub> <mo>)</mo> </mrow> <msub> <mi>&beta;</mi> <mi>t</mi> </msub> <mrow> <mo>(</mo> <mi>i</mi> <mo>)</mo> </mrow> </mrow> </math>

the MAP estimate is described as follows:

wherein,andis the mean vector after linear regression transformation, omega and tau are respectively the MAP estimated parameters of state output and time length distribution,andas an adaptive mean vectorAndweighted average MAP estimate of (a).

6. The method of synthesizing bilingual hanzang speech according to claim 1, wherein: inputting the text to be synthesized in the step E to generate the voice parameters, and synthesizing the Tibetan language or the Chinese voice comprises the following steps:

firstly, converting a given text into a pronunciation labeling sequence containing context description information by using a text analysis tool, predicting a context-dependent HMM (hidden Markov model) model of each pronunciation by using a decision tree obtained in a training process, and connecting the HMM models into an HMM model of a sentence;

secondly, generating a parameter sequence of the frequency spectrum, the duration and the fundamental frequency from the sentence HMM by using a parameter generation algorithm;

finally, a Mel log-spectrum approximation (MLSA) filter is used as a parameter synthesizer to synthesize the voice.

7. The bilingual speech synthesis device of tibetan, its characterized in that: the method comprises the following steps: the HMM model training unit is used for establishing an HMM model of the voice data; the speaker self-adaptive unit is used for normalizing and converting the characteristic parameters of the speaker in training to obtain a self-adaptive model; and the voice synthesis unit is used for synthesizing the Tibetan or Chinese voice to be synthesized.

8. The apparatus according to claim 7, wherein: the HMM model training unit comprises: the voice analysis subunit extracts acoustic parameters of voice data in a voice library, and mainly extracts fundamental frequency, frequency spectrum and duration parameters; and the target HMM model determining subunit is used for training a statistical model of the acoustic model by combining context labeling information of the sound library, determining fundamental frequency, frequency spectrum and duration parameters according to a context attribute set, and the voice analysis subunit is connected with the target HMM model determining subunit.

9. The apparatus according to claim 8, wherein: the speaker self-adaptive unit comprises a speaker training subunit, an average tone model determining subunit, a speaker self-adaptive transformation subunit and a self-adaptive model determining subunit which are connected in sequence, the target HMM model determining subunit is connected with the speaker training subunit,

the speaker training subunit is used for normalizing the difference between the state output distribution and the state duration distribution between the speaker and the average voice model in the training;

the average sound model determining subunit determines a Chinese-Tibetan bilingual mixed speech average sound model by adopting a maximum likelihood linear regression algorithm;

the speaker self-adaptive transformation subunit calculates the mean vector and the covariance matrix of the state output probability distribution and the duration probability distribution of the speaker by using self-adaptive data and converts the mean vector and the covariance matrix into a target speaker model;

the adaptive model determining subunit establishes an adaptive model of the MSD-HSMM of the target speaker.

10. The apparatus according to claim 9, wherein: the voice synthesis unit comprises an adaptive model modification subunit and a synthesis subunit which are connected in sequence, the adaptive model determination subunit is connected with the adaptive model modification subunit,

the self-adaptive model correcting subunit corrects and updates the self-adaptive model of the voice by utilizing an MAP algorithm, reduces the model deviation and improves the synthesis quality;

and the synthesis subunit predicts the voice parameters of the input text by using the corrected self-adaptive model, extracts the parameters and finally synthesizes the Chinese or Tibetan voice through the voice synthesizer.