AU2015411306A1 - Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system - Google Patents
Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system Download PDFInfo
- Publication number
- AU2015411306A1 AU2015411306A1 AU2015411306A AU2015411306A AU2015411306A1 AU 2015411306 A1 AU2015411306 A1 AU 2015411306A1 AU 2015411306 A AU2015411306 A AU 2015411306A AU 2015411306 A AU2015411306 A AU 2015411306A AU 2015411306 A1 AU2015411306 A1 AU 2015411306A1
- Authority
- AU
- Australia
- Prior art keywords
- band
- speech
- glottal
- sub
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims abstract description 83
- 230000005284 excitation Effects 0.000 title claims abstract description 47
- 230000015572 biosynthetic process Effects 0.000 title claims abstract description 33
- 238000003786 synthesis reaction Methods 0.000 title claims abstract description 30
- 239000013598 vector Substances 0.000 claims abstract description 47
- 238000012549 training Methods 0.000 claims abstract description 35
- 230000003595 spectral effect Effects 0.000 claims abstract description 30
- 238000001228 spectrum Methods 0.000 claims description 14
- 230000001419 dependent effect Effects 0.000 claims description 10
- 238000001914 filtration Methods 0.000 claims description 7
- 238000003066 decision tree Methods 0.000 claims description 3
- 238000013518 transcription Methods 0.000 claims description 3
- 230000035897 transcription Effects 0.000 claims description 3
- 238000002372 labelling Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 description 44
- 238000000605 extraction Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 7
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 238000000513 principal component analysis Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 238000010561 standard procedure Methods 0.000 description 2
- 238000012952 Resampling Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 238000003064 k means clustering Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000036651 mood Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000004513 sizing Methods 0.000 description 1
- 238000013179 statistical model Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000014616 translation Effects 0.000 description 1
- 210000001260 vocal cord Anatomy 0.000 description 1
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/75—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 for modelling vocal tract parameters
Landscapes
- Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Machine Translation (AREA)
Abstract
A system and method are presented for forming the excitation signal for a glottal pulse model based parametric speech synthesis system. The excitation signal may be formed by using a plurality of sub-band templates instead of a single one. The plurality of sub-band templates may be combined to form the excitation signal wherein the proportion in which the templates are added is dynamically based on determined energy coefficients. These coefficients vary from frame to frame and are learned, along with the spectral parameters, during feature training. The coefficients are appended to the feature vector, which comprises spectral parameters and is modeled using HMMs, and the excitation signal is determined.
Description
BACKGROUND [0001] The present invention generally relates to telecommunications systems and methods, as well as speech synthesis. More particularly, the present invention pertains to the formation of the excitation signal in a Hidden Markov Model based statistical parametric speech synthesis system.
SUMMARY [0002] A system and method are presented for forming the excitation signal for a glottal pulse model based parametric speech synthesis system. The excitation signal may be formed by using a plurality of sub-band templates instead of a single one. The plurality of sub-band templates may be combined to form the excitation signal wherein the proportion in which the templates are added is dynamically based on determined energy coefficients. These coefficients vary from frame to frame and are learned, along with the spectral parameters, during feature training. The coefficients are appended to the feature vector, which comprises spectral parameters and is modeled using HMMs, and the excitation signal is determined.
[0003] In one embodiment, a method is presented for creating parametric models for use in training a speech synthesis system, wherein the system comprises at least a training text corpus, a speech database, and a model training module, the method comprising: obtaining, by the model training module, speech data for the training text corpus, wherein the speech data comprises recorded speech signals and corresponding transcriptions; converting, by the model training module, the training text corpus into context dependent phone labels; extracting, by the model training module, for each frame of speech in the speech signal from the speech training database, at least one of: spectral features, a plurality of band excitation energy coefficients, and fundamental frequency values; forming, by the model training module, a feature vector stream for each frame of speech using the at least one of:
WO 2017/061985
PCT/US2015/054122 spectral features, a plurality of band excitation energy coefficients, and fundamental frequency values;
labeling speech with context dependent phones; extracting durations of each context dependent phone from the labelled speech; performing parameter estimation of the speech signal, wherein the parameter estimation is performed comprising the features, HMM, and decision trees; and identifying a plurality of sub-band Eigen glottal pulses, wherein the sub-band Eigen glottal pulses comprise separate models used to form excitation during synthesis.
[0004] In another embodiment, a method is presented for identification of sub-band Eigen pulses from a glottal pulse database for training a speech synthesis system, wherein the method comprises:
receiving pulses from the glottal pulse database; decomposing each pulse into a plurality of sub-band components; dividing the sub-band components into a plurality of databases based on the decomposing; determining a vector representation of each database; determining Eigen pulse values, from the vector representation, for each database; and selecting a best Eigen pulse for each database for use in synthesis.
BRIEF DESCRIPTION OF THE DRAWINGS [0005] Figure 1 is a diagram illustrating an embodiment of a Hidden Markov Model based text to speech system.
[0006] Figure 2 is a flowchart illustrating an embodiment of a process for feature vector extraction.
[0007] Figure 3 is a flowchart illustrating an embodiment of a process for feature vector extraction.
[0008] Figure 4 is a flowchart illustrating an embodiment of a process for identification of Eigen pulses.
[0009] Figure 5 is a flowchart illustrating an embodiment of a process for speech synthesis.
CROSS-REFERENCE TO RELATED APPLICATIONS
WO 2017/061985
PCT/US2015/054122 [0010] This application is a Continuation-In-Part of U.S. application 14/288,745 filed May 28, 2014, entitled Method for Forming the Excitation Signal for a Glottal Pulse Model Based Parametric Speech
Synthesis System, the contents of which are incorporated in part herein.
DETAILED DESCRIPTION [0011] For the purposes of promoting an understanding of the principles of the invention, reference will now be made to the embodiment illustrated in the drawings and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended. Any alterations and further modifications in the described embodiments, and any further applications of the principles of the invention as described herein are contemplated as would normally occur to one skilled in the art to which the invention relates.
[0012] In speech synthesis, excitation is generally assumed to be a quasi-periodic sequence of impulses for voiced regions. Each sequence is separated from the previous sequence by some duration, such as
To = where To represents pitch period and Fo represents fundamental frequency. In unvoiced regions, it is modeled as white noise. However, in voiced regions, the excitation is not actually impulse sequences. The excitation is instead a sequence of voice source pulses which occur due to vibration of the vocal folds and their shape. Further, the pulses' shapes may vary depending on various factors such as: the speaker, the mood of the speaker, the linguistic context, emotions, etc.
[0013] Source pulses have been treated mathematically as vectors by length normalization (through resampling) and impulse alignment, as described in European Patent EP 2242045 (granted June 27,
2012, inventors Thomas Drugman, et al.), for example. The final length of the normalized source pulse signal is resampled to meet the target pitch. The source pulse is not chosen from a database, but obtained over a series of calculations which compromise the pulse characteristics in the frequency domain. Modeling of the voice source pulses has traditionally been done using acoustic parameters or
WO 2017/061985
PCT/US2015/054122 excitation models for HMM based systems, however, the models interpolate/re-sample the glottal/residuaI pulse to meet the target pitch period, which compromises the model pulse characteristics in the frequency domain. Other methods have used canonical ways of choosing the pulse, but convert residual pulses into equal length vectors by length normalization. These methods also perform PCA over these vectors, which makes the final pulse selected to be a computed one, rather than something selected directly from training data.
[0014] To achieve a final pulse through selection directly from training data, as opposed to computation, glottal pulses may be modeled by defining metrics and providing a vector representation.
Excitation formation, given a glottal pulse and fundamental frequency, is also presented which does not re-sample or interpolate on the pulse.
[0015] In statistical parametric speech synthesis, speech unit signals are represented by a set of parameters which can be used to synthesize speech. The parameters may be learned by statistical models, such as HMMs, for example. In an embodiment, speech may be represented as a source-filter model, wherein source/excitation is a signal which, when passed through an appropriate filter, produces a given sound. Figure 1 is a diagram illustrating an embodiment of a Hidden Markov Model (HMM) based Text to Speech (TTS) system, indicated generally at 100. An embodiment of an exemplary system may contain two phases, for example, the training phase and the synthesis phase, each of which are described in greater detail below.
[0016] The Speech Database 105 may contain an amount of speech data for use in speech synthesis.
Speech data may comprise recorded speech signals and corresponding transcriptions. During the training phase, a speech signal 106 is converted into parameters. The parameters may be comprised of excitation parameters, F0 parameters, and spectral parameters. Excitation Parameter Extraction 110a,
Spectral Parameter Extraction 110b, and F0 Parameter Extraction 110c occur from the speech signal
106, which travels from the Speech Database 105. A Hidden Markov Model may be trained using a
WO 2017/061985
PCT/US2015/054122 training module 115 using these extracted parameters and the Labels 107 from the Speech Database
105. Any number of HMM models may result from the training and these context dependent HMMs are stored in a database 120.
[0017] The synthesis phase begins as the context dependent HMMs 120 are used to generate parameters 135. The parameter generation 135 may utilize input from a corpus of text 125 from which speech is to be synthesized from. Prior to use in parameter generation 135, the text 125 may undergo analysis 130. During analysis 130, labels 131 are extracted from the text 125 for use in the generation of parameters 135. In one embodiment, excitation parameters and spectral parameters may be generated in the parameter generation module 135.
[0018] The excitation parameters may be used to generate the excitation signal 140, which is input, along with the spectral parameters, into a synthesis filter 145. Filter parameters are generally Mel frequency cepstral coefficients (MFCC) and are often modeled by a statistical time series by using
HMMs. The predicted values of the filter and the fundamental frequency as time series values may be used to synthesize the filter by creating an excitation signal from the fundamental frequency values and the MFCC values used to form the filter. Synthesized speech 150 is produced when the excitation signal passes through the filter.
[0019] The formation of the excitation signal 140 in Figure 1 is integral to the quality of the output, or synthesized, speech 150. Generally, spectral parameters used in a statistical parametric speech synthesis system comprise MCEPS, MGC, Mel-LPC, or Mel-LSP. In an embodiment, spectral parameters are mel-generalized cepstral (MGC) computed from the pre-emphasized speech signal, but the zeroth energy coefficient is computed from the original speech signal. In traditional systems, the fundamental frequency value alone is considered as a source parameter and the entire spectrum is considered as a system parameter. However, the spectral tilt, or the gross spectral shape, of the speech spectrum is actually a characteristic of the glottal pulse and is thus considered as a source parameter. The spectral
WO 2017/061985
PCT/US2015/054122 tilt is captured and modeled for glottal pulse based excitation and excluded as a system parameter.
Instead, pre-emphasized speech is used for computing the spectral parameter (MGC) with exception of the zeroth energy coefficient (energy of speech). This coefficient varies slowly in time and may be treated as a prosodic parameter computed directly from unprocessed speech.
Training and Model Construction [0020] Figure 2 is a flowchart illustrating an embodiment of a process for feature vector extraction, indicated generally at 200. This process may occur during spectral parameter extraction 110b of Figure
1. As previously described, the parameters may be used for model training, such as with an HMM model.
[0021] In operation 205, the speech signal is received for conversion into parameters. As shown in
Figure 1, the speech signal may be received from a speech database 105. Control is passed to operations 210 and 220 and process 200 continues. In an embodiment, operations 210 and 215 occur simultaneously with operation 220 and the determinations are all passed to operation 225.
[0022] In operation 210, the speech signal undergoes pre-emphasis. For example, pre-emphasizing the speech signal at this stage prevents low frequency source information from being captured in the determination of MGC coefficients in the next operation. Control is passed to operation 215 and process 200 continues.
[0023] In operation 215, spectral parameters are determined for each frame of speech. In an embodiment, the MGC coefficients 1-39 may be determined for each frame. Alternatively, MFCC and
LSP may also be used. Control is passed to operation 225 and process 200 continues.
[0024] In operation 220, the zeroth coefficient is determined for each frame of speech. In an embodiment, this may be determined using unprocessed speech as opposed to pre-emphasized speech.
Control is passed to operation 225 and process 200 continues.
WO 2017/061985
PCT/US2015/054122 [0025] In operation 225, the coefficients from operations 220 and 215 are appended to 1-39 MGC coefficients to form the 39 coefficients for each frame of speech. The spectral coefficients of a frame may then be referred to as the spectral vector. Process 200 ends.
[0026] Figure 3 is a flowchart illustrating an embodiment of a process for feature vector extraction, indicated generally at 300. This process may occur during excitation parameter extraction 110a of
Figure 1. As previously described, the parameters may be used for model training, such as with an HMM model.
[0027] In operation 305, the speech signal is received for conversion into parameters. As shown in
Figure 1, the speech signal may be received from a speech database 105. Control is passed to operations 310, 320, and 325 and process 300 continues.
[0028] In operation 310, pre-emphasis is performed on the speech signal. For example, preemphasizing the speech signal at this stage prevents low frequency source information from being captured in the determination of MGC coefficients in the next operation. Control is passed to operation
315 and process 300 continues.
[0029] In operation 315, linear predictive coding, or LPC Analysis is performed on the pre-emphasized speech signal. For example, the LPC Analysis produces the coefficients which are used in the next operation to perform inverse filtering. Control is passed to operation 320 and process 300 continues.
[0030] In operation 320, inverse filtering is performed on the analyzed signal and on the original speech signal. In an embodiment, operation 320 is not performed until after pre-emphasis has been performed (operation 310). Control is passed to operation 330 and process 300 continues.
[0031] In operation 325, the fundamental frequency value is determined from the original speech signal. The fundamental frequency value may be determined using any standard techniques known in the art. Control is passed to operation 330 and process 300 continues.
WO 2017/061985
PCT/US2015/054122 [0032] In operation 330, glottal cycles are segmented. Control is passed to operation 335 and process
300 continues.
[0033] In operation 335, the glottal cycles are decomposed. For each frame, in an embodiment, the corresponding glottal cycles are decomposed into sub-band components. In an embodiment, the subband components may comprise a plurality of bands, wherein the bands may comprise lower and higher components.
[0034] In the spectrum of a typical glottal pulse, there is may be a higher energy bulge in the low frequency and typically flat structure in the higher frequencies. The demarcation between those bands varies from pulse to pulse as well as the energy ratio. Given a glottal pulse, the cut off frequency which separates the higher and lower bands is determined. In an embodiment, a ZFR method may be used with suitable window sizing, but applied on the spectral magnitude. A zero crossing at the edge of the low frequency bulge results, which is taken as the demarcation frequency between lower and higher bands. Two components in the time domain may be obtained by placing zeros in the higher band region of the spectrum before taking the inverse FFT to obtain the time domain version of the low frequency component of the glottal pulse and vice versa to obtain the high frequency component. Control is passed to operation 340 and process 300 continues.
[0035] In operation 340, the energies are determined for the sub-band components. For example, the energies of each sub-band component may be determined to form the energy coefficients for each frame. In an embodiment, the number of sub-band components may be two. The determination of the energies for the sub-band components may be made using any of the standard techniques known in the art. The energy coefficients of a frame is then referred to as the energy vector. Process 300 ends.
[0036] In an embodiment, two-band energy coefficients for each frame are determined from the inverse filtered speech. The energy coefficients may represent the dynamic nature of glottal excitation.
The inverse filtered speech comprises an approximation to the source signal, after being segmented into
WO 2017/061985
PCT/US2015/054122 glottal cycles. The two-band energy coefficients comprise energies of the low and high band components of the corresponding glottal cycle of the source signal. The energy of the lower frequency component comprises the energy coefficient of the lower band and similarly the energy of the higher frequency component comprises the energy coefficient of the higher band. The coefficients may be modeled by including them in the feature vector of corresponding frames, which are then modeled by
HMM-GMM in HTS.
[0037] The two-band energy coefficients, in this non-limiting example, of the source signal are appended to the spectral parameters determined in the process 200 to form the feature stream along with the fundamental frequency values and modeled using HMMs as in a typical HMM-GMM(HTS) based TTS system. The model may then be used in Process 500, as described below, for speech synthesis.
Training for Eigen Pulse Identification [0038] Figure 4 is a flowchart illustrating an embodiment of a process for identification of Eigen pulses, indicated generally at 400. The Eigen pulses may be identified for each sub-band glottal pulse database and used in synthesis as further described below.
[0039] In operation 405, a glottal pulse database is created. In an embodiment, a database of glottal pulses is automatically created using training data (speech data) obtained from a voice talent. Given a speech signal, s(n), linear prediction analysis is performed. The signal s(n) undergoes inverse filtering to obtain the integrated linear prediction residual signal which is an approximation to glottal excitation.
The integrated linear prediction residual is then segmented into glottal cycles using a technique such as zero frequency filtering, for example. A number of small signals are obtained, referred to as glottal pulses, which may be represented as g^n), i = 1,2,3,... The glottal pulses are pooled to create the database. Control is passed to operation 410 and process 400 continues.
WO 2017/061985
PCT/US2015/054122 [0040] In operation 410, pulses from the database are decomposed into sub-band components. In an embodiment, the glottal pulses may be decomposed into a plurality of sub-band components, such as low and high band components, and the two band energy coefficients. In the spectrum of a typical glottal pulse, there is a high energy bulge in the low frequency and a typically flat structure in the high frequencies. However, the demarcation between the bands varies from pulse to pulse as does the energy ratio between these two bands. As a result, different models for both of these bands may be needed.
[0041] Given a glottal pulse, the cut off frequency is determined. In an embodiment, the cut off frequency is that which separates the higher and lower bands by using a Zero Frequency Resonator (ZRF) method with suitable window size, but applied on the spectral magnitude. A zero crossing at the edge of the low frequency bulge results, which is taken as the demarcation frequency between lower and higher bands. Two components in the time domain result from placing zeros in the higher band region of the spectrum before taking the inverse FFT to obtain the time domain version of the lower frequency component of glottal pulse and vice versa to obtain the higher frequency component.
Control is passed to operation 415 and process 400 continues.
[0042] In operation 415, the pulse databases are formed. For example, a plurality of glottal pulse databases, such as a low band glottal pulse database and a high band glottal pulse database, for example, result from operation 410. In an embodiment, the number of databases formed correspond to the number of bands formed. Control is passed to operation 420 and process 400 continues.
[0043] In operation 420, vector representations are determined of each database. In an embodiment, two separate models for lower and higher band components of glottal pulses have resulted, but the same method is applied to each of these models as further described. A sub-band glottal pulse refers, in this context, to a component of glottal pulse, either high or low band.
WO 2017/061985
PCT/US2015/054122 [0044] The space of sub-band glottal pulse signals may be treated as a novel mathematical metric space as follows:
[0045] Consider the function space M of functions that are continuous, of bounded variation and of unit energy. Translations in this space are identified where/is the same as g, if g is a translated/delayed version of/in time. An equivalence relation is imposed on this space where given/and g, where/and g represent any two sub-band glottal pulses,/is equivalent to g if there exists real constant θ E ]R, such that g = cos(0) + fh sin(0), where fh represents the Hilbert transform of/.
[0046] A distance metric, d, may be defined over the function space M. Given f,g E M, the normalized cross correlation between the two functions may be denoted as r(r) = f ® g. Let R(r) = ylr(r')2 + rh(r)2 where rh is the Hilbert transform of r. The angle between/and g may be defined as θ(/^) = suprR(r) meaning 0(/, g) assumes the maximum of value of the function R(r). The distance metric between/,g becomes d(/, g) = ^/2(1 — cos 0(/,//)). Together with the function space M, the metric d forms a metric space (M,d).
[0047] If the metric d is a Hilbertian metric, then the space can be isometrically embedded into a
Hilbert space. Thus x E M, for a given signal in a function space, may be mapped to a vector Ψχ(.) in a
Hilbert space, denoted as:
[0048] x -» ΨχΟ = | (—d2 (x,.) + d2 (x, x0) + d2 (., x0)) [0049] where x0 is a fixed element in M. The zero element is represented as Ψχ = 0. The mapping
Ψχ\χ E M represents the total in the Hilbert space. The mapping is isometric, meaning 1| — Ψγ11 = d(x,y).
[0050] The vector representation Ψχ(.) for a given signal x of the metric space depends on the set of distances of x from every other signal in the metric space. It is impractical to determine distances from all other points of the metric space, thus, the vector representation may depend only on the distances from a set of fixed number of points (cj of the metric space which are obtained as centroids after a
WO 2017/061985
PCT/US2015/054122 metric based clustering of a large set of signals from the metric space. Control is passed to operation
425 and process 400 continues.
[0051] In operation 425, Eigen pulses are determined and the process 400 ends. In an embodiment, to determine metrics for sub-band glottal pulses, a metric or notion of distance, d(x,y) between any two sub-band glottal pulses x and y is defined. The metric between two pulses f,g is defined as follows. The normalized circular cross correlation between f,g is defined as:
[0052] R(n) = f ° g [0053] The period for circular correlation is taken to be the highest of the lengths of f,g. The shorter signal is zero extended for the purpose of computing the metric and not modified in the database. The
Discrete Hilbert transform Rh(n) of R(n) is determined.
[0054] Next, the signal is obtained through the mathematical equation:
[0055] H(n) = V(ff(n))2 + (ffh(n))2 [0056] The cosine of the angle Θ between two signals/,g may be defined as:
[0057] cos0 = supnH(n) [0058] where supnH(n) refers to the maximum value among all the samples of the signal H(n). The distance metric may be given as:
[0059] d(f,g) = ^/2(1 — cos(0) [0060] The k-means clustering algorithm, which is well known in the art, may be modified to determine k cluster centroid glottal pulses from the entire glottal pulse database G. The first modification comprises replacing the Euclidean distance metric with the metric d(x,y), defined for glottal pulses as previously described. The second modification comprises updating the centroids of the clusters. The centroid glottal pulse of a cluster of glottal pulses whose elements are denoted as {glt g2,... g^} to be that element gc such that:
[0061] Dm = Tli=id\gi,gm)
WO 2017/061985
PCT/US2015/054122 [0062] is minimum for m=c. The clustering iterations are terminated when there is no shift in any of the centroids of the k clusters.
[0063] Vector representation for sub-band glottal pulses may then be determined. Given a glottal pulse xi; and assuming q, c2,.. cit c256 are the centroid glottal pulses determined by clustering as described in previously, let the size of the glottal pulse database be L. Assigning each one to one of the centroid clusters c; based on distance metric, the total number of elements assigned to centroid Cj may be defined as n;. Where x0 represents a fixed sub-band glottal pulse picked from the database, the vector representation may be defined as:
[0064] Ψ,·(Χί) = {d2(xi,c,·) - d2(xi,cJ) - d2(c7,x0)}^ [0065] Where V) is the vector representation for the sub-band glottal pulse xi; V) may be given as:
[0066] Vi = ['F1(xi),'F2(xi),'F3(xi),...¥5(xi),...'F256(xi)] [0067] For every glottal pulse in the database, a corresponding vector is determined and stored in the database.
[0068] The PCA in vector space is performed and the Eigen glottal pulses are identified. Principal component analysis (PCA) is performed on the collection of vectors associated with the glottal pulse database in order to obtain the Eigen vectors. The mean vector of the entire vector database is subtracted from each vector to obtain mean subtracted vectors. The Eigen vectors of the covariance matrix of the collection of vectors are then determined. With each Eigen vector obtained, a glottal pulse whose mean subtracted vector has minimum Euclidean distance from the Eigen vector is associated and called the corresponding Eigen glottal pulse. Eigen pulses for each sub-band glottal pulse database are thus determined and one from each is selected based on listening tests and may be used in synthesis as further described blow.
Use in Synthesis
WO 2017/061985
PCT/US2015/054122 [0069] Figure 5 is a flowchart illustrating an embodiment of a process for speech synthesis, indicated generally at 500. This process may be used to train the model obtained in the process 100 (Figure 1). In an embodiment, the glottal pulse used as excitation in a particular pitch cycle is formed by combining the lower band glottal template pulse and the higher band glottal template pulse after scaling each one to the corresponding two-band energy coefficient. The two-band energy coefficients for a particular cycle are taken to be that of the frame the pitch cycle corresponds to. The excitation is formed from the glottal pulse and filtered to obtain output speech.
[0070] Synthesis may occur in the frequency domain and in the time domain. In the frequency domain, for each pitch period, the corresponding spectral parameter vector is converted into a spectrum and multiplied with the spectrum of the glottal pulse. The result undergoes inverse Discrete Fourier
Transform (DFT) to obtain a speech segment corresponding to that pitch cycle. Overlap add is applied to all obtained pitch synchronous speech segments in the time domain to obtain the synthesized speech.
[0071] In the time domain, the excitation signal is constructed and filtered using a Mel Log Spectrum
Approximation (MLSA) filter to obtain the synthesized speech signal. The given glottal pulse is normalized to unit energy. For unvoiced regions, white noise of fixed energy is placed in the excitation signal. For voiced regions, the excitation signal is initialized with zeros. Fundamental frequency values, such as those given for every 5 ms frame, are used to compute the pitch boundaries. The glottal pulse is placed starting from every pitch boundary and overlap added onto the zero initialized excitation signal in order to obtain the signal. Overlap add is performed on the glottal pulse at each pitch boundary and a small fixed amount of band pass filtered white noise is added to ensure that there is a small amount of random/stochastic component present in the excitation signal. To avoid a windiness effect in the synthesized speech, a stitching mechanism is applied where a number of excitation signals are formed with using right-shifted pitch boundaries and circularly left-shifted glottal pulses. The right-shift in pitch boundary used for constructing comprises a fixed constant and the glottal pulse used for it is circularly
WO 2017/061985
PCT/US2015/054122 left shifted by the same amount. The final stitched excitation is the arithmetic average of the excitation signals. This is passed through the MLSA filter to obtain the speech signal.
[0072] In operation 505, text is input into the model in the speech synthesis system. For example, the model which was obtained in Figure 1 (context dependent HMMs 120), receives input text and provides features which are subsequently used to synthesize speech pertaining to the input text as described below. Control is passed to operation 510 and operation 515 and the process 500 continues.
[0073] In operation 510, the feature vector is predicted for each frame. This may be done using methods which are standard in the art, such as context dependent decision trees, for example. Control is passed to operations 525 and 540 and operation 500 continues.
[0074] In operation 515, the fundamental frequency value(s) are determined. Control is passed to operation 520 and process 500 continues.
[0075] In operation 520, pitch boundaries are determined. Control is passed to operation 560 and process 500 continues.
[0076] In operation 525, MGC are determined for each frame. For example, the 0-39 MGC are determined. Control is passed to operation 530 and process 500 continues.
[0077] In operation 530, the MGC are converted to the spectrum. Control is passed top operation 535 and process 500 continues.
[0078] In operation 540, energy coefficients are determined for each frame. Control is passed to operation 545 and process 500 continues.
[0079] In operation 545, Eigen pulses are determined and normalized. Control is passed to operation
550 and process 500 continues.
[0080] In operation 550, FFT is applied. Control is passed to operation 535 and process 500 continues.
WO 2017/061985
PCT/US2015/054122 [0081] In operation 535, data multiplication may be performed. For example, the data from operation
550 is multiplied with that in operation 535. In an embodiment, this may be done in sample by sample multiplication. Control is passed to operation 555 and process 500 continues.
[0082] In operation 555, inverse FFT is applied. Control is passed to operation 560 and process 500 continues.
[0083] In operation 560, overlap add is performed on the speech signal. Control is passed to operation
565 and process 500 continues.
[0084] In operation 565, the output speech signal is received and the process 500 ends.
[0085] While the invention has been illustrated and described in detail in the drawings and foregoing description, the same is to be considered as illustrative and not restrictive in character, it being understood that only the preferred embodiment has been shown and described and that all equivalents, changes, and modifications that come within the spirit of the invention as described herein and/or by the following claims are desired to be protected.
[0086] Hence, the proper scope of the present invention should be determined only by the broadest interpretation of the appended claims so as to encompass all such modifications as well as all relationships equivalent to those illustrated in the drawings and described in the specification.
WO 2017/061985
PCT/US2015/054122
Claims (19)
1. A method for creating parametric models for use in training a speech synthesis system, wherein the system comprises at least a training text corpus, a speech database, and a model training module, the method comprising:
a. obtaining, by the model training module, speech data for the training text corpus, wherein the speech data comprises recorded speech signals and corresponding transcriptions;
b. converting, by the model training module, the training text corpus into context dependent phone labels;
c. extracting, by the model training module, for each frame of speech in the speech signal from the speech training database, at least one of: spectral features, a plurality of band excitation energy coefficients, and fundamental frequency values;
d. forming, by the model training module, a feature vector stream for each frame of speech using the at least one of: spectral features, a plurality of band excitation energy coefficients, and fundamental frequency values;
e. labeling speech with context dependent phones;
f. extracting durations of each context dependent phone from the labelled speech;
g. performing parameter estimation of the speech signal, wherein the parameter estimation is performed comprising the features, HMM, and decision trees; and
h. identifying a plurality of sub-band Eigen glottal pulses, wherein the sub-band Eigen glottal pulses comprise separate models used to form excitation during synthesis.
2. The method of claim 1, wherein the spectral features are determined comprising the steps of:
a. determining an energy coefficient from the speech signal;
WO 2017/061985
PCT/US2015/054122
b. pre-emphasizing the speech signal and determining MGC coefficients for each frame of the pre-emphasized speech signal;
c. appending the energy coefficient and the MGC coefficients to form a MCG for each frame of the signal; and
d. extracting spectral vectors for each frame.
3. The method of claim 1, wherein the plurality of band excitation energy coefficients are determined comprising the steps of:
a. determining, from the speech signal, fundamental frequency values;
b. performing pre-emphasis on the speech signal;
c. performing LCP Analysis on the pre-emphasized speech signal;
d. performing inverse filtering on the speech signal and the LCP analyzed signal;
e. segmenting glottal cycles using the fundamental frequency values and the inversely filtered speech signal;
f. decomposing corresponding glottal cycles for each frame into sub-band components;
g. computing energies of each sub-band component to form a plurality of energy coefficients for each frame; and
h. using the energy coefficients to extract excitation vectors for each frame.
4. The method of claim 3, wherein the sub-band components comprise at least 2 bands.
5. The method of claim 4, wherein the sub-band components comprises at least a high band component and a low band component.
6. The method of claim 1, wherein the identifying a plurality of sub-band Eigen glottal pulses further comprises the steps of:
a. creating a glottal pulse databased using the speech data;
b. decomposing each pulse into a plurality of sub-band components;
WO 2017/061985
PCT/US2015/054122
c. dividing the sub-band components into a plurality of databases based on the decomposing;
d. determining a vector representation of each database;
e. determining Eigen pulse values, from the vector representation, for each database; and
f. selecting a best Eigen pulse for each database for use in synthesis.
7. The method of claim 6, wherein the plurality of sub-band components comprises low band and high band.
8. The method of claim 6, wherein the glottal database is created by:
a. performing linear prediction analysis on a speech signal;
b. performing inverse filtering of the signal to obtain an integrated linear prediction residual; and
c. segmenting the integrated linear prediction residual into glottal cycles to obtain a number of glottal pulses.
9. The method of claim 6, wherein the decomposing further comprises:
a. determining a cut off frequency, wherein said cut off frequency separates the sub-band components into groupings;
b. obtaining a zero crossing at the edge of the low frequency bulge;
c. placing zeros in the higher band region of the spectrum and obtaining the time domain version of the low frequency component of glottal pulse, wherein the obtaining comprises performing inverse FFT; and
d. placing zeros in the lower band region of the spectrum prior to obtaining the time domain version of the high frequency component of the glottal pulse, wherein the obtaining comprises performing inverse FFT.
WO 2017/061985
PCT/US2015/054122
10. The method of claim 9, wherein the groupings comprise a lower band grouping and a higher band grouping.
11. The method of claim 9, wherein the separating of sub-band components into groupings is performed using a ZFR method with suitable window size and applied on the spectral magnitude.
12. The method of claim 6, wherein the determining a vector representation of each database further comprises a set of distances from a set of fixed number of points of a metric space, obtained as centroids after a metric based clustering of a large set of signals from the metric space.
13. A method for identification of sub-band Eigen pulses from a glottal pulse database for training a speech synthesis system, wherein the method comprises:
a. receiving pulses from the glottal pulse database;
b. decomposing each pulse into a plurality of sub-band components;
c. dividing the sub-band components into a plurality of databases based on the decomposing;
d. determining a vector representation of each database;
e. determining Eigen pulse values, from the vector representation, for each database; and
f. selecting a best Eigen pulse for each database for use in synthesis.
14. The method of claim 13, wherein the plurality of sub-band components comprises a low band and a high band.
15. The method of claim 13, wherein the glottal pulse database is created by:
a. performing linear prediction analysis on a speech signal;
b. performing inverse filtering of the signal to obtain an integrated linear prediction residual; and
WO 2017/061985
PCT/US2015/054122
c. segmenting the integrated linear prediction residual into glottal cycles to obtain a number of glottal pulses.
16. The method of claim 13, wherein the decomposing further comprises:
a. determining a cut off frequency, wherein said cut off frequency separates the sub-band components into groupings;
b. obtaining a zero crossing at the edge of the low frequency bulge;
c. placing zeros in the high band region of the spectrum prior to obtaining the time domain version of the low frequency component of glottal pulse, wherein the obtaining comprises performing inverse FFT; and
d. placing zeros in the lower band region of the spectrum prior to obtaining the time domain version of the high frequency component of the glottal pulse, wherein the obtaining comprises performing inverse FFT.
17. The method of claim 16, wherein the groupings comprise a lower band grouping and a higher band grouping.
18. The method of claim 16, wherein the separating of sub-band components into groupings is performed using a ZFR method with suitable window size and applied on the spectral magnitude
19. The method of claim 13, wherein the determining a vector representation of each database further comprises a set of distances from a set of fixed number of points of a metric space, obtained as centroids after a metric based clustering of a large set of signals from the metric space.
WO 2017/061985
PCT/US2015/054122
1/5
FIG. 1
PCT/US2015/054122
WO 2017/061985
2/5
200
Fig 2
WO 2017/061985
PCT/US2015/054122
300 £ig_3
PCT/US2015/054122
WO 2017/061985
4/5
400
410
415
420
425
405
Fig 4
PCT/US2015/054122
WO 2017/061985
5/5
500
Fig 5
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/US2015/054122 WO2017061985A1 (en) | 2015-10-06 | 2015-10-06 | Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| AU2015411306A1 true AU2015411306A1 (en) | 2018-05-24 |
Family
ID=58488102
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| AU2015411306A Abandoned AU2015411306A1 (en) | 2015-10-06 | 2015-10-06 | Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system |
Country Status (6)
| Country | Link |
|---|---|
| EP (1) | EP3363015A4 (en) |
| KR (1) | KR20180078252A (en) |
| CN (1) | CN108369803B (en) |
| AU (1) | AU2015411306A1 (en) |
| CA (1) | CA3004700C (en) |
| WO (1) | WO2017061985A1 (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3857541B1 (en) | 2018-09-30 | 2023-07-19 | Microsoft Technology Licensing, LLC | Speech waveform generation |
| CN109767755A (en) * | 2019-03-01 | 2019-05-17 | 广州多益网络股份有限公司 | A kind of phoneme synthesizing method and system |
| CN111862931B (en) * | 2020-05-08 | 2024-09-24 | 北京嘀嘀无限科技发展有限公司 | Voice generation method and device |
| CN112365875B (en) * | 2020-11-18 | 2021-09-10 | 北京百度网讯科技有限公司 | Voice synthesis method, device, vocoder and electronic equipment |
| CN113571079A (en) * | 2021-02-08 | 2021-10-29 | 腾讯科技（深圳）有限公司 | Voice enhancement method, device, equipment and storage medium |
Family Cites Families (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6070140A (en) * | 1995-06-05 | 2000-05-30 | Tran; Bao Q. | Speech recognizer |
| US5937384A (en) * | 1996-05-01 | 1999-08-10 | Microsoft Corporation | Method and system for speech recognition using continuous density hidden Markov models |
| US20020116196A1 (en) * | 1998-11-12 | 2002-08-22 | Tran Bao Q. | Speech recognizer |
| US6970820B2 (en) * | 2001-02-26 | 2005-11-29 | Matsushita Electric Industrial Co., Ltd. | Voice personalization of speech synthesizer |
| DE60232560D1 (en) * | 2001-08-31 | 2009-07-16 | Kenwood Hachioji Kk | Apparatus and method for generating a constant fundamental frequency signal and apparatus and method of synthesizing speech signals using said constant fundamental frequency signals. |
| EP2058803B1 (en) * | 2007-10-29 | 2010-01-20 | Harman/Becker Automotive Systems GmbH | Partial speech reconstruction |
| WO2009144368A1 (en) * | 2008-05-30 | 2009-12-03 | Nokia Corporation | Method, apparatus and computer program product for providing improved speech synthesis |
| PL2242045T3 (en) * | 2009-04-16 | 2013-02-28 | Univ Mons | Speech synthesis and coding methods |
| CN102231275B (en) * | 2011-06-01 | 2013-10-16 | 北京宇音天下科技有限公司 | Embedded speech synthesis method based on weighted mixed excitation |
| CN102270449A (en) * | 2011-08-10 | 2011-12-07 | 歌尔声学股份有限公司 | Method and system for synthesising parameter speech |
| US20130080172A1 (en) * | 2011-09-22 | 2013-03-28 | General Motors Llc | Objective evaluation of synthesized speech attributes |
| US10453479B2 (en) * | 2011-09-23 | 2019-10-22 | Lessac Technologies, Inc. | Methods for aligning expressive speech utterances with text and systems therefor |
| GB2508417B (en) * | 2012-11-30 | 2017-02-08 | Toshiba Res Europe Ltd | A speech processing system |
| TWI573129B (en) * | 2013-02-05 | 2017-03-01 | 國立交通大學 | Streaming encoder, prosody information encoding device, prosody-analyzing device, and device and method for speech-synthesizing |
-
2015
- 2015-10-06 CN CN201580085103.5A patent/CN108369803B/en active Active
- 2015-10-06 CA CA3004700A patent/CA3004700C/en active Active
- 2015-10-06 EP EP15905930.2A patent/EP3363015A4/en not_active Ceased
- 2015-10-06 WO PCT/US2015/054122 patent/WO2017061985A1/en active Application Filing
- 2015-10-06 KR KR1020187012944A patent/KR20180078252A/en not_active Application Discontinuation
- 2015-10-06 AU AU2015411306A patent/AU2015411306A1/en not_active Abandoned
Also Published As
| Publication number | Publication date |
|---|---|
| CA3004700C (en) | 2021-03-23 |
| KR20180078252A (en) | 2018-07-09 |
| CA3004700A1 (en) | 2017-04-13 |
| CN108369803A (en) | 2018-08-03 |
| EP3363015A4 (en) | 2019-06-12 |
| WO2017061985A1 (en) | 2017-04-13 |
| CN108369803B (en) | 2023-04-04 |
| EP3363015A1 (en) | 2018-08-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10621969B2 (en) | Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system | |
| US9368103B2 (en) | Estimation system of spectral envelopes and group delays for sound analysis and synthesis, and audio signal synthesis system | |
| US7035791B2 (en) | Feature-domain concatenative speech synthesis | |
| AU2020227065B2 (en) | Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system | |
| WO2011026247A1 (en) | Speech enhancement techniques on the power spectrum | |
| CA3004700C (en) | Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system | |
| CN110459202A (en) | A kind of prosodic labeling method, apparatus, equipment, medium | |
| US10014007B2 (en) | Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system | |
| CN116994553A (en) | Training method of speech synthesis model, speech synthesis method, device and equipment | |
| Kadyan et al. | Prosody features based low resource Punjabi children ASR and T-NT classifier using data augmentation | |
| US10446133B2 (en) | Multi-stream spectral representation for statistical parametric speech synthesis | |
| CN102231275B (en) | Embedded speech synthesis method based on weighted mixed excitation | |
| CN111862931B (en) | Voice generation method and device | |
| Alhanjouri et al. | Robust speaker identification using denoised wave atom and GMM | |
| Rana et al. | Advancements in Real-Time Voice Conversion Technologies: A Comprehensive Analysis of Techniques | |
| Ye | Efficient Approaches for Voice Change and Voice Conversion Systems | |
| Jinachitra | Robust structured voice extraction for flexible expressive resynthesis | |
| HERMUS | Exponentieel Sinusoıdale Spraakcompressie voor Corpus-gebaseerde Tekst-naar-Spraak Synthese | |
| Gao et al. | A new approach to generating Pitch Cycle Waveform (PCW) for Waveform Interpolation codec |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| MK5 | Application lapsed section 142(2)(e) - patent request and compl. specification not accepted |