JP6884946B2 - Acoustic model learning device and computer program for it - Google Patents

Description

The present invention relates to a speech recognition technique, and more particularly to a learning device for improving the accuracy of a CTC (Connectionist Temporal Classification) acoustic model (CTC-AM) used in a speech recognition device.

An increasing number of devices and services use voice input / output as an interface between humans and computers. For example, voice input / output is also used for operating mobile phones. For voice input / output, it is necessary to make the recognition accuracy of the voice recognition device, which is the basis of the voice input / output, as high as possible.

A general technique for speech recognition uses a model obtained by statistical machine learning. For example, HMM (Hidden Markov Model) is often used as an acoustic model. In addition, a word pronunciation dictionary for calculating the probability that a phoneme string will be obtained from a character string generated in the process of speech recognition, and a word string for a certain language will appear with a probability. A language model or the like for calculating is also used.

The basic concept of speech recognition in a conventional speech recognition device using an HMM will be described with reference to FIG. Conventionally, it is considered that the word string 30 (word string W) is observed as the observation sequence 36 through the influence of various noises, and the word string having the highest probability of giving the final observation sequence 36 is voice-recognized. Output as a result of. In this process, the probability that the word string W is generated is represented by P (W). Let P (S | W) be the probability that the state sequence S (state sequence 34) of the HMM is generated from the word sequence W via the pronunciation sequence 32 which is an intermediate product. Furthermore, the probability that observation X can be obtained from the state series S is represented by P (X | S).

In the process of speech recognition, when the observation sequence X _{1: T} from the beginning to the time T is given, the word string that gives the maximum likelihood of giving such an observation sequence is output as the result of speech recognition. To. That is, the word string ^~ W as a result of speech recognition is obtained by the following equation (1). In the mathematical formula, the symbol "~" written immediately above the character is described immediately before the character in the specification.

The following is obtained by transforming the right side of this equation by Bayesian equation.

Furthermore, the first term of the numerator of this formula can be obtained by HMM as follows.

In this equation, the state series S _{1: T} indicates the state series S ₁ , ..., S _T of the HMM. The first term on the right side of Eq. (3) indicates the output probability of the HMM. From equations (1) to (3), the word string ~ W as a result of speech recognition can be obtained by the following equation.

In HMM, the observed value x _t at time t does not depend only on the state s _{t. Therefore, the output probability P (X 1: T} | S _{1: T} ) of the HMM in Eq. (4) can be calculated by the following equation.

Probability P (x _t | s _t) is calculated by the Gaussian mixture model (GMM).

Of the other terms in Eq. (4), P (S _{1: T} | W) is calculated by the product of the state transition probability of the HMM and the pronunciation probability of the word, and P (W) is calculated by the language model. The denominator P (X _{1: T} ) is a common value for each hypothesis and can therefore be ignored when performing arg max operations.

Recently, research has been conducted on a framework called the DNN-HMM hybrid method, in which the output probability in HMM is calculated by a deep neural network (DNN) instead of GMM. The DNN-HMM hybrid method has achieved higher accuracy than the acoustic model using GMM, and is drawing attention. In addition, because the DNN-HMM hybrid method has produced excellent results, neural networks such as convolutional neural networks (CNN), recurrent neural networks (RNN), or long short-term memory networks (LSTM) have replaced DNN. A method using (NN) has been proposed. It can be expected that the accuracy of voice recognition will be higher by these methods.

However, in such an NN-HMM hybrid method, since the output of NN _{represents the posterior probability P (S t} | X _t ), the conventional HMM using the output probability P (X _t | S _t ) is used as it is. Does not fit the framework of. To solve this problem, apply Bayes' law to the posterior probability P (S _t | X _t ) output by DNN to force the output probability P (X _t | S _{t) to fit equation (5).} It is necessary to transform the output of NN into a form that uses). If a voice recognition method that does not use such deformation can be realized, further improvement in accuracy can be expected.

Y. Miao, M. Gowayyed, and F. Metze, “EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding,” in Proc. ASRU, 2015, pp. 167-174. Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel and Yoshua Bengio, “End-to-end attention-based large vocabulary speech recognition”, in Proc. ICASSP, 2016, pp 4945-4949.

Recently, it has been proposed to use an end-to-end type NN as an acoustic model for speech recognition (Non-Patent Document 1). The End-to-End type NN sets the posterior probability P (s | X) of the subword string (phonetic sequence, phonetic symbol string, phoneme string, character string, etc.) s for the observation (speech feature) series X to HMM, etc. Express directly without going through. Therefore, there is a possibility that it can be applied to speech recognition without performing unreasonable deformation like the DNN-HMM hybrid. The End-to-End type NN will be described later in relation to the embodiment, but here, in order to describe the problems of the conventional method, the End-to-End type NN generally used in the End-to-End type NN will be described. The concept of speech recognition by type RNN will be explained. The present invention is applicable to all End-to-End type NNs, and is not necessarily limited to RNNs.

RNN is not only the connection between nodes in one direction from the input layer side to the output layer side, but also the connection between nodes from the output side layer to the adjacent input side layer, and the connection between nodes in the same layer. , And a structure including self-recurrent coupling and the like. Because of this structure, RNNs have the property of being able to represent time-dependent information, which is not found in ordinary feedforward neural networks. Speech is typical of time-dependent information. Therefore, RNN is considered to be suitable for acoustic models.

The label output by the End-to-End type RNN is, for example, an arbitrary subword such as a phoneme or a syllable, a character, or the state of an HMM. When the End-to-End type RNN is used for the acoustic model, it is not necessary to forcibly deform the output of the NN compared to the case where the HMM is used, so improvement in recognition accuracy can be expected.

As described above, the End-to-End type RNN learns the direct mapping from the input observation sequence X to the subword sequence s. A typical example of an End-to-End type RNN is a model called CTC. Since the observation sequence X is usually much longer than the subword sequence s, CTC adds an empty label φ to the output of the RNN to absorb the difference in length. That is, a node corresponding to the empty label φ is provided in the output layer. As a result, the subword string c = {c ₁ , ..., c _T } (including the empty label φ) for each frame is obtained in the output of the RNN. This subword string c is converted into a subword string s that does not depend on the number of frames by a function called a mapping function Φ. The mapping function Φ deletes the empty label φ from the subword string c in frame units, and outputs the subword string s that does not depend on the number of frames by regarding the repetition of the label as one output. By using the mapping function Φ, the probability P (s | X) in which the observation sequence X is the subword sequence s can be formulated as follows.

Here, y _t ^ct is the output score for _{the output label c t} of the RNN at time t. Φ ^-1 is the inverse function of the mapping function Φ. That is, Φ ^-1 (s) represents a set of all phoneme sequences c that can be mapped to the subword sequence s by the mapping function Φ.

The end-to-end type NN is characterized in that the probability P (s | X) that the observation sequence X represents the subword sequence s is directly learned by the neural network. As a method other than CTC, Non-Patent Document 2 expresses it by a model called Attention-based Recurrent Sequence Generator.

Unlike the HMM, the End-to-End type NN directly learns the probability P (s | X) that the observation sequence X represents the subword sequence s, so that the conventional decoding method using the HMM cannot be adopted. This NN also has the character of both an acoustic model and a language model. Therefore, at first, it was attempted to perform decoding using only NN without using the language model. However, it has been found that the best results cannot be obtained by decoding without an independent language model, and recently, the one using a language model in addition to the End-to-End type NN is the mainstream. However, in this case, the problem is how to combine the two. Furthermore, since the acoustic model based on the End-to-End type NN is usually learned in subword units (characters, phonemes, etc.), the output score is also in subword units. Since the score of the language model is at the word level, there is a problem that it is difficult to combine the two in this respect as well.

Conventionally, as a method of combining both scores, the word string ^to W has been calculated by simple interpolation of both scores as shown in the following equation.

The function Ψ is a function that transforms the word string W into a set of all possible subword strings s. Non-Patent Document 1 proposes to divide posterior probabilities by prior probabilities P ( _{ct) in each frame.}

However, there is no rationale for using the score calculated by such an interpolation method, and sufficiently high recognition performance has not been obtained. In an acoustic model using NN, it is necessary to further improve the accuracy of speech recognition by learning NN based on a clear rationale.

Therefore, an object of the present invention is to provide an acoustic model learning device capable of improving speech recognition accuracy in an acoustic model utilizing the characteristics of NN.

The acoustic model learning device according to the first embodiment of the present invention is used to calculate the probability that the observation sequence is an arbitrary subword string when a speech observation sequence is given. Learn the acoustic model based on the End type neural network. The learning device of this acoustic model stores the learning data consisting of an aligned pair of the observation sequence of the learning voice and the correct subword string corresponding to the learning voice, and the word model that stores the frequency of appearance of the word string. It is used by connecting to a computer-readable storage means. This learning device optimizes the End-to-End type neural network so that the sum of the posterior probability of the correct subword sequence of the training data is maximized when the observation sequence of the learning voice is given. Given the first optimization means to be converted and the observation sequence of the evaluation data, the expected value of the accuracy of the word string hypothesis estimated using the End-to-End type neural network and the language model is It includes a second optimization means that further optimizes the End-to-End type neural network so as to be maximized.
Preferably, the second optimization means generates a word sequence hypothesis by performing speech recognition on the observation sequence using an end-to-end type neural network and a language model over the entire learning speech. The speech recognition means to be performed, the first calculation means for calculating the recognition accuracy for the word string constituting the hypothesis based on the correct answer subword string of the hypothesis and the learning data, and the whole learning speech. A second calculation means for calculating the expected value by calculating the sum of the products of the posterior probability of the hypothesis calculated by the language model at the time of hypothesis generation and the recognition accuracy of the word strings constituting the hypothesis. In response to the update means for updating the parameter set of the acoustic model and the completion of the update of the parameter set for the acoustic model by the update means so that the expected value calculated by the second calculation means increases. Judgment means for executing the judgment process regarding whether or not the end condition is satisfied, the first process for ending the learning of the end-to-end type neural network in response to the judgment by the judgment means, and the learning voice. Speech recognition means, first calculation means, second calculation means, update means, and so as to perform the hypothesis generation process, recognition accuracy calculation, expected value calculation, parameter set update, and determination process again. It includes a second process for controlling the determination means and a control means for selectively executing the second process.
More preferably, the observation sequence is prepared for each frame of the voice signal representing the learning voice, and the first calculation means is input by each subword of the hypothetical word string output by the end-to-end type neural network. It includes a subword match number calculation means for calculating the number of matches for each subword of the subword sequence paired with the observed observation sequence in frame units.
More preferably, the determination means is predetermined by the speech recognition means for generating a hypothesis over the entire learning voice, the first calculation means for calculating the recognition accuracy, and the second calculation means for calculating the sum. It includes means for determining that the termination condition is satisfied when it is performed the number of times.
The determination means includes means for determining that the end condition is satisfied in response to the difference between the parameter set defining the End-to-End type neural network and the time of the previous processing being equal to or less than the threshold value. It may be.
The computer program according to the second aspect of the present invention functions to operate the computer as each means of the learning device of any of the acoustic models described above.

It is a figure which shows the concept of the conventional speech recognition. It is a figure which shows typically the structure of a normal DNN. It is a figure which shows typically the composition of RNN and the example of the connection between nodes of RNN at different time. It is a figure which shows the concept of speech recognition in one Embodiment of this invention. It is a block diagram which shows the structure of the voice recognition apparatus which adopts NN learned by the method which concerns on 1 Embodiment of this invention. It is a schematic block diagram of the apparatus which executes the learning method of CTC-AM which concerns on one Embodiment of this invention. It is a flowchart which shows the control structure of the program which realizes the learning method of CTC-AM which concerns on one Embodiment of this invention. It is a flowchart which shows the control structure of the program which realizes the process which performs the initial learning of CTC-AM in the method shown in FIG. FIG. 5 is a flowchart showing a control structure of a program that realizes a process of improving the accuracy of the initially learned CTC-AM in the method shown in FIG. 7. It is a graph which shows the effect by repeating learning by the method which concerns on 1 Embodiment of this invention. It is a graph which shows the effect by repeating learning by the method which concerns on 1 Embodiment of this invention. It is a figure which shows the appearance of the computer which realizes the voice recognition apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the hardware configuration of the computer shown in FIG.

In the following description and drawings, the same parts are given the same reference numbers. Therefore, detailed explanations about them will not be repeated.

First, the differences between DNNs and RNNs used in conventional techniques will be explained. With reference to FIG. 2, the DNN 70 includes an input layer 72 and an output layer 78, and a plurality of hidden layers 74 and 76 provided between the input layer 72 and the output layer 78. In this example, only two hidden layers are shown, but the number of hidden layers is not limited to two. Each layer has multiple nodes. In FIG. 2, the number of nodes in each layer is the same at 5, but these numbers are usually various. Adjacent nodes are connected to each other. However, the data flows only in one direction from the input layer side to the output layer side. Weights and biases are assigned to each bond. These weights and biases are learned from the training data by the error back propagation method using the training data.

In the DNN 70, when the input layer 72 _{is given the voice feature amount X t} at the time t at the time t, the state prediction value S _{t of the} HMM is output from the output layer 78. In the case of an acoustic model, the number of nodes in the output layer 78 is often designed to match the number of phonemes in the target language, in which case the output of each node in the output layer is the input voice. Shows the probability that the feature is the phoneme represented by the node. Therefore, the sum of the state prediction values output by each node of the output layer 78 is 1.

What is obtained by the DNN shown in FIG. 2 is P (S _t | X _t ). That is, it is the probability of _{the state S t of the HMM when the voice feature X t} is observed at time t. In this example, state S _t of the HMM correspond to phonemes. Comparing this with the above equation (5), it can be seen that in the case of DNN, the output cannot be directly applied (assigned) to the equation (5). Therefore, conventionally, the output of DNN is converted to _{P (X t} | _{St) using Bayes' law as shown below.}

In equation (10), P (x _t ) is common to the states of each HMM and can therefore be ignored in the arg max operation. P (s _t ) can be estimated by counting the number of each state in the aligned training data. After all, in the case of the DNN-HMM hybrid method, _{by dividing the output P (S t} | X _t ) of the DNN by the probability P (S _t ), the recognition score is calculated using the DNN within the framework using the conventional HMM. You are calculating.

On the other hand, an example of the configuration of the End-to-End type RNN is shown in FIG. FIG. 3 shows the relationship between RNN100 (t-1) at time t-1, RNN100 (t) at time t, and RNN (t + 1) at time t + 1. In this example, each node in the hidden layer of RNN100 (t) receives its own output of RNN100 (t-1) as well as each node of the input layer. That is, the RNN100 can generate an output for a time series of input audio features. Furthermore, among the End-to-End type RNNs, in CTC, in addition to the node corresponding to the label (for example, phoneme), the node corresponding to the empty label φ (shown at the right end in FIG. 3) is provided in the output layer of the RNN. Including. That is, the number of nodes in the output layer is the number of labels + 1.

The End-to-End type RNN as shown in FIG. 3 directly models the probability P (s | X) in which the voice (voice feature amount) X is the pronunciation sequence s. Therefore, speech recognition using such RNN does not depend on HMM. The output of the RNN is formulated as in Eqs. (6) and (7) above.

In order to take advantage of the characteristics of end-to-end RNNs and perform highly accurate speech recognition, it is necessary to use a framework other than the DNN-HMM hybrid method. Figure 4 shows such a new framework. The present embodiment relates to a device that performs voice recognition according to this framework. In this embodiment, CTC is adopted as the End-to-End type RNN, and the pronunciation sequence is adopted as the unit of the subword. Based on a new framework for voice recognition utilizing the characteristics of end-to-end RNNs, the decoding method using CTC will be improved, and the learning method of CTC itself will be improved accordingly.

With reference to FIG. 4, in the present embodiment, the probabilities of a plurality of phoneme strings 110 composed of a label sequence including an empty label φ are obtained from the observation sequence 36 using an RNN. This probability is modeled as in Eq. (7) above. A mapping function Φ is applied to these phoneme strings 110 to obtain a plurality of pronunciation strings (subword strings) 112 which are intermediate products. For example, the label sequence "AAφφBφCCφ" and the label sequence "φAφBBφCφ" are both mapped to the subword sequence "ABC" by the mapping function Φ. By this mapping function, the probability of the pronunciation sequence s given the observation sequence X is modeled as in Eq. (6) above. Here, the probabilities of a plurality of word strings 30 obtained from the pronunciation string (subword string) 112 are further obtained. This probability is modeled as P (W) by a word-level language model. Finally, the word string 30 having the maximum probability is output as the voice recognition result. From the above relationship, the word string ~ W of the speech recognition result for the observation sequence X can be obtained by the following equation.

This equation can be transformed and approximated as follows.

In equation (12), P (s | X) represents the score (posterior probability) of the acoustic model by CTC. α is the scaling factor. The pronunciation sequence s and the observation sequence X must satisfy the constraint of Eq. (9). The Viterbi algorithm is used for the approximation of Eq. (12). When learning the RNN, P (W | s) is calculated over all s according to the second equation of the equation (12), but when decoding, it is often approximated as in the third equation.

In equation (12), P (W | s) can be calculated by the following equation (13).

In equation (13), P (s) is the language model probability in subword units, and β is its scaling factor. P (s) can be calculated in the same way as the conventional language model. That is, it can be realized by either an N-gram language model or a neural network. However, the language model for each subword needs to be learned with the subword corpus. A subword corpus can be easily realized by converting a word into a subword with respect to a normal text corpus.

The first term of the numerator in equation (13), P (s | W), indicates the word-subword conversion probability. The word-to-subword conversion is often a one-to-one conversion (eg, breaking a word into letters). In such a case, P (s | W) becomes 1, and Eq. (13) is simplified as in Eq. (14) below.

The above can be summarized as follows. Substituting the right-hand side of equation (13) into P (W | s) of equation (12) gives the following equation (15). The hypothesis score is calculated according to this equation (15), and the hypothesis with the best score is selected as the speech recognition result.

After all, in the conventional method using RNN, the recognition score is calculated by interpolating the posterior probability output by RNN and the language model probability as shown in equations (6) to (9). On the other hand, in the method according to the present embodiment, as shown in Eq. (15), the word-subword conversion probability P (s | W) related to a certain hypothesis, the word obtained from the same word-level language model as before. The hypothetical score is calculated by dividing the product of the language model score P (W) and the subword posterior probability P (s | X) ^{α output by the RNN by the probability P (s) β} obtained from the language model at the subword level. To do. This score is calculated for each hypothesis, and the hypothesis that gives the best score is selected as the speech recognition result. This method is called maximum a posteriori (MAP) method decoding in the sense that it maximizes the posterior probability output by the RNN.

In the learning of CTC-AM in the above equation, the target function F ^CTC (θ) (θ is a parameter including the input / output weight matrix and bias value of each node constituting CTC-AM) expressed by the following equation. Find the parameter set θ that maximizes the set).

この式において、s_uはu番目の学習音声に対する正解サブワード列、X_uはu番目の学習音声、Pr_θはパラメータセットθのもとでCTC-AMが出力するスコアを表す。この条件でCTC-AMの出力におけるsoftmax関数の活性化関数値は以下の式により計算される。

In this equation, s _u is the correct subword sequence for the u-th learning voice, X _u is the u-th learning voice, and Pr _θ is the score output by CTC-AM under the parameter set θ. Under this condition, the activation function value of the softmax function in the output of CTC-AM is calculated by the following formula.

上式は、フォワード‐バックワードアルゴリズムを用いて効率的に計算できることが知られており、NNパラメータセットの誤差逆伝搬法による学習に用いられている。

It is known that the above equation can be calculated efficiently using the forward-backward algorithm, and is used for learning the NN parameter set by the error back propagation method.

^{By the way, maximizing this F CTC} with respect to MAP method decoding can be said to be optimizing CTC-AM itself. However, in reality, CTC-AM is combined with a language model to decode speech, so optimizing CTC-AM does not necessarily maximize the word recognition rate. Therefore, in the present embodiment, after learning to maximize ^{F CTC} , CTC-AM is further learned to maximize the ^{target function FM BR represented by the following equation.}

The following equation (21) is obtained by differentiating the ^{F MBR} by y _t ^{t (c).}

式(22)はフォワード・バックワードアルゴリズムを用いて効率的に計算できる。 _{As a result, the error signal regarding the activation function value a u} ^t (c) of the softmax layer of the final layer is calculated as follows.

Equation (22) can be calculated efficiently using the forward-backward algorithm.

A voice recognition device 280 using CTC-AM learned by the method according to the present embodiment will be described with reference to FIG. The voice recognition device 280 has a function of performing voice recognition on the input voice 282 and outputting it as voice recognition text 284. The voice recognition device 280 is an A / D conversion circuit 300 that performs analog / digital (A / D) conversion on the input voice 282 and outputs it as a digital signal, and a digitized A / D conversion circuit 300 that outputs the data. A predetermined signal processing is performed on each frame output by the framing processing unit 302 that frames the audio data by using a window that partially overlaps with a predetermined length and a predetermined shift amount, and the framing processing unit 302. This includes a feature amount extraction unit 304 that extracts the voice feature amount of the frame and outputs the feature amount vector. Information such as the relative time to the beginning of the input voice 282 is attached to each frame and the feature amount vector. As the audio feature quantity, MFCC (Mel-Frequency Cepstrum Coefficient), its first-order differential, second-order differential, power, etc. are used, but the output of the filter bank may be used as it is as the feature quantity. The observation series is composed of the feature vectors obtained in time series.

Further, the voice recognition device 280 receives the feature amount storage unit 306 for temporarily storing the feature amount vector output by the feature amount extraction unit 304 and the feature amount vector stored in the feature amount storage unit 306 as inputs at each time. An acoustic model 308 consisting of an End-to-End type RNN (CTC-AM) based on CTC, which outputs a vector showing the posterior probability corresponding to a certain phoneme in each frame, and a vector output by the acoustic model 308. It includes a decoder 310 for outputting a word string having the highest score (probability) as a speech recognition text 284 corresponding to an input speech 282. The vector element output by the acoustic model 308 is a value indicating the probability that the frame is each phoneme for each phoneme. Label string candidates can be obtained in lattice format by selecting each phoneme for each frame from this vector obtained in time series, connecting them with posterior probabilities, and representing each phoneme with a corresponding label. This label column candidate may also include an empty label φ. The posterior probabilities of each label string candidate can be calculated from the posterior probabilities of phonemes on each path of the lattice that make up the label string candidate.

The decoder 310 uses the posterior probabilities of the label string candidates calculated by the acoustic model to calculate a plurality of hypotheses that can be represented by the input observation series together with those probabilities, outputs them as hypotheses with a recognition score, and recognizes them. Based on the score, the hypothesis with the highest score (probability) is output as voice recognition text 284.

The number of nodes in the input layer of the RNN constituting the acoustic model 308 according to the present embodiment matches the number of elements of the input vector (observation vector). The number of nodes in the output layer of the RNN matches the number of subwords in the target language plus one. That is, the nodes of the output layer represent each subword (for example, a phoneme) of the acoustic model by HMM and the empty label φ. To each node of the output layer, the probability that the voice input at a certain time is a subword (including an empty label) represented by that node is output. Therefore, the output of the acoustic model 308 is a vector whose element is the probability that the input voice at that time is a subword represented by each node. The sum of the values of the elements of this vector is 1.

The decoder 310 calculates the probability of the candidate of the word string W for each element of the vector output by the acoustic model 308, and generates a lattice while appropriately pruning the branch with a low probability, and calculates the hypothesis and the probability. Calculate the recognition score including. The decoder 310 outputs the word string having the highest recognition score (high probability of occurrence) among the finally obtained word strings as the speech recognition text 284. At this time, the decoder 310 calculates the recognition score while directly using the output of the acoustic model 308. Unlike the conventional DNN-HMM framework, it is not necessary to convert the RNN output according to the HMM output format, and recognition efficiency can be improved. In addition, the word posterior probability P (W | X) is calculated by combining the posterior probability P (s | X) obtained from the End-to-End type RNN and the probability P (W | s). Search for the hypothesis that maximizes the posterior probability P (W | X). Unlike the conventional method that uses an end-to-end type RNN that uses an interpolated score that has no theoretical basis, it is possible to improve the recognition accuracy theoretically. In addition, as a learning method for CTC-AM, we adopted a method of optimizing the parameter set so that ^{the error is minimized (so that the FM BR} is maximized) when speech recognition is performed in combination with the language model as described above. doing. Therefore, the final recognition accuracy can be further improved compared to the method that maximizes ^{F CTC.}

A learning system 350 for learning the CTC-AM364 according to the present invention will be described with reference to FIG. The learning system 350 uses the learning data storage unit 360 that stores the learning data of the CTC-AM364 and the learning data stored in the learning data storage unit 360, and when an observation sequence of the learning voice is given, the learning system 350 is used. Learning processing unit for learning (optimizing) CTC-AM364 so as to maximize ^{F CTC} shown in Eq. (16), which is the sum of the posterior probabilities of the correct subword strings of the learning data over the entire learning data. For CTC-AM364 that has been trained by the learning processing unit 362 and 362, the learning data stored in the learning data storage unit 360 is used, and when the observation sequence of the learning voice is given, the CTC-AM364 and the language model An expression that is the expected value of the word recognition accuracy consisting of the sum of the sum of the product of the posterior probability of the word string hypothesis estimated using and and the word recognition accuracy that constitutes the word string hypothesis. The MBR learning processing unit 366 for performing the above-mentioned MBR learning and the MBR learning processing unit 366 are based on the CTC-AM364 so as to further optimize the CTC-AM364 by maximizing ^{the F MBR} shown in (18). It includes a word language model 368, a phonetic language model 370, and a word pronunciation dictionary 372 to be referred to when performing learning.

Further, the learning system 350 completes the learning process of the evaluation data storage unit 376 for storing the evaluation data for evaluating the accuracy of the hypothesis by voice recognition by the CTC-AM364 and the learning process of the CTC-AM364 by the MBR learning processing unit 366 once. For each, using the evaluation data stored in the evaluation data storage unit 376, the word language model 368, the phonetic language model 370, and the word pronunciation dictionary 372, speech recognition was performed using CTC-AM364, and based on that hypothesis. , The recognition accuracy of the words that make up the hypothesis and the posterior probability of the hypothesis calculated by the language model at the time of hypothesis generation are calculated, and the product of the recognition accuracy of the words that make up the hypothesis over the entire learning speech. MBR learning based on the accuracy evaluation unit 374 for evaluating the ^{value of the target function F MBR} , which is the expected value of speech recognition accuracy by calculating the sum of, and the expected accuracy value evaluated by the accuracy evaluation unit 374. It includes a learning / evaluation control unit 378 for determining whether or not the end condition of MBR learning by the processing unit 366 is satisfied and controlling the MBR learning processing unit 366 according to the result.

FIG. 7 shows a control structure of a program that realizes learning of CTC-AM364 by the learning system 350 in a flowchart format. ^{With reference to FIG. 7, this program maximizes the F CTC} value using the training data stored in the training data storage unit 360 based on Eq. (17) (the F ^CTC value increases). Step 400 for learning by updating the parameter set of CTC-AM364, step 402 for evaluating the accuracy of CTC-AM364 for which learning was completed in step 400, and for determining the end of MBR learning. Step 404 to store the accuracy of CTC-AM364 evaluated immediately before in a storage device such as a memory (not shown), and maximize the value of the ^{target function F MBR shown in Eq. (18) for CTC-AM364 (F). (As} the value of MBR increases), the accuracy of step 406, which performs MBR learning by updating the parameter set of CTC-AM364, and CTC-AM364, which has completed MBR learning by step 406, is evaluated using the evaluation data. The evaluation results obtained in step 408 and step 408 are compared with the previous evaluation values stored in step 404, and the learning of CTC-AM364 is completed in response to whether or not the difference is equal to or less than a predetermined threshold value. This includes step 410, which selectively executes the process of returning the control to step 404 and repeating the MBR learning. That is, in the present embodiment, the learning is terminated when the voice recognition accuracy by CTC-AM364 obtained as a result of the MBR learning is slightly improved from the previous voice recognition accuracy. Of course, the learning end condition is not limited to this. For example, the learning may be terminated when the MBR learning is completed a predetermined number of times.

FIG. 8 shows the control structure of the program for initializing the CTC-AM executed in step 400 of FIG. 6 in the form of a flowchart. With reference to FIG. 8, this program includes step 440 to initialize CTC-AM364. In this step, for example, each parameter of CTC-AM364 is initialized with a random number according to a normal distribution.

The training data is divided into a plurality of batches. In the following processing, learning of CTC-AM364 is performed for each batch. That is, this program further completes step 442, which executes the process 443 for all batches, step 448, which evaluates the CTC-AM364 after learning after step 442 is completed, and the evaluation result in step 448. Includes step 450 to determine if the condition is satisfied. If the determination in step 450 is affirmative, the execution of this program ends. Otherwise control returns to step 442.

Process 443 includes step 444 that executes process 446 for each statement in the batch.

The process 446 first inputs the voice data of the sentence into the voice recognition device using CTC-AM364 to estimate the phoneme string, and the phoneme string estimated in step 460 and the phoneme label string attached to the learning voice. by comparing the preparative step 462 of calculating an error, using the error calculated in step 462, as the value of the objective function F ^MBR shown in equation (18) is increased, the back propagation method CTC-AM364 Includes step 464 and the modification of the parameter set.

[motion]
The learning of CTC-AM364 by the learning system 350 described above is performed as follows. First, the learning data including the correct answer subword string, which is a phoneme string of the learning voice and its transcription, is stored in the learning data storage unit 360. Similarly, the evaluation data including the voice and its transcription is stored in the evaluation data storage unit 376. As for the word language model 368, the phoneme language model 370, and the word pronunciation dictionary 372, those that already exist may be used, or may be created from the learning data storage unit 360. The learning data stored in the learning data storage unit 360 is divided into several batches.

First, the learning processing unit 362 learns the CTC-AM364 using the learning data stored in the learning data storage unit 360 (step 400 in FIG. 7). Specifically, referring to FIG. 8, first, each parameter of CTC-AM364 is initialized with a random number according to a normal distribution. Subsequently, the following processing is performed for each batch (step 442 in FIG. 8).

First, for the voice of a certain sentence in the batch being processed, the phoneme label string is estimated by voice recognition by CTC-AM364 (step 460). Subsequently, the error is calculated using the estimation result and the transcription of the voice (step 462). Further, this error is used to modify the parameter set of CTC-AM364 so that the value of the ^{target function F CTC becomes large (step 464).}

The above processing 446 is executed for all the statements in the batch being processed. When the processing for one batch is completed, the same processing is repeated for the next batch. When step 444 is completed for all batches of training data in this way, CTC-AM364 is evaluated in step 448 (this is called one epoch). This evaluation is performed not by the accuracy evaluation unit 374 shown in FIG. 6 but by the learning processing unit 362, and the error of the result of voice recognition of the evaluation data (not shown) by the learning processing unit 362 is integrated over the entire evaluation data to obtain the accuracy. Is obtained by calculating. In the present embodiment, if the difference between this accuracy and the accuracy obtained in the previous process is equal to or greater than the threshold value, the same training process for CTC-AM364 is repeated again using the entire training data. The initial learning of CTC-AM364 ends when the difference in accuracy becomes less than the threshold value.

When the initial learning of CTC-AM364 is completed, the MBR learning processing unit 366 performs MBR learning for CTC-AM364 (step 406 in FIG. 7). In the present embodiment, the learning data stored in the learning data storage unit 360 is also used for this learning.

Specifically, with reference to FIG. 9, processing 482 is executed for each learning voice included in the learning data storage unit 360 (step 480). In the process 482, the CTC-AM364 is used as an acoustic model, the word language model 368, the phoneme language model 370, and the word pronunciation dictionary 372 are used to perform speech recognition on the speech data to be processed, and a lattice consisting of the speech recognition hypothesis is created. (Step 510). In this lattice, the error calculation is performed according to the above-mentioned equation (19) (step 512). Using this error, to CTC-AM364, modifies the parameter set of CTC-AM364 as the value of the objective function F ^MBR is increased by backpropagation (step 514). This process is executed for all voice data (this is also referred to as one epoch as in the process by the learning processing unit 362). When one epoch is completed, the accuracy of CTC-AM364 is evaluated in step 484. This evaluation is performed by the accuracy evaluation unit 374 of FIG. 6 using the evaluation data stored in the evaluation data storage unit 376, the word language model 368, the phoneme language model 370, and the word pronunciation dictionary 372. The evaluation itself of CTC-AM364 is the same as that performed by the learning processing unit 362.

Subsequently, in step 486, it is determined whether or not the MBR learning end condition is satisfied (by the learning / evaluation control unit 378 of FIG. 6). Specifically, it is determined in step 486 whether or not the difference between the accuracy evaluated in step 484 and the previous accuracy is less than the threshold value. If the judgment is affirmative, MBR learning for CTC-AM364 is complete. If the judgment is negative, that is, if the difference between the current accuracy and the previous accuracy is equal to or greater than the threshold value, the control returns to step 480, and again, MBR learning by the MBR learning processing unit 366 using the entire training data. Is executed for CTC-AM364.

When voice recognition is performed using the CTC-AM364 that has been learned in this way, the CTC-AM364 may be used for the acoustic model 308 of FIG.

[Experimental result]
10 and 11 show the experimental results regarding the changes in the speech recognition accuracy according to the above-described embodiment of the present invention and the speech recognition accuracy according to the conventional interpolation method with the repetition of MRB learning.

In the experiment, the Wall Street Journal (WSJ) corpus known as LDC93S6B and LDC94S13 was used as the learning corpus. The learning voice was for 77.5 hours, and the verification data was for 3.8 hours. As the CTC-AM, a phoneme-based bidirectional LSTM (BLSTM) having four hidden layers was used. Each hidden layer had 320 nodes and was trained by 120-dimensional filter bank features (40-dimensional filter bank features + Δ + ΔΔ) in which both the mean and the variance were normalized. The initial learning was performed with a learning rate of 0.00004 and a momentum parameter of 0.95. After training CTC-BLSTM-AM, a lattice was generated based on this acoustic model. At this time, a 1-gram word language model trained with a scaling factor α = 1 using the transcribed data in the training data was used. In addition, when generating the lattice by the MAP method, the bigram phoneme language model learned by converting the transcription of the learned speech into phonemes was used with β = 0.5 (Equations (13) (14) (15). reference). MBR learning was performed with 5 epochs fixed at a learning rate of 0.000001 and a momentum parameter of 0.9.

In the evaluation, the WSJ standard pruned trigram language model (pruned trigram LM) was used as the word language model. In decoding by the MAP method, a bigram phoneme language model was used. At the time of decoding, the parameters (scaling factors α and β, as well as the word insertion penalty) were adjusted by the “dev93” set in the WSJ corpus, and the best parameters were used to decode the “eval92” set in the WSJ corpus.

In FIGS. 10 and 11, the horizontal axis shows the number of repetitions of MBR learning, and the vertical axis shows the word error rate (WER) of the speech recognition result by CTC-AM at the end of each repetition. FIG. 10 is for dev93 and FIG. 11 is for eval92.

In FIG. 10, graph 530 is a graph by the conventional interpolation method, and graph 532 is based on the above embodiment. Similarly, in FIG. 11, graph 540 is based on the conventional interpolation method, and graph 542 is based on the above embodiment.

In FIGS. 10 and 11, the accuracy of the MAP method when the number of MBR repetitions = 0 represents the accuracy of CTC-AM obtained only by learning by ^{F CTC.} At this point, it can be seen that the word error rate (7.5%) of CTC by the MAP method is considerably lower than that of the interpolation method (8.5%). MBR learning improves the word error rate in both cases. However, in this case as well, the result was that the word error rate of the MAP method was consistently lower than the word error rate of the interpolation method.

That is, the accuracy of the method according to equation (15) is higher than that of the interpolation method, and the accuracy of CTC-AM is further increased by performing MBR learning for the CTC-AM. Was confirmed.

[Realization by computer]
The speech recognition device 280 and the learning system 350 according to the embodiment of the present invention can be realized by computer hardware and a computer program executed on the computer hardware. FIG. 12 shows the appearance of the computer system 630, and FIG. 13 shows the internal configuration of the computer system 630.

With reference to FIG. 12, the computer system 630 includes a computer 640 having a memory port 652 and a DVD (Digital Versatile Disk) drive 650, a keyboard 646, a mouse 648, and a monitor 642.

With reference to FIG. 13, the computer 640 includes a CPU (central processing unit) 656, a CPU 656, a bus 666 connected to the memory port 652 and the DVD drive 650, and a boot program, in addition to the memory port 652 and the DVD drive 650. It includes a read-only memory (ROM) 658 for storing and the like, a random access memory (RAM) 660 connected to the bus 666 and storing program instructions, system programs, work data, and the like, and a hard disk 654. The computer system 630 further includes a network interface (I / F) 644 that provides a connection to a network 668 that allows communication with other terminals.

The computer program for making the computer system 630 function as each functional unit of the voice recognition device 280 and the learning system 350 according to the above-described embodiment is stored in the DVD 662 or the removable memory 664 mounted on the DVD drive 650 or the memory port 652. And then transferred to the hard disk 654. Alternatively, the program may be transmitted to the computer 640 via the network 668 and stored on the hard disk 654. The program is loaded into RAM 660 at run time. Programs may be loaded directly into RAM 660 from DVD 662, from removable memory 664, or via network 668.

This program includes an instruction sequence including a plurality of instructions for causing the computer 640 to function as each functional unit of the voice recognition device 280 and the learning system 350 according to the above embodiment. Some of the basic functions required to cause the computer 640 to perform this operation are operating systems or third-party programs running on the computer 640 or various dynamically linkable programming toolkits or programs installed on the computer 640. Provided by the library. Therefore, the program itself does not necessarily have to include all the functions necessary to realize the systems, devices and methods of this embodiment. This program is a system described above by dynamically calling at runtime the appropriate function or programming toolkit or appropriate program in a program library of instructions in a controlled manner to obtain the desired result. It only needs to include instructions that implement the function as a device or method. Of course, the program alone may provide all the necessary functions.

In the above embodiment, in the learning of CTC-AM, learning that maximizes the objective function is performed. However, the present invention is not limited to such embodiments. For example, instead of such an objective function, a loss function may be determined and learning may be performed to maximize the value of the loss function.

In the above experiment, a CTC-AM having LSTM as a component was used. However, as will be apparent to those skilled in the art, CTC-AM is not limited to those using LSTM. For example, the target may be expanded to RNN in general, or CNN may be used. Further, in the above embodiment, in both the learning by the learning processing unit 362 and the learning by the accuracy evaluation unit 374, the difference between the accuracy of CTC-AM after learning and the accuracy before learning is less than a predetermined value. It is a condition. However, the present invention is not limited to such embodiments. For example, in either or both of the above-mentioned learnings, it is possible to set the number of repetitions to a fixed value and end the learning when the number of repetitions of learning reaches that value.

Further, in the above embodiment, the value represented by the equation (19) is used as a measure for expressing the accuracy of the word string W. However, the present invention is not limited to such embodiments. For example, among the lattice paths obtained by voice-recognizing the evaluation data using CTC-AM, the average of the probabilities obtained for those passing through the word W is adopted as a measure of the accuracy of the word string W. You may. Alternatively, this value may be divided by the probabilities of all Lattice passes.

The embodiments disclosed this time are merely examples, and the present invention is not limited to the above-described embodiments. The scope of the present invention is indicated by each claim of the scope of claims, taking into consideration the description of the detailed description of the invention, and all changes within the meaning and scope equivalent to the wording described therein. Including.

30 Word sequence 32 Pronunciation sequence 34 State sequence 36 Observation sequence 70 DNN
100 RNN
110 Phoneme sequence 112 Pronunciation sequence (subword sequence)
280 Speech Recogniser 282 Input Speech 302 Framed Processing Unit 304 Feature Extraction Unit 306 Feature Storage 308 Acoustic Model 310 Decoder 350 Learning System 362 Learning Processing Unit 364 CTC-AM
366 MBR Learning processing unit 374 Accuracy evaluation unit 378 Learning / evaluation control unit 630 Computer system 640 Computer 654 Hard disk 656 CPU
658 ROM
660 RAM

Claims (6)

An acoustic model learning device that learns an acoustic model based on an end-to-end neural network to calculate the probability that the observation sequence is an arbitrary subword string when a speech observation sequence is given. There,
The learning device of the acoustic model is a word-level language model that stores learning data consisting of an aligned pair of an observation sequence of the learning voice and a correct subword string corresponding to the learning voice, and the frequency of occurrence of the word string. Used by connecting to a computer-readable storage means to store
The observation sequence of the learning voice is given by the word-subword conversion probability, the language model score obtained from the language model, and the subword posterior probability output by the End-to-End type neural network for each hypothesis. As a first optimization means for optimizing the End-to-End type neural network so that the sum of the posterior probabilities of the correct subword strings of the training data over the entire training data is maximized. ,
When the observation sequence of training speech is given, as the expected value of the accuracy of the word sequence hypothesis estimated using the previous SL End-to-End neural network and the language model is maximized, the first An acoustic model learning device including a second optimizing means for further optimizing the End-to-End type neural network optimized by the optimizing means of 1. The second optimization means is
A speech recognition means that generates a hypothesis of a word string by performing speech recognition on the observation sequence using the End-to-End type neural network and the language model over the entire learning speech.
A first calculation means for calculating the recognition accuracy for the word string constituting the hypothesis based on the hypothesis and the correct subword string of the learning data over the entire learning voice.
The expected value is calculated by calculating the sum of the products of the posterior probability of the hypothesis calculated by the language model at the time of generating the hypothesis and the recognition accuracy of the word strings constituting the hypothesis over the entire learning voice. A second calculation method for calculating the value and
An update means for updating the parameter set of the acoustic model so that the expected value calculated by the second calculation means increases.
A determination means that executes a determination process regarding whether or not the end condition is satisfied in response to the completion of updating the parameter set of the acoustic model by the update means.
The first process of ending the learning of the End-to-End type neural network in response to the determination by the determination means, the generation of the hypothesis using the learning voice, the calculation of the recognition accuracy, and the expected value. The voice recognition means, the first calculation means, the second calculation means, the update means, and the second determination means are controlled so that the calculation, the update of the parameter set, and the determination process are performed again. The acoustic model learning device according to claim 1, further comprising a control means for selectively executing the processing of the above. The observation sequence is prepared in units of voice signals representing the learning voice.
In the first calculation means, each subword of the hypothetical word string output by the End-to-End type neural network is one with each subword of the subword string paired with the input observation sequence in frame units. The acoustic model learning apparatus according to claim 2, further comprising a subword matching number calculating means for calculating the number of operations. The determination means includes a hypothesis generation process for the entire learning voice by the voice recognition means, a recognition accuracy calculation process by the first calculation means, a sum calculation process by the second calculation means, and the above. The acoustic model according to claim 2 or 3, further comprising means for determining that the termination condition is satisfied when the update means updates the parameter set of the acoustic model a predetermined number of times. Learning device. The determination means responds to the difference between the latest processing value and the previous processing value of the parameter set defining the End-to-End type neural network becoming less than the threshold value. The acoustic model learning device according to claim 2 or 3, further comprising means for determining that the termination condition is satisfied. A computer program that functions to operate a computer as the means according to any one of claims 1 to 5.

Images