JP6674706B2 - Program, apparatus and method for automatically scoring from dictation speech of learner - Google Patents

Description

  The present invention relates to a technique of a speaking test that automatically scores from a dictation voice of a learner.

  For a description problem (for example, a translation problem) such as language learning, the grader compares the answer sentence by the learner with the correct answer sentence, and scores according to the degree of coincidence. However, human scoring is not only time-consuming and costly, but can also be influenced by arbitrary judgment. Therefore, the description problem itself becomes such that it is easy for a person to score.

  2. Description of the Related Art Conventionally, there is a technique for automatically scoring an answer sentence by a learner for a description problem (for example, see Patent Document 1). According to this technology, the E-Learning system and the language processing system are connected. The E-Learning system gives a question to a learner and transfers the answer sentence to a language processing system. The language processing system linguistically compares the answer sentence and the correct answer sentence, and responds to the E-learning system with the degree of coincidence. Thereby, the E-learning system gives a score according to the degree of matching.

  Further, for example, there is a technique for evaluating the translation ability of a translation agent while allowing a variety of expressions in an answer sentence of a learner for a translation problem (for example, see Patent Document 2). According to this technology, even if the answer sentence and the correct answer sentence are described in different expressions, their translation ability can be properly evaluated. Specifically, the correct answer sentence for the source language test sentence and the correct answer sentence for the source language reference sentence similar to the source language test sentence are each compared with the answer sentence for the source language test sentence to calculate the translation correct answer rate. I do.

  These conventional techniques are effective when the degree of freedom of the contents of the answer to the description problem is low, for example, in a question-and-answer format.

JP 2006-244003 A JP 2004-019131 A JP 2002-544570A

Shyamaa E. Sorour, Kazaumasa Goda and Tsunemori Mine, “Student performance Estimation Based on Topic Models Considering a Range of Lessons,” Proc. Of AIED2015 pp.790-793, 2015. Quoc Le, Tomas Mikolov, "Distributed Representations of Sentences and Documents", [online], [Search July 16, 2016], Internet <URL: http://cs.stanford.edu/~quocle/paragraph_vector.pdf > Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu and Christian Hadiwinoto, Joel Tetreault, "The CoNLL-2013 Shared Task on Grammatical Error Correction", [online], [July 16, 2016 search], Internet <URL: http : //www.comp.nus.edu.sg/~nlp/conll13st/CoNLLST01.pdf> Kazumi Aizawa, Shinichiro Ishikawa, Toshio Murata, Tatsuo Iso, Toshihiko Uemura, Takahiro Ogawa, Shinichi Shimizu, Naoki Sugimori, Akihiko Hai, Masamichi Mochizuki, "JACET8000 English Words", [online], [July 16, 2016 Search], the Internet <URL: http://iss.ndl.go.jp/books/R100000002-I000008184038-00> Yukio Tono, "Creating and Using the CAN-DO List English Achievement Index CEFR-J Guidebook", [online], [Search on July 16, 2016], Internet <URL: http: //www.taishukan. co.jp/book/b197158.html> Julius, [online], [Search July 8, 2016], Internet <URL: http://julius.osdn.jp/> Generalized Word Posterior Probability (GWPP) for Measuring reliability of Recognized Words, Frank K. Soong et al. Proc. SWIM 2004. [online], [Search September 5, 2016], Internet <URL: file: /// C: /Users/hayah/Desktop/SoongSWIM2004.pdf>

  However, according to the above-described conventional technology, it cannot be simply applied to a speaking test for automatically giving a score to a dictation voice of a learner. As the speaking test has a higher degree of freedom in conversation, it is more difficult to prepare a correct answer in advance, and a problem arises in that the accuracy of automatic scoring cannot be obtained.

  Further, even if the degree of linguistic agreement between the answer sentence and the correct answer is low, it may be necessary to determine that the meaning of the answer sentence of the learner is close to the meaning of the correct answer.

  Furthermore, according to the speaking test, not only the answer error in the learner but also the recognition error in the speech recognition system may be mixed. In this case, it is necessary to acquire the recognition error accuracy in the speech recognition system in advance by inputting the sample speech to the speech recognition system (for example, see Patent Document 3).

  Accordingly, an object of the present invention is to provide a speaking test program, an apparatus, and a method capable of improving the automatic scoring accuracy of a dictation voice of a learner.

According to the present invention, a speaking test program that allows a computer to input a dictation voice that is a foreign language for a learner and output a scoring result,
As a learning stage, input dictation voice and teacher data in which the grading results are associated,
And spoken audio or et speech recognized dictated text training data, and voice recognition engine that outputs the reliability of the speech recognition result,
And linguistic features from dictation text training data, a feature amount extracting section which extracts a voice feature amount from the internal speech recognition engine,
A feature amount selection engine that selects a feature amount so as to include a linguistic feature amount more than a speech feature amount as the reliability output from the speech recognition engine is higher ,
A feature quantity selected by the feature quantity selection engine, in association with the rating result of the training data to function as a scoring engine learns,
As a grading stage, input the dictation voice of the learner,
The speech recognition engine outputs the dictation text speech-recognized from the learner's dictation speech and the reliability of the speech recognition result,
The feature amount extracting means extracts a linguistic feature amount from the dictation text based on the learner's dictation voice and a speech feature amount from inside the speech recognition engine,
The feature selection engine outputs the feature selected according to the reliability output from the speech recognition engine,
The scoring engine inputs the feature amount selected by the feature amount selection engine and outputs a scoring result based on the dictation voice of the learner.
The computer is caused to function as described above .

According to another embodiment of the speaking test program of the present invention,
The feature selection engine causes the computer to select a feature amount such that the lower the reliability output from the speech recognition engine is, the larger the speech feature amount is than the linguistic feature amount. Is also preferred.

According to another embodiment of the speaking test program of the present invention,
As a learning stage,
And further functioning as an utterance environment synthesizing unit that outputs one or more dictation sounds obtained by synthesizing different utterance environment sounds with the dictation voice of the teacher data;
The speech recognition engine outputs the dictation text and reliability recognized from the dictation voice without environmental sound synthesis, and the dictation text and reliability recognized from the dictation voice with environmental sound synthesis ,
The feature extraction means includes linguistic features from dictation text based on dictation speech without environmental sound synthesis, speech features from inside a speech recognition engine based on dictation speech without environment sound synthesis, and presence of environment sound synthesis. Linguistic features from dictation text based on dictation speech and speech features from inside a speech recognition engine based on dictation speech with environmental sound synthesis,
For the linguistic feature based on the dictation voice without environmental sound synthesis, a linguistic feature based on the dictation voice with environmental sound synthesis, in which the difference is smaller than a predetermined threshold or becomes a predetermined number in the order of smaller differences, is used. The voice based on the dictation voice with environmental sound synthesis, which is detected and has a predetermined difference with respect to the vocal feature based on the dictation voice without environmental sound synthesis, the difference being smaller than a predetermined threshold value or a predetermined number in the order of smaller differences. Further functioning as a feature amount difference detecting means for detecting a characteristic feature amount,
The feature selection engine may also cause the computer to function so as to learn the linguistic feature and the speech feature output from the feature difference detecting means in association with the reliability output from the speech recognition engine. preferable.

According to another embodiment of the speaking test program of the present invention,
It is also preferable that the speech environment synthesizing means causes the computer to function so as to synthesize different noises as speech environment sounds.

According to another embodiment of the speaking test program of the present invention,
The linguistic features are the total number of words based on the dictation text,
Different word count,
Bag-of-Words space vector,
Bag-of-ngram space vector Dimensional vector of LSA (Latent Semantic Analysis),
LDA (Latent Dirichlet Allocation) dimension vector,
Distributed expression vector,
The number and / or type of grammatical errors,
One or more in the number of words by difficulty,
Speech features are utterance time based on dictated speech,
Words per unit time,
Acoustic likelihood,
It is also preferable to have the computer function as one or more phonemes per unit time.

According to the present invention, there is provided a speaking test device that inputs a dictation voice that is a foreign language for a learner and outputs a scoring result,
As a learning stage, input dictation voice and teacher data in which the grading results are associated,
A speech recognition engine that outputs dictation text that is speech- recognized from the dictation speech of the teacher data and reliability for the speech recognition result;
And linguistic features from dictation text training data, a feature amount extracting section which extracts a voice feature amount from the internal speech recognition engine,
A feature amount selection engine that selects a feature amount so as to include a linguistic feature amount more than a speech feature amount as the reliability output from the speech recognition engine is higher ,
A feature quantity selected by the feature quantity selection engine, the scoring engine learns associates the scoring result of the training data
And
As a grading stage, input the dictation voice of the learner,
The speech recognition engine outputs the dictation text speech-recognized from the learner's dictation speech and the reliability of the speech recognition result,
The feature extraction means extracts a linguistic feature from the dictation text based on the learner and a speech feature from inside the speech recognition engine,
The feature selection engine outputs the feature selected according to the reliability output from the speech recognition engine,
The scoring engine is characterized by inputting the feature amount selected by the feature amount selection engine and outputting a scoring result based on the learner .

According to another embodiment of the speaking test apparatus of the present invention,
As a learning stage,
And further functioning as an utterance environment synthesizing unit that outputs one or more dictation sounds obtained by synthesizing different utterance environment sounds with the dictation voice of the teacher data;
The speech recognition engine outputs the dictation text and reliability recognized from the dictation voice without environmental sound synthesis, and the dictation text and reliability recognized from the dictation voice with environmental sound synthesis ,
The feature extraction means includes linguistic features from dictation text based on dictation speech without environmental sound synthesis, speech features from inside a speech recognition engine based on dictation speech without environment sound synthesis, and presence of environment sound synthesis. Linguistic features from dictation text based on dictation speech and speech features from inside a speech recognition engine based on dictation speech with environmental sound synthesis,
For the linguistic feature based on the dictation voice without environmental sound synthesis, a linguistic feature based on the dictation voice with environmental sound synthesis, in which the difference is smaller than a predetermined threshold or becomes a predetermined number in the order of smaller differences, is used. The voice based on the dictation voice with environmental sound synthesis, which is detected and has a predetermined difference with respect to the vocal feature based on the dictation voice without environmental sound synthesis, the difference being smaller than a predetermined threshold value or a predetermined number in the order of smaller differences. Further functioning as a feature amount difference detecting means for detecting a characteristic feature amount,
It is also preferable that the feature selection engine learns the linguistic feature and the speech feature output from the feature difference detecting means in association with the reliability output from the speech recognition engine.

According to the present invention, there is provided a speaking test method for a device that inputs a dictation voice that is a foreign language for a learner and outputs a scoring result,
The equipment is
As a learning stage, input dictation voice and teacher data in which the grading results are associated,
A first step of outputting a dictation text speech- recognized from the dictation speech of the teacher data using the speech recognition engine and a reliability for the speech recognition result;
And linguistic features from dictation text training data, a second step of extracting a voice feature amount from the internal speech recognition engine,
A third step of selecting a feature using the feature selection engine such that the higher the reliability output in the first step, the more the linguistic feature is included than the vocal feature;
Using scoring learning engine, and a feature quantity selected by the third step, a fourth step of learning associates the scoring result of the training data
Run
As a grading stage, input the dictation voice of the learner,
A fifth step of outputting, using a speech recognition engine, the dictation text speech-recognized from the dictation speech of the learner and the reliability of the speech recognition result;
A sixth step of extracting linguistic features from the dictation text based on the learner and speech features from inside the speech recognition engine;
A seventh step of using the feature selection engine to output a feature selected according to the reliability output from the fifth step;
An eighth step of inputting the feature amount selected in the seventh step using a scoring engine and outputting a scoring result based on the learner;
Is performed .

According to another embodiment of the speaking test method of the present invention,
The device, as a learning phase,
Outputting one or more dictation voices in which different utterance environment sounds are synthesized with the dictation voice of the teacher data;
For the first step, the dictation text and the reliability that are speech-recognized from the dictation speech without environmental sound synthesis using the speech recognition engine, and the dictation text and the reliability that are speech-recognized from the dictation speech with environmental sound synthesis are used. And output
For the second step,
Linguistic features from dictation text based on dictation speech without environmental sound synthesis, vocal features from inside a speech recognition engine based on dictation speech without environmental sound synthesis, and dictation based on dictation speech with environmental sound synthesis Extract linguistic features from text and speech features from inside the speech recognition engine based on dictation speech with environmental sound synthesis,
For the linguistic feature based on the dictation voice without environmental sound synthesis, a linguistic feature based on the dictation voice with environmental sound synthesis, in which the difference is smaller than a predetermined threshold or becomes a predetermined number in the order of smaller differences, is used. The voice based on the dictation voice with environmental sound synthesis, which is detected and has a predetermined difference with respect to the vocal feature based on the dictation voice without environmental sound synthesis, the difference being smaller than a predetermined threshold value or a predetermined number in the order of smaller differences. Detects characteristic features,
In the third step, a linguistic feature and a speech feature output from the second step are associated with the reliability output from the speech recognition engine in the first step by using a feature selection engine. It is also preferable to attach and learn.

  ADVANTAGE OF THE INVENTION According to the speaking test program, apparatus, and method of this invention, the automatic scoring precision with respect to the dictation voice of a learner can be improved.

FIG. 4 is a functional configuration diagram at a grading stage in the speaking test program of the present invention. FIG. 5 is an explanatory diagram illustrating a feature amount extracted using a speech recognition engine. It is a functional block diagram of the learning stage of the scoring engine in the speaking test program of the present invention. It is a functional block diagram of a learning stage of the feature-value selection engine in the speaking test program of the present invention. It is a sequence diagram in this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Scoring stage>
FIG. 1 is a functional block diagram of the speaking test program of the present invention at the scoring stage.

  The “scoring stage” is a process of inputting a dictation voice of a learner and automatically outputting a scoring result using a scoring engine. For example, the process is performed by a terminal owned by the learner.

  In the case of a speaking test program in language learning, the dictation voice of the learner is in another language (eg, English) different from the native language (eg, Japanese) for the learner. According to the present invention, for example, an answer sentence thought by a Japanese person can be pronounced in English, and the dictation voice can be automatically scored.

  The speaking test program 1 shown in FIG. 1 causes a computer to function as a speech recognition engine 11, a feature extraction unit 12, a feature selection engine 13, and a scoring engine 14. The flow of processing of these functional components can be understood as a speaking test apparatus and method.

[Speech recognition engine 11]
The speech recognition engine 11 outputs “dictation text” recognized from the learner's dictation voice and “reliability” for the recognition result. The dictation text is output to the feature extraction unit 12.
As a speech recognition engine, for example, there is Julius (registered trademark) that can execute continuous speech recognition of tens of thousands of vocabulary words in real time (for example, see Non-Patent Document 6). This speech recognition engine is composed of an "acoustic model" (a model representing acoustic features) using a GMM-HMM (Gaussian Mixture Model-Hidden Markov Model) or a DNN-HMM (Deep Neural Network-Hidden Markov Model), and a word. A "language model" (a model representing language connection) using an N-gram, a description grammar, and a word dictionary is mounted. These language model and acoustic model modules can be rearranged according to the purpose of the speaking test.

The acoustic model expresses the frequency characteristics of each phoneme, and a hidden Markov model is generally used.
The language model expresses restrictions on how words are arranged. For example, there is a restriction that the word "ga" or "ha" is likely to follow immediately after "I (watashi)".

[Feature Extraction Unit 12]
The feature amount extraction unit 12 extracts a plurality of feature amounts from the dictation text based on the learner and / or from inside the speech recognition engine.

  FIG. 2 is an explanatory diagram illustrating a feature amount extracted from the speech recognition engine.

The feature quantity extracted from the dictation text is a “linguistic feature quantity”.
The feature amount extracted from the inside of the speech recognition engine is a “speech feature amount”.
These features are output to the feature selection engine 14.

<Linguistic features>
The linguistic features are the total number of words based on the dictation text,
Different word count,
Bag-of-Words space vector
Bag-of-ngram space vector Dimensional vector of LSA (Latent Semantic Analysis),
LDA (Latent Dirichlet Allocation) dimension vector,
Distributed expression vector,
The number and / or type of grammatical errors,
It is one or more in the number of words for each difficulty level.

The “number of total words” refers to the number of all words (vocabulary) included in the dictation text as the answer.
"Different word count" refers to the number of words counted as one for the same word. Regardless of the usage form, they are all considered the same.
“Bag-of-Words space vector” refers to a vector expressing only the appearance frequency of each word included in the text. Here, the order of appearance of the words is ignored. The space vector is obtained by positioning the text at one point in the space, with the word as an axis and the appearance frequency as a value. In addition, the similarity between sentences is derived using the previously derived IDF (Inverse Document Frequency) as the weight of the word.
“Bag-of-ngram space vector” refers to a vector of a framework including Bag-of-Words where the number of elements is n = 1. The expression differs depending on what the consecutive n elements represent.
The “dimensional vector of LSA (Latent Semantic Analysis)” is based on latent semantic analysis and refers to a dimensional compression vector generated from a document group and a term group included therein (for example, see Non-Patent Document 1). . According to LSA, a document-word matrix representing the appearance of terms for each document is used. This is a sparse matrix in which each row corresponds to each word and each column corresponds to each document. TF-IDF (Term Frequency-Inverse Document Frequency) is used for weighting each component of this matrix. Each element of the matrix is a value proportional to the number of times the word has been used in the document, and the words are weighted to reflect their relative importance.
The “dimensional vector of LDA (Latent Dirichlet Allocation)” refers to a dimensionally compressed vector that stochastically generates a “topic” of a word in a document (for example, see Non-Patent Document 1). Specifically, the text is represented by the likelihood (topic ratio) belonging to each topic group. Words do not exist independently, but can be classified into any one topic group, and words in the same topic group are likely to appear in the same sentence. It is.
“Distributed representation” refers to a real number vector expressing a word in a text in a high dimension (for example, see Non-Patent Document 2). A word having a closer meaning is associated with a closer vector. It has additive constructivity, and the addition of vectors will correspond to the addition of meaning. For example, there is a tool like doc2vec. By using such a tool, the meaning of the minute can be represented by a hundred-dimensional vector.
According to “the number and / or type of grammatical error locations”, the score tends to decrease as the number of grammatical error locations increases (for example, see Non-Patent Document 3).
The “number of words for each difficulty level” is obtained by counting words for each difficulty level using a vocabulary list with a difficulty level (for example, see Non-Patent Documents 4 and 5).

<Speech feature>
Speech features are utterance time based on dictated speech,
Words per unit time,
Acoustic likelihood,
It may be one or more in the number of phonemes per unit time.

The “utterance time” is the time of the dictation voice that is the answer.
The “number of words per unit time” refers to, for example, the number of words dictated by dividing the dictation sound per unit time (for example, 5 seconds), detecting the number of words per unit time, and averaging the number of words.
The “acoustic likelihood” refers to the degree of likelihood of the phoneme from a statistical viewpoint using an acoustic model. Words with higher acoustic likelihood tend to be acoustically correct, and words with lower acoustic likelihood tend to be acoustically incorrect.
The "number of phonemes per unit time" refers to the number of phonemes (speech speed) detected in a unit time (for example, 5 seconds).

  The “reliability” output by the speech recognition engine 11 is, for example, a Confidence Measure such as GWPP (Generalized Word Posterior Probability) (for example, see Non-Patent Document 7). It is determined that the lower the value of the reliability is, the higher the possibility that a speech recognition error is included.

[Feature selection engine 13]
The feature selection engine 13 learns in advance the reliability and the feature in association with each other, and selects one or more features according to the reliability output from the speech recognition engine 11. One feature amount may be selected, and when a plurality of feature amounts are selected, a linguistic feature amount and a speech feature amount may be mixed.

Here, specifically, the feature selection engine 13 selects a speech feature rather than a linguistic feature because the lower the reliability, the higher the error in speech recognition. The speech feature is a parameter that can be stably extracted with high accuracy regardless of the degree of error in speech recognition by the speech recognition engine 11. Therefore, the speech feature can be used as a robust feature against speech recognition errors.
That is, the feature quantity selection engine 13
The higher the reliability is, the more features that include more linguistic features than phonetic features,
It is preferable to select a feature value that includes more speech features than linguistic features as reliability is lower.

[Scoring engine 14]
The scoring engine 14 learns in advance the feature amount and the scoring result in association with each other, and outputs the scoring result according to the feature amount selected by the feature amount selecting unit 13. Here, the scoring engine 14 internally builds the scoring model parameters learned by itself.
The scoring result may be a continuous value such as 0 to 100 points, or may be a discrete value such as grade or pass / fail.

The scoring engine 14 uses a machine learning method such as regression analysis, multiple regression analysis, Lasso regression, Ridge regression, SVR (Support Vector Regression), or NN (Neural Net) when the scoring result is a continuous value. Can be.
If the scoring result is a discrete value, logistic regression or a machine learning method such as SVM (Support Vector Machine) or NN can be used.
One criterion for selecting a learning method of the scoring engine is whether or not linear separation is possible with teacher data.

Regression analysis refers to fitting a model between a dependent variable (object variable) Y and an independent variable (explanatory variable) X on a continuous scale in statistics (Y = f (X)). . The most basic model is Y = aX + b. If X is one-dimensional, it is called simple regression. If X is two-dimensional or more, it is called multiple regression. The multiple regression analysis is one of the multivariate analyses, and generally uses a least squares method.
Among regression analysis, linear regression includes Lasso regression and Ridge regression, and nonlinear regression includes SVR and NN.

Logistic regression is a type of statistical classification model for variables that follow a Bernoulli distribution.
Support vector regression is one type of nonlinear regression analysis called a kernel method. This is a regression version of the Support Vector Machine used in the field of pattern recognition. Support vector regression is a non-parametric model that does not assume a model in advance, and does not need to consider the distribution of data.
A neural network is a mathematical model that expresses the characteristics of brain functions by computer simulation. Artificial neurons (nodes) that form a network by synaptic connections change the synaptic connection strength through learning, and refer to all models that have problem-solving ability.

  A support vector machine is one of the pattern recognition models using supervised learning, and can be applied to classification and regression. The support vector machine forms two classes of pattern classifiers using linear input elements. The parameters of the linear input element are learned from the teacher data based on the criterion (hyperplane separation theorem) for finding a margin-maximizing hyperplane that maximizes the distance to each data point.

<Scoring engine, learning stage>
FIG. 3 is a functional configuration diagram of the scoring engine in the learning stage in the speaking test program of the present invention.

  The “scoring engine / learning stage” is processing for inputting teacher data and constructing a scoring model parameter inside the scoring engine. The scoring model parameters are incorporated into the scoring engine. In the case of a speaking test, the test is performed by, for example, a test operator.

The teacher data group input according to the present invention is obtained by associating dictation voices and scores of many learners in the past.
(Voice dictation) <-> (Scoring)
V1 <-> A1
V2 <-> A2
V3 <-> A3
・ ・ ・ ・ ・ ・ ・
The dictation voice of the teacher data group is input to the voice recognition engine 11, and the scoring result is input to the scoring engine 14.

[Speech recognition engine 11]
The speech recognition engine 11 outputs the dictation text from the dictation speech of the teacher data and the reliability of the speech recognition result. The dictation text is output to the feature extraction unit 12, and the reliability is output to the feature selection engine 13. The speech recognition engine 11 itself is exactly the same as that described above with reference to FIG.

[Feature Extraction Unit 12]
The feature amount extraction unit 12 extracts a plurality of feature amounts from the dictation text based on the teacher data and / or from inside the speech recognition engine. The plurality of feature amounts are output to the feature amount selection engine 13. The feature amount extraction unit 12 itself is exactly the same as that described above with reference to FIG.

[Feature selection engine 13]
The feature quantity selection engine 13 selects one or more feature quantities according to the reliability based on the teacher data. The feature selection engine 13 itself is exactly the same as that described above with reference to FIG.

[Scoring engine 14]
The scoring engine 14 learns the feature amount selected by the feature amount selection engine 13 in association with the scoring result of the teacher data. Thereby, a scoring model parameter is constructed inside the scoring engine 14. The scoring engine 14 itself is exactly the same as that described above with reference to FIG. 1, and the scoring model parameters constructed according to FIG. 3 are used by the scoring engine 14 in the scoring stage of FIG.

<Feature selection engine, learning stage>
FIG. 4 is a functional block diagram of the feature selection engine in the learning stage in the speaking test program of the present invention.

The “feature selection engine / learning stage” is a process of inputting the dictation voice of the teacher data and constructing a selection model parameter inside the feature selection engine. The selected model parameters are incorporated into a feature selection engine.
According to FIG. 4, in addition to the functional components of FIGS. 1 and 3, a speech environment synthesizing unit 15 and a feature difference detecting unit 16 are further provided.

[Speech environment synthesis unit 15]
The utterance environment synthesis unit 15 outputs one or more dictation voices obtained by synthesizing different utterance environment sounds with the dictation voice of the teacher data. These dictation voices are input to the voice recognition engine 11.
The utterance environment synthesizing unit 15 inputs, to the speech recognition engine 11, speech in which various noises are synthesized, even if the same dictation speech as the teacher data. As a result, if the dictation speech is the same, even if there is an error in the recognition of the dictation text output from the speech recognition engine 11, learning is performed assuming that the same grading is given. That is, in the learning stage, a selection model parameter is constructed such that even if noise in the utterance environment sound is mixed in the dictation voice of the learner, resistance to scoring is increased. As another example of the utterance environment sound, an effector that can simulate the frequency characteristics of a microphone that collects the uttered voice of the speaker, the reverberation characteristics of the room where the speaker is located, and the like can be considered.

[Speech recognition engine 11]
The speech recognition engine 11 outputs the dictation text recognized from the dictation voice without noise, the dictation text recognized from the dictation voice with noise, and the reliability of the recognition result. At this time, the dictation text of the dictation voice with noise has more misrecognition and lower reliability than the dictation text of the dictation voice without noise. The speech recognition engine 11 itself is exactly the same as that described above with reference to FIGS.

[Feature Extraction Unit 12]
The feature amount extraction unit 12 extracts one or more feature amounts from the recognized dictation text and / or from inside the speech recognition engine for each of the dictation voice without noise and the dictation voice with noise. The lower the reliability of the linguistic feature, the higher the error in speech recognition. On the other hand, speech features are less susceptible to speech recognition errors. The feature extraction unit 12 itself is exactly the same as that described above with reference to FIGS.

[Feature amount difference detection unit 16]
The feature amount difference detection unit 16 extracts a feature amount based on the dictation sound with noise, which is smaller than a predetermined threshold or becomes a predetermined number in the order of smaller differences with respect to the feature amount based on the dictation sound without noise. To detect. The predetermined threshold value or the predetermined number for the feature amount is set in advance. Here, it is intended to detect a small difference in the feature amount based on the dictation voice between the absence of noise and the presence of noise, that is, the feature amount that is not easily affected by noise. One or more detected feature amounts are output to the feature amount selection engine 13.

[Feature selection engine 13]
The feature selection engine 13 learns the feature output from the feature difference detector 16 in association with the reliability output from the speech recognition engine 11. As a result, a selection model parameter is constructed inside the feature quantity selection engine 13. The feature selection engine 13 itself is exactly the same as that described above with reference to FIGS. 1 and 3, and the selection model parameters constructed according to FIG. 4 are used in the scoring stage of FIG. 1 and the scoring engine / learning stage of FIG. Used by the feature quantity selection engine 13.

  FIG. 5 is a sequence diagram in the present invention.

  According to FIG. 5, the server operated by the speaking test operator and the terminal possessed by the learner are connected via a network. The terminal is preferably a smartphone or a tablet terminal equipped with a user interface such as a microphone and a display in advance.

According to FIG. 5A, the learning step is performed by a server operated by a speaking test provider, and the scoring step is performed by a terminal owned by the learner.
The server transmits the scoring model parameters and the selected model parameters generated in the learning stage to the terminal.
The terminal holds the received scoring model parameters and selection model parameters, and scores from the dictation voice of the learner.

According to FIG. 5B, both the learning stage and the scoring stage are executed by the server operated by the speaking test company.
The server holds the scoring model parameters and the selected model parameters generated in the learning stage.
The terminal transmits the dictation voice of the learner as it is to the server.
The server scores from the dictation voice received from the terminal, and returns the scoring result to the terminal.

  As described above in detail, according to the program, the apparatus, and the method of the present invention, it is possible to enhance the automatic scoring accuracy of the dictation voice of the learner. In particular, there is no need to prepare a correct sentence in advance even for a speaking test with a high degree of freedom in conversation.

  For the above-described various embodiments of the present invention, various changes, modifications, and omissions in the scope of the technical idea and viewpoint of the present invention can be easily performed by those skilled in the art. The foregoing description is merely an example, and is not intended to be limiting. The invention is limited only as defined by the following claims and equivalents thereof.

1 Speaking Test Program 11 Speech Recognition Engine 12 Feature Extraction Unit 13 Feature Selection Engine 14 Scoring Engine 15 Speech Environment Synthesis Unit 16 Feature Difference Detection Unit

Claims (9)

A speaking test program that allows a learner to input a dictation voice that is a foreign language and function a computer to output scoring results,
As a learning stage, input dictation voice and teacher data in which the grading results are associated,
And spoken audio or et speech recognized dictated text training data, and voice recognition engine that outputs the reliability of the speech recognition result,
And linguistic features from dictation text training data, a feature amount extracting section which extracts a voice feature amount from the internal speech recognition engine,
A feature amount selection engine that selects a feature amount so as to include a linguistic feature amount more than a speech feature amount as the reliability output from the speech recognition engine is higher ,
And the feature quantity selected by the feature quantity selection engine, in association with the rating result of the training data to function as a scoring engine learns,
As a grading stage, input the dictation voice of the learner
The speech recognition engine outputs a dictation text that is speech-recognized from the dictation speech of the learner, and a reliability for the speech recognition result,
The feature amount extracting means extracts a linguistic feature amount from a dictation text based on a learner's dictation voice and a speech feature amount from inside a speech recognition engine,
The feature amount selection engine outputs a feature amount selected according to the reliability output from the speech recognition engine,
The scoring engine inputs a feature amount selected by the feature amount selection engine and outputs a scoring result based on a dictation voice of a learner.
Speaking test program for causing a computer to function as.   The feature amount selection engine selects a feature amount such that the lower the reliability output from the speech recognition engine is, the larger the vocal feature amount is than the linguistic feature amount.
2. The speaking test program according to claim 1, wherein the computer functions as described above. As a learning stage,
And further functioning as a speech environment synthesizing unit that outputs one or more speech voices obtained by synthesizing different speech environment sounds with the speech data of the teacher data;
The speech recognition engine outputs the dictation text and reliability recognized from the dictation voice without environmental sound synthesis, and the dictation text and reliability recognized from the dictation voice with environmental sound synthesis ,
The feature amount extraction means includes: a linguistic feature amount from a dictation text based on an dictation voice without environmental sound synthesis; a speech feature amount from inside a speech recognition engine based on the dictation voice without environmental sound synthesis; Extracting linguistic features from dictation text based on dictated speech with speech and speech features from inside a speech recognition engine based on dictation speech with environmental sound synthesis,
For the linguistic feature based on the dictation voice without environmental sound synthesis, a linguistic feature based on the dictation voice with environmental sound synthesis, in which the difference is smaller than a predetermined threshold or becomes a predetermined number in the order of smaller differences, is used. The voice based on the dictation voice with environmental sound synthesis, which is detected and has a predetermined difference with respect to the vocal feature based on the dictation voice without environmental sound synthesis, the difference being smaller than a predetermined threshold value or a predetermined number in the order of smaller differences. Further functioning as a feature amount difference detecting means for detecting a characteristic feature amount,
The feature selection engine functions a computer so as to perform learning by associating the linguistic feature and the speech feature output from the feature difference detecting means with the reliability output from the speech recognition engine. The speaking test program according to claim 1 or 2, wherein the speaking test program is executed. The speaking test program according to claim 3, wherein the utterance environment synthesizing means causes a computer to function so as to synthesize different noises as utterance environment sounds. The linguistic feature quantity is the total number of words based on the dictation text,
Different word count,
Bag-of-Words space vector,
Bag-of-ngram space vector Dimensional vector of LSA (Latent Semantic Analysis),
LDA (Latent Dirichlet Allocation) dimension vector,
Distributed expression vector,
The number and / or type of grammatical errors,
One or more in the number of words by difficulty,
The speech feature amount is a speech time based on the dictated speech,
Words per unit time,
Acoustic likelihood,
The speaking test program according to any one of claims 1 to 4 , wherein the computer is caused to function as one or more phonemes per unit time. A speaking test device that inputs a dictation voice that is a foreign language for a learner and outputs a scoring result,
As a learning stage, input dictation voice and teacher data in which the grading results are associated,
A speech recognition engine that outputs dictation text that is speech- recognized from the dictation speech of the teacher data and reliability for the speech recognition result;
And linguistic features from dictation text training data, a feature amount extracting section which extracts a voice feature amount from the internal speech recognition engine,
A feature amount selection engine that selects a feature amount so as to include a linguistic feature amount more than a speech feature amount as the reliability output from the speech recognition engine is higher ,
And the feature quantity selected by the feature quantity selection engine, the scoring engine learns in association with the scoring results for the training data
And
As a grading stage, input the dictation voice of the learner
The speech recognition engine outputs a dictation text that is speech-recognized from the dictation speech of the learner, and a reliability for the speech recognition result,
The feature amount extracting means extracts a linguistic feature amount from the dictation text based on the learner and a speech feature amount from inside the speech recognition engine,
The feature amount selection engine outputs a feature amount selected according to the reliability output from the speech recognition engine,
The speaking test device , wherein the scoring engine inputs a feature amount selected by the feature amount selection engine and outputs a scoring result based on a learner . As a learning stage,
And further functioning as a speech environment synthesizing unit that outputs one or more speech voices obtained by synthesizing different speech environment sounds with the speech data of the teacher data;
The speech recognition engine outputs the dictation text and reliability recognized from the dictation voice without environmental sound synthesis, and the dictation text and reliability recognized from the dictation voice with environmental sound synthesis ,
The feature amount extraction means includes: a linguistic feature amount from a dictation text based on an dictation voice without environmental sound synthesis; a speech feature amount from inside a speech recognition engine based on the dictation voice without environmental sound synthesis; Extracting linguistic features from dictation text based on dictated speech with speech and speech features from inside a speech recognition engine based on dictation speech with environmental sound synthesis,
For the linguistic feature based on the dictation voice without environmental sound synthesis, a linguistic feature based on the dictation voice with environmental sound synthesis, in which the difference is smaller than a predetermined threshold or becomes a predetermined number in the order of smaller differences, is used. The voice based on the dictation voice with environmental sound synthesis, which is detected and has a predetermined difference with respect to the vocal feature based on the dictation voice without environmental sound synthesis, the difference being smaller than a predetermined threshold value or a predetermined number in the order of smaller differences. Further functioning as a feature amount difference detecting means for detecting a characteristic feature amount,
The feature quantity selection engine learns the linguistic feature quantity and the speech feature quantity output from the feature quantity difference detection means in association with the reliability output from the speech recognition engine. The speaking test apparatus according to claim 6 . A speaking test method for a device that inputs a dictation voice that is a foreign language for a learner and outputs a scoring result,
The device comprises:
As a learning stage, input dictation voice and teacher data in which the grading results are associated,
A first step of outputting a dictation text speech- recognized from the dictation speech of the teacher data using the speech recognition engine and a reliability for the speech recognition result;
And linguistic features from dictation text training data, a second step of extracting a voice feature amount from the internal speech recognition engine,
A third step of using a feature selection engine to select a feature such that the higher the reliability output in the first step is, the more linguistic features are included than speech-based features;
Using scoring learning engine, and the feature quantity selected by the third step, a fourth step of learning in association with the scoring results for the training data
Run
As a grading stage, input the dictation voice of the learner,
A fifth step of outputting, using the speech recognition engine, a dictation text speech-recognized from the dictation speech of the learner and a degree of reliability for the speech recognition result;
A sixth step of extracting linguistic features from the dictation text based on the learner and speech features from inside the speech recognition engine;
A seventh step of using the feature selection engine to output a feature selected according to the reliability output from the fifth step;
An eighth step of inputting the feature amount selected in the seventh step using the scoring engine and outputting a scoring result based on the learner;
Speaking test method apparatus, characterized by the execution. The device, as a learning phase,
Outputting one or more dictation sounds obtained by combining different utterance environment sounds with the dictation speech of the teacher data;
In the first step, the dictation text and reliability recognized by the dictation speech without environmental sound synthesis using the speech recognition engine, and the dictation text and reliability recognized by the dictation speech with environmental sound synthesis are used. And output
For the second step,
Linguistic features from dictation text based on dictation speech without environmental sound synthesis, vocal features from inside a speech recognition engine based on dictation speech without environmental sound synthesis, and dictation based on dictation speech with environmental sound synthesis Extract linguistic features from text and speech features from inside the speech recognition engine based on dictation speech with environmental sound synthesis,
For the linguistic feature based on the dictation voice without environmental sound synthesis, a linguistic feature based on the dictation voice with environmental sound synthesis, in which the difference is smaller than a predetermined threshold or becomes a predetermined number in the order of smaller differences, is used. The voice based on the dictation voice with environmental sound synthesis, which is detected and has a predetermined difference with respect to the vocal feature based on the dictation voice without environmental sound synthesis, the difference being smaller than a predetermined threshold value or a predetermined number in the order of smaller differences. Detects characteristic features,
In the third step, the linguistic feature and the speech feature output from the second step using the feature selection engine, and the reliability output from the speech recognition engine in the first step. 9. The method according to claim 8 , wherein the learning is performed by associating with.

Images