KR20200063281A - Apparatus for generating Neural Machine Translation model and method thereof - Google Patents

Abstract

A translation method based on neural machine translation model including an encoder and a decoder comprises: a step of separating input tokens of a sub-word unit from an input sentence to generate vocabulary sequence column including number information given to each input token according to an order to vocabularies included in the input sentences by a pre-processor; a step of generating a location embedding vector of a vector type indicating location information of each input token using the vocabulary sequence column information by a location embedding vector generator; a step of combining the location embedding vector with an attention score calculated based on a concealment state value of a top concealment layer input from the encoder and a current concealment state value input from the decoder by an attention layer block; a step of generating a weight average value used to estimate a band token with respect to each input token in the decoder using the attention score combined with the location embedding vector and the concealment state value of the top concealment layer by token of the encoder by the weight average value calculator; and a step of predicting the band token using the weight average value input from the weight average value calculator by the decoder.

Description

Neural network automatic translation device and its method {Apparatus for generating Neural Machine Translation model and method thereof}

The present invention relates to a neural network automatic translation technology.

The neural network translation (NMT) model based on the encoder-decoder mechanism is a recurrent neural network with long short term memory (RNN-LSTM) or a convolutional artificial neural network (Convolutional). Neural Network) to generate output text (translation text) for input text.

In the neural network automatic translation model using the existing encoder-decoder mechanism, the encoder converts the input vocabulary (input tokens) of the input statement into an N-dimensional single vector and outputs it, and the decoder compares the single vector output from the encoder. The method of predicting the next band vocabulary (next band token) based on the currently generated band vocabulary (band token) is performed recursively.

On the other hand, in order for the encoder to convert the input vocabularies (input tokens) of the input statement into a single N-dimensional vector, the input vocabularies of the input statement are simply sub-words without considering the correlation. After going through a pre-processing process to generate input vocabularies (input tokens) separated by a unit, it is input to the encoder.

Therefore, since the encoder generates each single vector from input vocabularies (input tokens) that do not contain any information specifying the correlation, and passes them to the decoder, the decoder is between input vocabularies (input tokens). The meaning of the vocabulary that implies interrelationships is not received correctly.

For this reason, the decoder cannot correctly predict the band vocabulary (band token) for each of the input vocabularies (input tokens). This serves as an important factor that degrades the translation quality of the band text composed of the band tokens.

The present invention generates a condition variable reflecting a correlation between input vocabularies (input tokens) constituting an input statement, and transmits the condition variable to a decoder together with the output of the encoder, so that each input vocabulary (each input An object of the present invention is to provide an apparatus for generating a neural network automatic translation model and a method for generating a correct prediction of a band token (or band vocabulary) for a token.

A neural network automatic translation model-based translation method including an encoder and a decoder according to an aspect of the present invention for achieving the above-mentioned object, in a pre-processing unit, inputs input into sub-word unit input tokens Separating and generating lexical order sequence information including number information assigned to each input token according to the lexical order of the words included in the input statement; Generating, by the location embedding vector generation unit, a location embedding vector in vector format representing location information of each input token using the lexical sequence information; In the attention layer block, combining the position embedding vector with the attention score calculated based on the hidden state value of the highest hidden layer for each token input from the encoder and the current hidden state value input from the decoder; And in the weighted average value calculation unit, Generating a weighted average value used for predicting a band token for each input token in the decoder by using the attention score combined with the location embedding vector and the hidden state value of the highest hidden layer for each token of the encoder; And in the decoder, predicting the band token using the weighted average value input from the weighted average value calculating unit.

According to the neural network auto-translation model based on the encoder-decoder mechanism of the present invention, a conditional variable reflecting a correlation between input vocabularies (input tokens) constituting an input sentence input to the encoder by the decoder along with the output of the encoder By receiving the, the decoder can improve the translation quality by correctly predicting the band vocabulary (band token) for each input vocabulary constituting the input statement based on the condition variable received along with the output of the encoder. have.

1 is a block diagram of a neural network automatic translation model for explaining the attention mechanism applied to the present invention.
2 is a block diagram of an automatic network translation model based on an encoder-decoder according to an embodiment of the present invention.
3 is a block diagram of a device equipped with a neural network automatic translation model according to an embodiment of the present invention.
4 is a flowchart illustrating a translation method of a neural network automatic translation model according to an embodiment of the present invention.

Various embodiments of the present invention may have various modifications and various embodiments, and specific embodiments are illustrated in the drawings and related detailed descriptions are described. However, this is not intended to limit the various embodiments of the present invention to specific embodiments, and should be understood to include all modifications and/or equivalents or substitutes included in the spirit and scope of the various embodiments of the present invention. In connection with the description of the drawings, similar reference numerals have been used for similar elements.

Expressions such as “include” or “can include” that may be used in various embodiments of the present invention indicate the existence of a corresponding function, operation, or component disclosed, and additional one or more functions, operations, or The components and the like are not limited. In addition, in various embodiments of the present invention, terms such as “include” or “have” are intended to designate the existence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one Or other features or numbers, steps, actions, components, parts, or combinations thereof, should not be excluded in advance.

Hereinafter, with reference to the drawings, a description of embodiments of the present invention will be described in detail. Prior to this, in order to help the understanding of the present invention, an attention mechanism (Attention Mechanism: Bahnadau et al.,) applied to an encoder-decoder mechanism and a neural automatic translation model based on the encoder-decoder mechanism of the present invention. 15, Luong et al., 15) will be described in detail.

Encoder-decoder mechanism

The neural network automatic translation model (or neural network machine translation model) of the present invention performs automatic translation based on a mechanism including an encoder and a decoder.

In the neural network automatic translation model, the encoder compresses (or abstracts) the input text composed of the original language into a single or multiple N-dimensional vectors, and the decoder performs a band language from the compressed (abstracted) single or multiple N-dimensional vectors. Generates output statement (translation result) composed of.

As is well known, many studies have been conducted on the neural network structure and learning method for implementing this mechanism, and it is now known as a core technology of services provided by companies such as Google and Naver.

The characteristics of the neural network automatic translation model based on the encoder-decoder mechanism is that the encoder compresses the input text composed of the original language to generate a single N-dimensional vector, and then the decoder builds on the encoder output and the currently generated band vocabulary information. Perform recursively to predict the next vocabulary.

It can be interpreted that the encoder compresses the input text of the original language as a series of sentence vectors including both the semantic and syntactic characteristics of the input original language.

The decoder acts as a kind of language model that generates random sentences composed of band vocabulary. That is, the decoder receives abstract information of the original language represented by the output of the encoder as a condition variable, and predicts the band vocabularies based on the method of generating the band language learned by the decoder.

The process of predicting the band vocabularies performed by the decoder is repeatedly performed until a reserved sentence ending word indicating the completion of the band sentences including the band vocabularies appears, and is determined to be the most natural among the candidate band sentences generated during the iteration process. This is a process of selectively outputting a band statement.

Since this encoder-decoder mechanism does not know whether the band vocabulary (band token) constituting the band text is generated by the input vocabulary (input token) constituting the input text (or the original text), the input text (or the original text) is not known. It has a characteristic that there is no direct relationship between the input vocabulary constituting and the band vocabulary constituting the band sentence.

In order to construct a neural network-based machine translation system having such characteristics, there is a neural network learning step of receiving a set of sentences pairs (double language corpus) composed of sentences in a original language and a band language, and learning them, and generated by the learning step There is a predictive step of performing machine translation based on the neural network model.

For neural network learning, the input vocabulary composed of the original language is cut to a specific standard, and each converted into a one-dimensional vector, which is expressed as a two-dimensional embedding vector. In this process, the input vocabulary (input token) constituting the input text used in the learning step and the band vocabulary (band token) constituting the band text are fixed. That is, when there are 1000 unique vocabularies each constituting the corpus of the original language used for learning, and when one vocabulary is expressed as a one-dimensional vector represented by 500 floating-point numbers, a two-dimensional vector of 1000 x 500 This 2D vector is the embedding vector.

When the number of unique vocabulary tokens constituting the corpus of the band language is 500, when expressed as a one-dimensional vector of the same length, it can be converted into a 500-500 two-dimensional vector.

In the neural network learning, the original vocabulary of the input text and the band vocabulary of the band text converted into an embedding vector suitable for each unique vocabulary token are processed as input information.

As described above, while expressing a specific unique vocabulary token as an embedding vector, if a vocabulary that does not appear frequently or a vocabulary that has never appeared in a bilingual corpus used for learning appears, it is regarded as a collectively reserved non-registered token, It is replaced with a single embedding vector for unregistered words. That is, if the embedding vector is viewed as a kind of dictionary, whenever a vocabulary token that is not in the dictionary appears, a specific embedding vector defined in advance is returned. This occurs when an unregistered word token is output when an untrained word sequence needs to be generated in an encoder that abstracts the original text as well as a decoder that generates a band text.

Due to the nature of the neural network automatic translation technology, the vocabulary is determined at the learning stage, and there is a problem that the determined vocabulary cannot be updated without learning with a new neural network.

In addition, various vocabularies can be processed by setting the number of vocabularies very large, but the number of vocabularies used has a problem of increasing the number of parameters handled by the neural network exponentially, so that the number cannot be increased indefinitely. .

Accordingly, the input vocabulary is cut into statistically frequent character sequences, and "sub-word units" having a length less than "words" generally considered to have meaning. You can reduce the number of word dictionaries by dropping to. The token-separation method of the sub-vocabulary unit may be utilized by the Wordpiece technique (Schuster' 12) and the BPE technique (Byte-Pair Encoding, Sennrich'15).

On the other hand, due to the characteristics of the automatic translation of the neural network using the above-described encoder-decoder mechanism, the entire sentence input to the encoder is converted into a single vector format, and thus, a band vocabulary (band token) constituting the band sentence is generated. However, there are cases where the meaning of individual vocabulary cannot be conveyed correctly.

To solve this, whenever a band token is generated using the attention mechanism (Attention Mechanism: Bahnadau et al., '15, Luong et al., 15), a condition in which a specific input token in the input text is weighted The variable is passed to the decoder. At this time, since the weight is transmitted to the decoder based on the unit of the input token, the weight vector for the partial vocabulary token is transmitted instead of the entire vocabulary in the form in which the token is divided into sub-vocabulary units. This is newly calculated whenever the decoder generates sub-vocabulary tokens composed of band words.

According to the neural network auto-translation model based on the encoder-decoder mechanism of the present invention, a conditional variable reflecting a correlation between input vocabularies (input tokens) constituting an input sentence input to the encoder by the decoder along with the output of the encoder By receiving the, the decoder can improve the translation quality by correctly predicting the band vocabulary (band token) for each input vocabulary constituting the input statement based on the condition variable received along with the output of the encoder. have.

When this effect is extensively interpreted, the present invention effectively reduces the number of parameters of the model and uses a generally robust sub-sord unit dictionary for unregistered words in a neural network automatic translation system using the attention model. In this case, in generating a band vocabulary, the existing attention concentration score does not indicate the entire vocabulary to which the corresponding sub-vocabulary token belongs, and is an invention for effectively improving a problem in which meaning is not properly transmitted. There is an advantage that it can be easily added to a neural network automatic translation system implemented in the conventional art.

Hereinafter, the attention mechanism applied to the encoder-decoder based neural network automatic translation model of the present invention will be described in detail.

Attention Mechanism

FIG. 1 is a block diagram of an encoder-decoder-based neural network automatic translation model to which the attention mechanism applied to the present invention is applied.

Referring to FIG. 1. The encoder-decoder-based neural network automatic translation model to which the attention mechanism applied to the present invention is applied basically includes a preprocessor 101, an encoder 102, and a decoder 201, and implements an attention mechanism. An attention layer block 105 including an attention concentration score (a _t (s)) calculation unit 103 and a weighted average value (C _t ) calculation unit 104 is further included.

The pre-processing unit 101 performs pre-processing that separates the input statement 10 into input tokens 20 in sub-word units, and input tokens 20 in sub-word units. Is input to the encoder 102.

As an input statement 10, for example, "This is called a thread, and has a lighter structure than a process." For example, the input statement 10 is preprocessed, and "This * is a thread *thread." It is divided into something called *, *, *, *, * lighter, *lighter than structure *, **. Here,'*' means that it continues without the words and spaces immediately preceding it.

This pre-processed input statement is an input token 20 by a recurrent neural network (RNN) using a Long Short-Term Memory (LSTM) used to construct the encoder 102. The iterative vector operation is performed by accumulating the calculated value for each vocabulary token included in the next vocabulary token.

Each layer constituting the encoder 102 is composed of at least one, N number of hidden layers, and the highest hidden layer state value h _'s is the current hidden state of the decoder 201 Along with the value (h _t ), it is passed to the attention layer (105), and the attention layer (105) is a weighted average value indicating the element of the token most needed for the current hidden state value (h _t ) of the decoder 201 ( Calculate C _t ).

The calculation process of the attention-level hierarchical block 105 is "global attention concentration" among the attention models proposed by Luong'15 ("Effective Approaches to Attention-based Neural Machine Translation", in Proc. of EMNLP'15, arXiv:1508.04025). Model" (global attention model).

The processing performed in the attention layer block 105 will be described in detail as follows.

Assuming that the hidden state value of the highest hidden layer for each token of the encoder 102 is h _'s , and the current hidden state value of the decoder 201 is h _t , the attention concentration score in the attention concentration layer block 105 (a _t (s)) Attention concentration score a _t (s) calculated by the calculation unit 103 is calculated by Equation 1 below.

는 아래의 수학식 2로 계산된다.In Equation 1,

Is calculated by Equation 2 below.

에서, 위 첨자 T는 현재 스텝(디코더에서 생성한 토큰의 개수만큼 증가)의 번호이고,

는 디코더의 현재 은닉 상태 값에 따른 디코더의 상태를 의미한다. 가장 첫 번째 스텝은

이고, 두 번째 스텝은

이다. 즉,

은 매 대역 T번째 토큰을 생성할 때 마다 그 시점에서의 디코더의 상태를 의미한다.

는 학습 파라미터로서, 신경망 학습 과정에서 최적의 결과를 나오도록 학습하는 과정에서 바뀌는 가중치 벡터이다. 즉, 이 가중치 벡터는 올바른 주의 집중 벡터의 위치를 가리키는 방향으로 최적화된 학습 파라미터이다.Where, above

In, superscript T is the number of the current step (increased by the number of tokens generated by the decoder),

Denotes the state of the decoder according to the current state value of the decoder. The first step

And the second step

to be. In other words,

Denotes the state of the decoder at that time each time a T-th token is generated for each band.

As a learning parameter, is a weight vector that is changed in the process of learning to produce an optimal result in the neural network learning process. That is, this weight vector is a learning parameter optimized in the direction indicating the position of the correct attention vector.

)과 함께 벡터 곱 연산(104)을 통해 각 입력 토큰(20)의 가중치 평균(weighted average) 값(C_t)으로 계산되어, 디코더(201)에서 생성하는 대역 토큰(202,

)을 생성 및 예측하기 위한 조건 변수로 활용된다.The finally generated attention score (a _t (s)) is the highest hidden state value for each token of the encoder 102 (

) Along with the vector multiplication operation 104 to calculate the weighted average value (C _t ) of each input token 20, the band token 202 generated by the decoder 201,

) As a condition variable for generating and predicting.

As described above, when the input token 2- is finely divided into sub-vocabulary units, since these weighted average values C _t are calculated based on the sub-vocabulary unit tokens, they contain all the meanings of expressing the vocabulary. It becomes impossible. In other words, the weighted average value C _t does not reflect the correlation between the input vocabulary (input tokens), such as the order of listing, so the entire meaning of expressing the vocabulary is not transmitted to the decoder.

Accordingly, in the present invention, a new neural network automatic translation model capable of transmitting a condition variable (weighted average value C _t ) reflecting a correlation such as the order of listing of input vocabularies (input tokens) to a decoder is provided.

Hereinafter, with reference to FIG. 2, the encoder-decoder based neural network automatic translation model of the present invention to which the new attention mechanism is applied will be described in detail.

2 is a block diagram of an encoder-decoder based neural network automatic translation model to which a new attention-attention mechanism is applied according to an embodiment of the present invention.

2, an encoder-decoder-based neural network automatic translation model to which a new attention-focusing mechanism according to an embodiment of the present invention is applied (hereinafter, referred to as'automatic translation model of a neural network according to an embodiment of the present invention') It includes a pre-processing unit 301, an encoder 303, a positional embedding (Positional Embedding) vector (E _p ) generating unit 305, the attention layer layer block 307 and the decoder 401.

The encoder 303 and the decoder 401 perform substantially the same functions as those of the encoder 102 and decoder 201 of the neural network automatic translation model of FIG. 1. Therefore, the description of the encoder 303 and the decoder 401 replaces the description of the encoder 102 and the decoder 201 of the neural network automatic translation model of FIG. 1.

The pre-processing unit 301 has a function similar to that of the pre-processing unit 101 of FIG. 1 in that it performs a pre-processing process of cutting (separating) the input sentence 10 into sub-vocabulary input tokens 20. This is different from the pre-processing unit 101 of FIG. 1 in that the process of generating correlation information indicating correlation between vocabularies in the token 20 is additionally performed. The generated correlation information is input to the position embedding vector (E _p ) calculator 305.

The correlation information is the vocabulary order (①→②→③→) of input vocabularies (“this”, “*silver”, “write”, “*red” and “*go”) included in the input token 20. ④→⑤) is lexical order column information 30.

In order to generate such lexical sequence column information, the pre-processing unit 301 performs a process of separating the input sentence 10 into morpheme units using an analysis technique such as morpheme analysis.

By the separation process of morpheme units, the “* silver” and “*”, which are included in the input token 20, are treated as separate input tokens. That is, even in the case of vocabulary strings connected without spaces in expression, they are separated into input tokens.

The pre-processing unit 301 sequentially assigns numbers to input tokens separated by morpheme units. The numbering process performed by the pre-processing unit 301 is as follows.

Original text
"This is called a thread and has a lighter structure than a process."
Vocabulary separation result by morpheme analysis (each vocabulary is separated by'/' separator)
"This / silver / thread / said / called / is / thing / into / process / to / in comparison / light / light / structure / into / do / uh /"
Sub-vocabulary token separation result
"This * is * that is called *thread * *, * is * lighter than the * process * and * has a light structure * *
Vocabulary sequence information
"This(1) *(2) write(3) *red(3) *(4) disadvantage(5) *(6) thing(7) *u(8) *to(8), pro( 9) * Seth (9) * At (10) Rain (11) * Year (11) Light (12) * Luck (13) Structure (14) * (15) Back (16) * Uh (17) 18) *c(18)"

The position embedding vector (E _p ) generator 305 converts the position of each vocabulary in the input token 20 into a vector format based on the lexical sequence information 30 generated by the preprocessor 301. Create a vector (E _p ).

Since the location embedding vector E _p is generated from the vocabulary sequence sequence information 30 reflecting the vocabulary order of the vocabularies in the input token 20, information reflecting the correlation between input vocabularies included in the input statement 10 is reflected. to be.

The position embedding vector E _p uses, for example, a lexical embedding, that is, a two-dimensional real valued vector that depends on the order value of the token before inputting the input token 20 into the encoder 303. It can be calculated as a substitution, or can be calculated according to a sinusoidal (sine wave) position embedding technique (sinusodial positional embedding, Vaswani et al., “Attention is all you need”, 2017).

)의 요소를 가리키는 조건 변수로서 디코더(401)에게 전달할 수 있게 된다.The position embedding vector E _p reflecting the correlation between the input vocabularies is input to the attention-focused hierarchical block 307, and the attention-focused hierarchical block 307 is the weighted average value C reflecting the correlation between the input vocabularies. _t ) is the most necessary band token for the current concealed state value h _t of the decoder 401 (

) As a condition variable indicating an element of ).

To this end, the attention concentration layer block 307 includes a combination unit 307A, an attention concentration score (a _t (S)) calculation unit 307B, and a weighted average value (C _t ) calculation unit.

The joining unit 307A focuses the position embedding vector E _p input from the position embedding vector (E _p ) generator 305 by the attention concentration score (a _t (S)) calculation unit 307B. score (a _t (S)) to bind to, a configuration for generating a new attention score (a _t (S)) the lexical order of the input words is reflected, for engagement, for example, a position embedded vector (E _p ) And attention score (a _t (S)). Here, the attention score (a _t (S)) can be calculated by the above equations (1) and (2).

)과 함께 벡터 곱 연산을 통해 입력 토큰(20) 내의 전체 어휘들에 대한 가중치 평균 값(C_t)으로 계산되어, 디코더(201)에서 생성하는 대역 토큰(202,

)을 생성 및 예측하기 위해 조건 변수로 활용된다.The result combined by the combining unit (307A) has a weight average value (C _t) is input to the calculation unit (307C), a weight average value (C _t) calculation unit (307C) is a top-level concealment status of each token in the encoder 102 value(

Band token 202 generated by the decoder 201 by calculating the weighted average value C _t for all vocabularies in the input token 20 through a vector multiplication operation with ).

) Is used as a condition variable to generate and predict.

)을 생성 및 예측함으로써, 입력문을 구성하는 각 입력 어휘에 대한 대역 어휘(대역 토큰)을 올바르게 예측함으로써, 번역품질을 개선할 수 있게 된다.Since the condition variable, that is, the weighted average value (C _t ) is calculated from the new attention concentration score (a _t (S)) in which the lexical order of the input vocabulary is reflected, as described above, the decoder 201 inputs the vocabulary. Band token 202, using a condition variable reflecting the correlation between (input tokens)

By generating and predicting ), it is possible to improve the translation quality by correctly predicting the band vocabulary (band token) for each input vocabulary constituting the input text.

3 is a block diagram of a device equipped with a neural network automatic translation model according to an embodiment of the present invention.

Referring to FIG. 3, a device 500 with a neural network automatic translation model according to an embodiment of the present invention may be an electronic device. The electronic device includes, for example, a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop personal computer (PC), Laptop personal computers (PCs), netbook computers, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, mobile medical devices, cameras, or wearable devices (eg : Includes at least one of head-mounted-device (HMD) such as electronic glasses, electronic clothing, electronic bracelets, electronic necklaces, electronic accessories, or smart watches, servers, gateways, and routers Can be.

The neural network automatic translation model generating apparatus 500 that can be implemented as such an electronic device includes a processor 510, a memory 520, a storage medium 530, a user interface 540, an output unit 550, and a communication unit 560. It includes.

The processor 510 may be configured for one or more general purpose microprocessors, digital signal processors (DSPs), hardware cores, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any combination thereof. As implemented by the user interface 540, the neural network automatic translation model shown in FIG. 2 may be processed based on the user input input in the form of a program or software. Here, the user input may be a program code, a command, etc. written by the user to generate the neural network automatic translation model shown in FIG. 2.

That is, the processor 510, according to a user input, an executable program or software type pre-processing unit 301, an executable program or software type encoder 303, an executable program or software type position embedding vector E _p ) Generates the generating unit 305, the attention-focused hierarchical block 307 in the form of an executable program or software, and the decoder 401 in the form of an executable program or software, to build a neural network automatic translation model and store it in the storage medium 530 ). Here, the storage medium 530 may be a non-volatile memory.

The processor 510 may be configured for one or more general purpose microprocessors, digital signal processors (DSPs), hardware cores, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any combination thereof. Can be implemented by

The neural network automatic translation model stored in the storage medium 530 may be executed in the execution space provided by the memory 520 by the processor 510. Here, the memory 520 may be a volatile memory.

The user interface 540 may be a key input device that generates user input written by a user. The key input device includes a keyboard, a touch screen, and the like.

The output unit 550 is a display device configured to provide a user with translation results of an automatic translation model of a neural network executed by the processor 510 or intermediate data generated during execution.

The communication unit 560 transmits a translation result generated by the neural network automatic translation model to an external device through a wired or wireless communication method, and includes appropriate modems, amplifiers, filters, and frequency conversion components to support modulation and wireless transmission. Can be implemented.

4 is a flowchart illustrating a translation method of a neural network automatic translation model according to an embodiment of the present invention.

Referring to FIG. 4, in step S410, by the pre-processing unit, the input text is divided into input tokens in sub-word units, and each input token according to the lexical order of the words included in the input text A process of generating vocabulary sequence column information including number information assigned to is performed.

In step S420, a process of generating a location embedding vector in a vector format representing location information of each input token is performed by the location embedding vector generator, using the lexical sequence information.

In step S430, the location embedding vector is calculated by the combiner of the attention-focused layer block based on the hidden state value of the highest hidden layer for each token input from the encoder and the current hidden state value input from the decoder. The process of combining is performed.

In step S440, by the weighted average value calculating unit of the attention layer, the decoder uses the attention score combined with the position embedding vector and the hidden state value of the highest hidden layer for each token of the encoder to each input token in the decoder. A process of generating a weighted average value that is used to predict the band tokens is performed.

In step S450, an exaggeration of predicting the band token is performed by the decoder using the weighted average value input from the weighted average value calculator.

)을 예측하게 된다. As described above, in the automatic translation model of the encoder-decoder based neural network of the present invention, in the operation process of the attention-focused hierarchical blocks 307: 307A, 307B, and 307C, it is calculated using the lexical order sequence information 30. After generating the position embedding vector (E _p ), and performing this multiplication with the attention concentration score (a _t (S)), the weighted average value (C _t ) of all vocabulary connected by sub-vocabulary through this value is the decoder ( 401), and the decoder utilizes the new weighted average value C _t calculated in the attention-attention layer blocks 307: 307A, 307B, and 307C, and the band token 202 for each token in the input token 20,

).

As described above, for a neural network automatic translation model using tokens separated by sub-vocabulary units, in the process of generating the band token, instead of the sub-vocabulary token of the original vocabulary, the entire vocabulary to which the sub-vocabulary belongs is more correct You can generate a band token. This can improve the disadvantage of not correctly converting the entire meaning of the vocabulary in the process of generating sub-vocabulary tokens constituting the band sentence in the prior art.

In the above, the present invention has been mainly described with reference to examples, but this is merely an example, and is not intended to limit the present invention. It will be appreciated that various modifications and applications not illustrated in the examples are possible. For example, each component specifically shown in the embodiments of the present invention can be implemented by modification. And differences related to these modifications and applications should be construed as being included in the scope of the invention defined in the appended claims.

Claims (1)

In a translation method based on the neural network automatic translation model, which includes an encoder and a decoder.
In the pre-processing unit, the input text is divided into sub-word unit input tokens, and a lexical order column including number information assigned to each input token according to the lexical order of the words included in the input text Generating information;
Generating, by the location embedding vector generation unit, a location embedding vector in vector format representing location information of each input token using the lexical sequence information;
In the attention layer block, combining the position embedding vector with the attention score calculated based on the hidden state value of the highest hidden layer for each token input from the encoder and the current hidden state value input from the decoder; And
In the weighted average value calculation unit, the weighted average used to predict the band token for each input token in the decoder by using the attention score score combined with the location embedding vector and the hidden state value of the highest hidden layer for each token of the encoder Generating a value; And
In the decoder, predicting the band token using the weighted average value input from the weighted average value calculator;
Neural network automatic translation model based translation method comprising an encoder and a decoder comprising a.

Images