CN116150334A - Chinese Empathy Sentence Training Method and System Based on UniLM Model and Copy Mechanism - Google Patents

Abstract

The invention belongs to the technical field of Chinese-oriented natural language generation, and provides a Chinese cosolvents sentence training method and system based on a UniLM model and a Copy mechanism. Meanwhile, aiming at the lack of sufficient and diversified training corpuses, the generated co-emotion replies are comprehensively evaluated, and the high-quality co-emotion replies meeting the expected standard and user input are put into the original training corpuses to carry out compound automatic iterative training, so that training data are enhanced. The invention aims to solve the problem that a Copy mechanism is fused in a decoder and emotion keywords and complex event details are copied into output. Aiming at the problem of corpus shortage of Chinese mind dialogue with co-emotion capability, the invention adopts compound automatic iterative training to enhance training data.

Description

Chinese Empathy Sentence Training Method and System Based on UniLM Model and Copy Mechanism

technical field

The invention belongs to the technical field of Chinese-oriented natural language generation, and in particular relates to a Chinese empathy reply generation method based on UniLM and Copy mechanism.

Background technique

With the application of deep learning in various fields, intelligent conversational systems have also developed rapidly. Users hope to communicate emotionally with intelligent conversational systems, and empathy can achieve this goal. Therefore, empathy recovery generation came into being. Empathy is defined by Carl Ransom Rogers as: in the process of interpersonal communication, imagining the experience and logic of others from the standpoint of others, understanding the thoughts and feelings of others, looking at and solving problems from the perspective of others. Empathy reply generation refers to the intelligent conversation system judging the user's emotional state through historical conversations, so as to generate an emotional reply that understands the user's feelings. Existing studies have shown that an intelligent conversational system with empathy can not only improve user satisfaction, but also obtain more positive feedback from users.

In the mental health counseling session, the intelligent conversation system, as an auxiliary tool, can help the counselor solve some tasks, and is considered to be the key to the application of mental health intervention, auxiliary counseling and diagnosis and other service applications. Therefore, intelligent conversational systems endowed with empathy have gradually become a research hotspot. A good conversation model must have a strong contextual correlation between its input and output. Contextual relevance refers to the interrelationship between the user's input and the model's output. Currently, the mainstream reply generation methods are sequence-to-sequence methods based on deep learning, or based on pre-trained models.

The traditional sequence-to-sequence encoder end is mainly RNN, LSTM, etc. Compared with Transformer, RNN, LSTM, etc. are not capable of semantically extracting features, and are lacking in long-distance dependencies. Although the readability of replies generated by various Transformer-based language models is higher than that of RNN, LSTM, etc., there is still the problem of inaccurate generation details resulting in irrelevant context.

Contents of the invention

Aiming at the problems existing in the prior art, the present invention proposes a Chinese empathy sentence training method based on the UniLM model and the Copy mechanism.

The present invention is achieved in this way, a Chinese empathy reply generation method based on the UniLM model and the Copy mechanism, the purpose of integrating the Copy mechanism is to copy the emotional keywords and complex event details in the source sequence to the output; and then use the perplexity Evaluate the output empathy responses according to the evaluation criteria, put the expected replies and user statements into the original training corpus for compound automatic iterative training, and obtain a further updated and optimized empathy response generation model.

The technical solution adopted by the present invention is a Chinese empathy reply generation method based on the UniLM model and the Copy mechanism, which specifically includes the following steps:

Step 1, use crawler technology to crawl the corpus with empathy ability in the field of psychological dialogue, and perform preprocessing to obtain input representation;

Step 2, pre-training based on the UniLM model, using three types of language models at the same time, each language model uses a different self-attention mask mechanism;

Step 3, use the cross-entropy loss function to calculate the loss, complete the pre-training based on the UniLM model, and obtain the empathy reply generation model;

Step 4, based on the UniLM model, the empathy reply generation task is performed, and the vocabulary probability distribution is obtained through the self-attention mechanism decoding of the sequence-to-sequence language model;

Step 5, build a decoder including Copy mechanism on the basis of step 4, introduce generation probability and copy probability, and optimize the vocabulary probability distribution in step 4;

Step 6, use the cross-entropy loss function as the loss function of the model, and use the BeamSearch algorithm to obtain the generated empathy reply;

Step 7. Put the generated high-quality empathy replies and user statements into the corpus in step 1, and further perform compound automatic iterative training based on the UniLM model to obtain an updated and optimized empathy reply generation model.

Further, each time you input two text sequences Segment1, denoted as s1 and Segment2, denoted as s2, for example: "[CLS] always thinks of some people or things that I hate very much in my mind [SEP] understands that you are because of entanglement Confused and puzzled by negative events in life, forgetting positive events [SEP]". [CLS] marks the beginning of the sequence, [SEP] marks the end of the sequence, and the text sequence pair is represented by three types of Embedding.

Further, the UniLM model is stacked by a 12-layer Transformer structure, and the hidden layer of each Transformer has 768 hidden nodes and 12 heads. The structure is the same as that of BERT-BASE, so parameters can be initialized by the trained BERT-BASE model. The UniLM model can complete three pre-training objectives at the same time, and can complete the prediction tasks of one-way training language model, two-way training language model, and sequence-to-sequence language model, enabling the model to apply natural language generation tasks. For different language models, different MASK mechanisms are adopted, and the MASKING method: the overall ratio is 15%, of which 80% are directly replaced by [MASK], and in 10% cases, a word in the dictionary is randomly selected to replace, and the last 10% The case uses the true value and does not do any processing. In addition, in 80% of cases, only one word is masked at a time, and in the other 20% of cases, two words bigram or three words trigram are masked. For the MASK to be predicted, the one-way language model uses the context of one side, such as the mask in the prediction sequence "X1X2[MASK]X4", only X1, X2 and its own information are available, and the information of X4 is not available . The bidirectional language model encodes context information from two directions, taking "X1X2[MASK]X4" as an example, where X1, X2, X4 and its own information are all available. In the sequence-to-sequence language model, if the MASK is in S1, it can only encode the context information of S1; if the MASK is in S2, it can obtain the left side of the MASK, including the context information of S1.

Further, the text representation output by the Transformer network is input into the Softmax classifier to predict the masked words, use the cross-entropy loss function for the predicted word segmentation and the original word segmentation, optimize the model parameters, and complete the pre-training.

Further, by randomly covering up a certain proportion of the word segmentation in the target sequence, the sequence-to-sequence language model is used to learn to restore the covered word, and the training goal is to maximize the probability of the masked word segmentation based on context information. [SEP] at the end of the target sequence can also be masked, allowing the model to learn when to stop generating the target sequence. The model uses the MASK mechanism, combined with the attention mechanism to obtain the text feature vector, which is input to the fully connected layer to obtain the probability distribution of the vocabulary.

Further, the vocabulary probability distribution is input into the fully connected layer and the Sigmoid layer to obtain the generation probability. Then the copy probability is introduced, combined with the generation probability and the copy probability, an updated and improved vocabulary probability distribution is obtained.

Further, the cross-entropy loss function is used to complete the fine-tuning task of the model, and the Beam Search algorithm is used to generate an empathy reply.

Further, the four evaluation indicators such as perplexity, BLEU-4, F1 and expert evaluation are used to make a comprehensive evaluation of the empathy reply generated in step 6, and the empathy reply that meets the expected standard and user input are automatically put into step 1 The compound automatic iterative training is carried out in the original corpus, the training data is enhanced, and the updated and optimized Chinese empathy reply generation model is obtained.

The purpose of the present invention is to solve the problem that the empathy reply generated based on Transformer network cannot generate emotional keywords and complex event details, and proposes to integrate the Copy mechanism in the decoder to copy the emotional keywords and complex event details to the output.

Another purpose of the present invention is to solve the problem of lack of corpus with empathy ability in Chinese mental dialogue, and the present invention adopts compound automatic iterative training to enhance the training data.

Combining the above-mentioned technical solutions and technical problems to be solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving the problems, closely combine the technical solution to be protected in the present invention and the results and data in the research and development process, etc., to analyze in detail how the technical solution of the present invention solves it Technical problems, some creative technical effects brought about after solving the problems. The specific description is as follows:

In the process of interpersonal communication, people hope to imagine other people's experience and logic from the standpoint of others, understand other people's thoughts and feelings, and look at and solve problems from the perspective of others. Among them, the intelligent conversational system endowed with empathy has gradually become a research hotspot. The present invention solves the problem of empathy reply generation, which means that the intelligent conversation system judges the user's emotional state through historical conversations, thereby generating an emotional reply that understands the user's feelings. An intelligent conversational system with empathy can not only improve user satisfaction, but also get more positive feedback from users.

Second, regarding the technical solution as a whole or from the perspective of a product, the technical effects and advantages of the technical solution to be protected by the present invention are specifically described as follows:

The invention proposes a Chinese empathy reply generation method based on the UniLM model and the Copy mechanism. The invention uses the UniLM model as the basic framework, and aims at the problem that the empathy reply generated based on the Transformer network cannot generate emotional keywords and complex event details, and proposes to integrate the Copy mechanism in the decoder to copy the emotional keywords and complex event details to the output to solve. Aiming at the problem of lack of corpus with empathy ability in Chinese mental dialogue, the present invention adopts compound automatic iterative training to enhance training data.

The present invention copies the emotional keywords and complex event details in the source sequence to the output; then uses evaluation criteria such as perplexity to evaluate the output empathy replies, and puts the replies and user statements that meet expectations into the original training corpus. Compound automatic iterative training to obtain a further updated and optimized empathy response generation model.

Description of drawings

Fig. 1 is a framework diagram of the Chinese empathy reply generation model based on the UniLM model and the Copy mechanism provided by the embodiment of the present invention;

Fig. 2 is a schematic diagram of the UniLM model architecture provided by the embodiment of the present invention;

Fig. 3 is a specific flow chart of the Chinese empathy reply generation method based on the UniLM model and the Copy mechanism provided by the embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with the examples. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

In order to make those skilled in the art fully understand how to implement the present invention, this part is an explanatory embodiment for explaining the technical solution of the claims.

In the present invention, the method for generating Chinese empathy responses based on the UniLM model and the Copy mechanism will be further described in detail in conjunction with the accompanying drawings and specific implementation methods.

As shown in Figure 1, the present invention is mainly based on the UniLM model, and integrates the Copy mechanism at the decoding end to realize the purpose of conversational empathy to make full use of the context correlation of complex event details. It mainly includes four stages: input processing, pre-training, empathy response generation, and compound training. The specific implementation is as follows:

The pre-trained corpus includes psychological counseling visitors' statements about psychological problems and counselors' empathic responses. Segment1 of the visitor’s statement, denoted as S1, Segment2 of the consultant’s reply, denoted as S2, with special marks [CLS] and [SEP] added, in the form of “[CLS]S1[SEP]S2[SEP]”. As shown in Figure 2, the input representation of the model consists of the sum of SegmentEmbedding, Position Embedding, and Token Embedding.

Model pre-training, input Embedding vector, each layer of Transformer encodes the input vector, uses multi-head attention mechanism to aggregate the upper layer input, controls the range of attention of each word or position through the mask matrix, and obtains the attention distribution of the current position to other positions, Calculate the eigenvector for the current position of the decoder.

The attention distribution At of the generated word vector to the text feature vector XInput at time t is as follows:

The feature vector XOutput output by the decoder at time t is as follows:

X _Output = A _t *W _v *X _Input

Among them, Xt is the target vector at time t; XInput is the text feature vector at time t; M is the mask matrix, which is used to control the word attention range; dk is the dimension of the word vector; Wq, Wk, and Wv are learning parameters.

The Softmax function maps a vector of scores s to a probability distribution, which is defined as follows:

Among them, i represents the number of the output node; si is the output value of the i-th node; n is the number of output nodes, that is, the number of classification categories.

Further, calculate the cross-entropy loss for the model prediction result XOutput, denoted as s, and the masked original word st to optimize the parameters of the model. The cross entropy function is defined as follows:

Preprocessing process: Input the preprocessed data into the model for training. A total of 20 Epochs are trained, the Dropout is 0.1, the hidden vector dimension is 768, the learning rate Learning_rate is 2e-5, the Epochs is 20, and the batch size Batch_size is 32. The number of attention heads is 12, the number of hidden layers is 12, the number of embedding layers is 12, the number of hidden layer units is 768, and the vocabulary size is 21128. The maximum input length is set to 512, the maximum generated empathy reply length is set to 40, and the loss is calculated using the cross-entropy function.

After completing the pre-training, use UniLM's sequence-to-sequence language model for fine-tuning to perform empathy response generation tasks. When decoding, for example: the user enters a statement "X1" of inner psychological problems, and when t=1, enters the sequence "[CLS]X1[SEP]Y1[MASK]", and adds "[MASK]" at the end of the sequence, Its corresponding feature represents predicting the next word. "[CLS]X1[SEP]" is a known source sequence, and the context information in the sentence can be seen from each other in the encoding stage. "Y1[MASK]" is the predicted target sequence. In the decoding stage, the information of the source sequence and the information of the left part of the target sequence can be seen. The model fuses the encoder and decoder together via a mask matrix.

After the corpus sample is encoded by the UniLM model, a sequence length X hidden size matrix is obtained. The first row is the feature representation of [CLS], the second row is the feature representation of X1, and so on. In the decoding stage, use the [MASK] feature to represent the linear layer, use the Softmax function to obtain the probability distribution of words in the vocabulary, and select the word with the highest probability as the word obtained by decoding, repeat the above steps, and stop when [SEP] is generated , to get the feature vector XOutput output by the decoder at time t. The specific calculation is as follows:

XOutput obtains the vocabulary probability distribution Pv after two linear transformations and the Softmax function.

P _v ＝Softmax(W ^′ (W*X _Output +b)+b ^′ )

Among them, W ^′ , W, b, b ^′ are learnable parameters.

The generation probability Pg is introduced to represent the probability of generating words from the vocabulary; the copy probability Pc is introduced to represent the probability of copying words from the source text, where Pg+Pc=1. Calculate Pg by calculating XOutput, At, and Xt through the fully connected layer and the Sigmoid function.

P _g ＝Sigmoid(W[X _t , X _Output , A _t ]+b)

Among them, W and b are learnable parameters.

Further calculate the updated and improved vocabulary probability distribution:

P(w)＝P _g *P _v (w)+P _c *A _t

Among them, when w is not a word in the vocabulary, P _v (w)=0, the predicted word is generated from the source sequence; when w is not a word in the source sequence, A _t =0, the predicted word is generated from the vocabulary generated in. The copy mechanism copies emotional keywords and complex event details (words with high probability) from the source sequence as part of the generated empathy reply, which can control the accuracy of empathy reply generation to a certain extent. The copy mechanism also plays a role in dynamically expanding the vocabulary to a certain extent to reduce the probability of generating unregistered words.

Set the Beam size to 1, use the Beam Search algorithm to search for a target sequence that is close to the optimal, and generate an empathetic reply. Evaluate the generated empathy replies, put the standard empathy replies and user statements into the original corpus for compound automatic iterative training, enhance the training data, and obtain an updated and optimized Chinese empathy reply generation model.

The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Anyone familiar with the technical field within the technical scope disclosed in the present invention, whoever is within the spirit and principles of the present invention Any modifications, equivalent replacements and improvements made within shall fall within the protection scope of the present invention.

Claims (10)

1. A Chinese co-emotion reply generation method based on a UniLM model and a Copy mechanism is characterized in that emotion keywords and complex event details in a source sequence are copied into output; and evaluating the output co-emotion replies by using evaluation standards such as confusion, putting the replies meeting expectations and user statements into an original training corpus to perform compound automatic iterative training, and obtaining a further updated and optimized co-emotion reply generation model.

2. The method for generating Chinese co-emotion reply based on UniLM model and Copy mechanism as claimed in claim 1, which is characterized by comprising the following steps:

step 1, crawling corpus with co-emotion capacity in the field of the psychology dialogue by using a crawler technology, and preprocessing to obtain input representation;

step 2, pre-training is carried out based on a UniLM model, and simultaneously three types of language models are used, wherein each language model uses a different self-attention mask mechanism;

step 3, calculating loss by using a cross entropy loss function, and completing pre-training based on a UniLM model to obtain a co-condition reply generation model;

step 4, carrying out a co-emotion reply generation task based on the UniLM model, and decoding through a self-attention mechanism from a sequence to a sequence language model to obtain vocabulary probability distribution;

step 5, constructing a decoder containing a Copy mechanism on the basis of the step 4, introducing generation probability and Copy probability, and optimizing vocabulary probability distribution in the step 4;

step 6, using the cross entropy loss function as a loss function of the model, and obtaining a generated co-situation reply by using a beam search algorithm;

and 7, putting the generated high-quality co-emotion replies and the statements of the users into the corpus in the step 1, and further carrying out compound automatic iterative training based on the UniLM model to obtain an updated and optimized co-emotion reply generation model.

3. The method for generating Chinese co-emotion replies based on UniLM model and Copy mechanism as claimed in claim 2, wherein step 2 specifically comprises: initializing parameters by using a BERT-BASE pre-training model; based on the same transducer network structure, different MASK is predicted as a pre-training target, the prediction tasks of unidirectional, bidirectional and sequence-to-sequence language models are completed, and different language models are uniformly distributed and used.

4. The method for generating Chinese co-emotion replies based on UniLM model and Copy mechanism as claimed in claim 2, wherein step 4 specifically comprises: using a self-attention MASK mechanism of a sequence-to-sequence language model to randomly segment words in a MASK target sequence and MASK the end of the sequence to learn when to stop generating a co-emotion reply; taking the maximum word segmentation probability under the condition of given context information as a training target, fusing encoding and decoding by using a MASK mechanism, and obtaining text feature vectors by combining an attention mechanism; and inputting the feature vectors obtained by decoding into a full-connection layer, and obtaining vocabulary probability distribution by using a Softmax function.

5. The method for generating Chinese co-emotion reply based on UniLM model and Copy mechanism as claimed in claim 2, wherein step 5 specifically comprises: inputting the vocabulary probability obtained in the last step into a full-connection layer and a Sigmoid layer to obtain a generation probability, introducing a replication probability, and fusing the generation probability and the replication probability to obtain updated and improved vocabulary probability distribution; the Copy mechanism effectively copies emotion keywords and complex event details input by a user into output, accuracy of details in generated co-emotion replies is improved, and meanwhile probability of generating unregistered words can be effectively reduced.

6. The method for generating Chinese co-emotion replies based on UniLM model and Copy mechanism as claimed in claim 2, wherein step 7 specifically comprises: and (3) evaluating the co-emotion replies generated in the step (6) through evaluation standards such as confusion, automatically putting the expected co-emotion replies and user input into the corpus in the step (1) for iterative training, enhancing training data, and obtaining an updated and optimized co-emotion reply generation model.

7. A chinese co-emotion reply generation system based on the generation method of any one of claims 1 to 6, comprising:

the detail copying module is used for copying emotion keywords and complex event details in the source sequence into output;

and the co-emotion reply generation model module is used for evaluating the output co-emotion replies by using evaluation standards such as confusion, and putting expected replies and user statements into the original training corpus to perform compound automatic iterative training so as to obtain a further updated and optimized co-emotion reply generation model.

8. A computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of the method of improving pedestrian detection of a YOLOv4 network according to any one of claims 1-5.

9. A computer readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of the improved method of pedestrian detection in a YOLOv4 network according to any one of claims 1-5.

10. An information data processing terminal for realizing the pedestrian detection system of the improved YOLOv4 network according to claim 1.

Images