CN114781358A - Text error correction method, device, device and storage medium based on reinforcement learning - Google Patents

Abstract

The present application discloses a text error correction method, device, device and storage medium based on reinforcement learning, belonging to the technical field of natural language processing. This application uses the masked training corpus to construct a first training sample by masking the similarities of pronunciation and glyphs of the text in the training corpus, importing the first training sample into a pre-training language model, and outputting the error correction result of the first pre-training text , adjust the first training sample to generate a second training sample, use the second training sample to iteratively train the pre-trained language model to obtain a text error correction model, and finally import the text to be corrected into the text error correction model, and output the text correction model. wrong result. This application introduces pronunciation information and glyph information when training the error correction model, constructs training samples with rich noise through similar pronunciation masking and similar glyph masking, and further trains the error correction model through reinforcement learning technology, so that the model can better recognize spelling Error, generalization performance is stronger.

Description

Text error correction method, device, device and storage medium based on reinforcement learning

technical field

The present application belongs to the technical field of natural language processing, and specifically relates to a text error correction method, apparatus, device and storage medium based on reinforcement learning.

Background technique

The purpose of text error correction is to detect and correct spelling errors in text. It is an important task in natural language processing and has applications in many fields such as information retrieval, intelligent writing, and intelligent customer service. Most of the methods used in traditional text error correction are as follows: set up a confused text set containing a large number of Chinese characters that are easily misspelled, when the text in the document to be corrected matches the text in the obfuscated text set, replace it with the text in the obfuscated text set, but this is not the case. This mechanical matching method does not consider contextual semantics, and the effect is not ideal.

In recent years, some researchers have carried out text error correction based on the existing pre-trained language model Bert + fine-tuning method. In the implementation, the error-prone text is masked with the wrong text to construct a negative training sample. For example, in the sample "China is an ancient civilization with a history of 5,000 years", replace "history" with "Li Shi" ”, and then fine-tune the pre-trained language model to identify and correct erroneous text. However, since the pre-trained language model is not specially designed for text error correction tasks during training, the knowledge of how to correct errors is rarely learned in the fine-tuned model, and there is still much room for improvement in the practical effect.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present application is to propose a text error correction method, device, computer equipment and storage medium based on reinforcement learning, so as to solve the existing technical problem of poor text error correction effect on pre-trained language models.

In order to solve the above technical problems, the embodiment of the present application provides a text error correction method based on reinforcement learning, which adopts the following technical solutions:

A reinforcement learning-based text error correction method, including:

Collect training corpus, and mask the text in the training corpus according to the preset text masking ratio for pronunciation similarity masking and glyph similarity masking;

Use the masked training corpus to construct a first training sample;

Perform vector transformation on the first training sample to obtain the first sample embedding vector;

Import the first sample embedding vector into the pre-training language model, and output the error correction result of the first pre-training text;

Adjusting the first training sample based on the error correction result of the first pre-training text to generate a second training sample;

Use the second training sample to iteratively train the pre-trained language model to obtain a text error correction model;

Receive the text error correction instruction, obtain the text to be corrected, import the text to be corrected into the text error correction model, and output the text error correction result.

Further, collect the training corpus, and perform the steps of masking similar pronunciation and masking the text in the training corpus according to the preset text masking ratio, which specifically includes:

Collect training corpus, and divide the training corpus to obtain several training corpus segments;

Determine the target masking text in the training corpus segment according to the preset text masking ratio;

Determine the pronunciation-similar text and glyph-similar text corresponding to the target masked text in the preset text confusion set;

Pronunciation similarity masking and glyph similarity masking are performed on the target masked text based on the pronunciation similar text and the glyph similarity text.

Further, the step of constructing the first training sample using the masked training corpus specifically includes:

A first training sample is formed by combining the training corpus segments with similar masking of pronunciation and the training corpus segments with similar masking of glyphs.

Further, the step of constructing the first training sample using the masked training corpus specifically includes:

Obtain random text from a preset vocabulary, and use random text to perform random text masking on the target masked text;

A first training sample is formed by combining the training corpus segments with similar pronunciation masking, the training corpus segments with similar glyph masking, the training corpus segments with random text masking, and the training corpus segments without text masking.

Further, the step of adjusting the first training sample based on the error correction result of the first pre-training text to generate the second training sample specifically includes:

Calculate the action value score of each training corpus segment in the first training sample based on the error correction result of the first pre-training text;

Based on the action value score of each training corpus segment, the proportion of each training corpus segment in the first training sample is adjusted to obtain a second training sample.

Further, the step of calculating the action value score of each training corpus segment in the first training sample based on the error correction result of the first pre-training text specifically includes:

Determine the F1 value of the pre-trained language model based on the error correction result of the first pre-trained text;

Get the training cost of the pre-trained language model;

The action value score of each training corpus segment in the first training sample is calculated based on the F1 value of the pre-trained language model and the training cost.

Further, the steps of adjusting the proportion of each training corpus segment in the first training sample based on the action value score of each training corpus segment to obtain the second training sample specifically include:

Determine the positive or negative value of the action value score of each training corpus segment;

When the action value score is a positive number, the proportion of training corpus segments corresponding to the action value score is increased;

When the action value score is negative, the proportion of training corpus segments corresponding to the action value score is lowered.

Further, the steps of using the second training sample to iteratively train the pre-trained language model to obtain a text error correction model specifically include:

Import the second training sample into the pre-training language model to obtain the error correction result of the second pre-training text;

Calculate the action value score of each training corpus segment in the second training sample based on the error correction result of the second pre-training text;

Summing the action value scores of each training corpus segment in the second training sample to obtain the final action value score;

Based on the final action value score, the pre-trained language model is iteratively trained to obtain a text error correction model.

Further, the first sample embedding vector includes a text embedding vector, a position embedding vector, a pronunciation embedding vector and a glyph embedding vector, and the steps of performing vector transformation on the first training sample to obtain the first sample embedding vector specifically include:

Perform feature extraction on the first training sample to obtain text features, position features, pronunciation features and glyph features;

The text feature, position feature, pronunciation feature and glyph feature are vectorized respectively, and the text embedding vector, position embedding vector, pronunciation embedding vector and glyph embedding vector are obtained.

In order to solve the above technical problems, the embodiment of the present application also provides a text error correction device based on reinforcement learning, which adopts the following technical solutions:

A text error correction device based on reinforcement learning, comprising:

The text masking module is used to collect training corpus, and mask the text in the training corpus according to the preset text masking ratio for pronunciation similarity masking and glyph similarity masking;

The sample building module is used to construct the first training sample by using the masked training corpus;

a vector transformation module, which is used to perform vector transformation on the first training sample to obtain the first sample embedding vector;

The pre-training module is used to import the first sample embedding vector into the pre-training language model, and output the error correction result of the first pre-training text;

a sample adjustment module, configured to adjust the first training sample based on the error correction result of the first pre-training text to generate a second training sample;

The iterative training module is used to iteratively train the pre-trained language model by using the second training sample to obtain a text error correction model;

The text error correction module is used to receive the text error correction instruction, obtain the text to be corrected, import the text to be corrected into the text error correction model, and output the text error correction result.

In order to solve the above-mentioned technical problems, the embodiment of the present application also provides a computer device, which adopts the following technical solutions:

A computer device, comprising a memory and a processor, wherein computer-readable instructions are stored in the memory, and when the processor executes the computer-readable instructions, the reinforcement learning-based text error correction as described in any of the above is implemented steps of the method.

In order to solve the above technical problems, the embodiments of the present application also provide a computer-readable storage medium, which adopts the following technical solutions:

A computer-readable storage medium on which computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, the text correction based on reinforcement learning as described in any one of the above is realized; Steps of the wrong method.

Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects:

The present application discloses a text error correction method, device, device and storage medium based on reinforcement learning, belonging to the technical field of natural language processing. This application collects training corpus, performs similar pronunciation masking and glyph similarity masking on the text in the training corpus, uses the masked training corpus to construct a first training sample, imports the first training sample into a pre-training language model, and outputs the first pre-training language model. training text error correction results, adjusting the first training sample based on the first pre-training text error correction results, generating a second training sample, using the second training sample to iteratively train the pre-training language model, obtaining a text error correction model, and obtaining For the text to be corrected, import the text to be corrected into the text error correction model, and output the text error correction result. This application introduces pronunciation information and glyph information when training the error correction model, constructs training samples with rich noise through similar pronunciation masking and similar glyph masking, and further trains the error correction model through reinforcement learning technology, so that the model can better recognize spelling error, making the model more generalizable.

Description of drawings

In order to illustrate the solutions in the present application more clearly, the following will briefly introduce the accompanying drawings used in the description of the embodiments of the present application. For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 shows an exemplary system architecture diagram to which the present application can be applied;

FIG. 2 shows a flowchart of an embodiment of the reinforcement learning-based text error correction method according to the present application;

Fig. 3 shows a flowchart of an embodiment of step S201 in Fig. 2;

FIG. 4 shows a schematic structural diagram of an embodiment of an apparatus for text error correction based on reinforcement learning according to the present application;

FIG. 5 shows a schematic structural diagram of an embodiment of a text masking module 301 of a text error correction device based on reinforcement learning;

FIG. 6 shows a schematic structural diagram of an embodiment of a computer device according to the present application.

Detailed ways

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field of this application; the terms used herein in the specification of the application are for the purpose of describing specific embodiments only It is not intended to limit the application; the terms "comprising" and "having" and any variations thereof in the description and claims of this application and the above description of the drawings are intended to cover non-exclusive inclusion. The terms "first", "second" and the like in the description and claims of the present application or the above drawings are used to distinguish different objects, rather than to describe a specific order.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings.

As shown in FIG. 1 , the system architecture 100 may include terminal devices 101 , 102 , and 103 , a network 104 and a server 105 . The network 104 is a medium used to provide a communication link between the terminal devices 101 , 102 , 103 and the server 105 . The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages and the like. Various communication client applications may be installed on the terminal devices 101 , 102 and 103 , such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like.

The terminal devices 101, 102, and 103 may be various electronic devices that have a display screen and support web browsing, including but not limited to smart phones, tablet computers, e-book readers, and MP3 players (Moving Picture Experts Group Audio Layer III, moving picture experts). Compression Standard Audio Layer 3), MP4 (Moving PictureExperts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) Players, Laptops and Desktops, etc.

The server 105 can be a server that provides various services, such as a background server that provides support for the pages displayed on the terminal devices 101, 102, and 103. The server can be an independent server, or can provide cloud services, cloud databases, cloud computing, Cloud servers for basic cloud computing services such as cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

It should be noted that the text error correction method based on reinforcement learning provided by the embodiments of the present application is generally executed by a server, and accordingly, the text error correction device based on reinforcement learning is generally set in the server.

It should be understood that the numbers of terminal devices, networks and servers in FIG. 1 are merely illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

Continuing to refer to FIG. 2 , a flowchart of one embodiment of the reinforcement learning-based text error correction method according to the present application is shown. The embodiments of the present application may acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. .

The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning/deep learning. The text error correction method based on reinforcement learning includes the following steps:

S201 , collecting training corpus, and masking the text in the training corpus according to a preset text covering ratio for similar pronunciation and similar characters.

Specifically, news articles are collected from public Internet websites as training corpus, and each news article is cut to meet the input length of the pre-training language model. For example, Bert's input length is 510 Chinese characters, and the cut training corpus is used as pre-training Training samples for the language model.

After collecting the training corpus, the server divides the training corpus to obtain a plurality of training corpus segments, determines the text that needs to be covered in the training corpus segment according to the preset text masking ratio, that is, the target masked text, and masks the target in the training corpus segment. Pronunciation similarity masking and glyph similarity masking are performed on texts. Among them, pronunciation similarity masking means masking the training corpus with texts similar in pronunciation to the target masking text. For example, "Jingjing" and "Jinjin" belong to similar pronunciation; The training corpus is masked by texts with similar glyphs to the target masking text, for example, "Li" and "Li" belong to similar glyphs.

It should be noted that similar texts can be selected from a publicly available text confusion set, which contains two types of similar texts, ie similar texts in pronunciation and similar in glyphs.

Further, please refer to FIG. 3, step S201 collects training corpus, and performs the steps of masking similar pronunciation and masking the text in the training corpus according to a preset text masking ratio, specifically including:

S211, collect training corpus, and divide the training corpus to obtain several pieces of training corpus;

S212, determining the target masking text in the training corpus segment according to a preset text masking ratio;

S213, determining in the preset text confusion set, the text similar to the pronunciation and the similar text of the glyph corresponding to the target masked text;

S214 , performing pronunciation similarity masking and glyph similarity masking on the target masked text based on the pronunciation similar text and the glyph similarity text.

Specifically, the server collects the training corpus, divides the training corpus, obtains several training corpus segments, and then determines the target masking text in the training corpus segment according to the preset text masking ratio, and determines it in the preset text confusion set. Pronunciation similarity text and glyph similarity text corresponding to the target masking text, based on the pronunciation similarity text and the glyph similarity text, the pronunciation similarity masking and the glyph similarity masking are performed on the target mask text.

In a specific embodiment of the present application, the server divides the training corpus into two training corpus segments, selects one of the training corpus segments to complete pronunciation similarity concealment, and selects another training corpus segment to complete glyph similarity concealment.

In the above-mentioned embodiment, the present application constructs training samples with rich noise by dividing the training corpus, and masking the similarities of pronunciation and glyphs on the divided training corpus segments.

S202, using the masked training corpus to construct a first training sample.

Specifically, the server constructs the first training sample by using the masked training corpus, and constructs a training sample with rich noise by masking the training corpus with different texts, so that the trained model can better identify spelling errors.

Further, the step of constructing the first training sample using the masked training corpus specifically includes:

A first training sample is formed by combining the training corpus segments with similar masking of pronunciation and the training corpus segments with similar masking of glyphs.

Specifically, in the above-mentioned embodiment, the first training sample is formed by combining the training corpus segments for which the similar pronunciation masking is completed and the training corpus segments for which the similar glyph masking is completed.

Further, the step of constructing the first training sample using the masked training corpus specifically includes:

Obtain random text from a preset vocabulary, and use random text to perform random text masking on the target masked text;

A first training sample is formed by combining the training corpus segments with similar pronunciation masking, the training corpus segments with similar glyph masking, the training corpus segments with random text masking, and the training corpus segments without text masking.

Specifically, the server obtains random text from a preset vocabulary, and uses the random text to perform random text masking on the target masked text, and then combines the training corpus segments with similar pronunciation masking, the training corpus segments with similar glyph masking, and the random text masking. The training corpus segment and the training corpus segment without text masking form the first training sample, and the training corpus segment with more noise can be obtained by combining the above training corpus segments.

In another specific embodiment of the present application, the number of corpora used for training the pre-training language model is set to batch_size, and for the training corpus, a% of the characters in the batch_size are randomly selected for masking each time, and for the selected characters, there are 4 masking methods, the specific operations are as follows:

(1) Select a1% corpus to complete pronunciation similarity masking; (2) select a2% corpus to complete glyph similarity masking; (3) select a3% corpus without text masking; (4) select a4% The corpus performs random text masking from random text in the vocabulary. Among them, the vocabulary contains common text (such as containing Chinese characters and non-Chinese characters), and the pre-built vocabulary usually contains tens of thousands of characters.

For example, 10,000 corpora are collected for the training of the pre-trained language model. Among them, 2,500 corpora are selected for pronunciation similarity concealment, 2,500 corpora are selected for glyph similarity concealment, 2,500 corpora are selected for no processing, and 2,500 corpora are selected for text randomization. Masking, and then combining the above corpora to construct the first training sample.

S203: Perform vector transformation on the first training sample to obtain a first sample embedding vector.

Specifically, by performing vector transformation on the first training sample, the input vector of the pre-trained language model, that is, the first sample embedding vector, is obtained.

Further, the first sample embedding vector includes a text embedding vector, a position embedding vector, a pronunciation embedding vector and a glyph embedding vector, and the steps of performing vector transformation on the first training sample to obtain the first sample embedding vector specifically include:

Perform feature extraction on the first training sample to obtain text features, position features, pronunciation features and glyph features;

The text feature, position feature, pronunciation feature and glyph feature are vectorized respectively, and the text embedding vector, position embedding vector, pronunciation embedding vector and glyph embedding vector are obtained.

Specifically, the text embedding vector represents the text features of the training samples, and the text embedding vector can be obtained by searching a preset word vector embedding table, wherein the vocabulary table and the word vector embedding table have the same dimension. The position embedding vector represents the position feature of the training sample, and the position embedding vector is obtained by the sine and cosine function in the transformer model. The pronunciation embedding vector represents the pronunciation characteristics of the training samples. In the Chinese field, pronunciation (also called pinyin) is a sequence of lowercase letters, representing the pronunciation of a character. In practice, UnihanDatabase3 can be used to obtain the character-pronunciation mapping relationship, which will represent each character. Pronounced letters are sent to the LSTM network to get the pronunciation embedding vector. The glyph embedding vector represents the glyph features of the training samples, and the shape of the character is represented by the stroke order, which indicates the writing order of the strokes of Chinese characters. The stroke is the movement of the writing tool on the writing surface. The stroke data is obtained using the Chaizi database. In order to model The visual relationship between characters, the stroke order of each character is fed into another LSTM network to generate a glyph embedding vector.

S204, import the first sample embedding vector into the pre-training language model, and output the error correction result of the first pre-training text.

Specifically, the first sample embedding vector is imported into the pre-training language model, the first sample embedding vector is processed by the pre-training language model, the wrong text in the sample is predicted, the wrong text is corrected, and the corrected text is output , that is, the error correction result of the first pre-training text. It should be noted that the pre-trained language model may be a pre-trained language model widely used in the art, such as a Bert model and an AlBert model, which is not limited in this application.

In a specific embodiment of the present application, given a training sample, the predicted error probability of characters at the ith position in the training sample is:

P(y _i =j|X)=softmax(Wh _i +b)[j]

where P( _yi = j|X) is the conditional probability that the real character is predicted to be the _jth character in the vocabulary, X is the given corpus, hi is the output vector of the pre-trained language model, and W is the pre-training parameter, b is the bias in the neural network. The present application uses a supervised learning method to train the model, so that the pre-trained language model can learn the regularity of the appearance of correct Chinese characters as much as possible through training, so that the typos are restored to the correct Chinese characters at a specific position.

S205: Adjust the first training sample based on the error correction result of the first pre-training text to generate a second training sample.

Specifically, in this application, after the first sample embedding vector is imported into the pre-training language model, the first pre-training text error correction result is obtained, and the first pre-training text error correction result is the corrected text. The error correction result of the training text and the training corpus determine the error correction effect, and then adjust the proportion of each training corpus segment in the first training sample according to the error correction effect, so as to increase the proportion of samples that are beneficial to model training and reduce the proportion of samples that are not beneficial to model training. The proportion of samples obtained is a new training sample set, which further improves the error correction ability of the pre-trained language model.

Further, the step of adjusting the first training sample based on the error correction result of the first pre-training text to generate the second training sample specifically includes:

Calculate the action value score of each training corpus segment in the first training sample based on the error correction result of the first pre-training text;

Based on the action value score of each training corpus segment, the proportion of each training corpus segment in the first training sample is adjusted to obtain a second training sample.

Specifically, by comparing the error correction result of the first pre-training text with the training corpus to determine the number of error-corrected texts, the error correction effect of the pre-training language model is obtained, and the error correction effect of each training corpus segment in the first training sample is calculated according to the error correction effect. Action value score, adjust the proportion of each training corpus segment in the first training sample based on the action value score of each training corpus segment, obtain a second training sample, and iteratively train the pre-trained language model through the second training sample to further improve Error correction capabilities of pretrained language models.

Further, the step of calculating the action value score of each training corpus segment in the first training sample based on the error correction result of the first pre-training text specifically includes:

Determine the F1 value of the pre-trained language model based on the error correction result of the first pre-trained text;

Get the training cost of the pre-trained language model;

The action value score of each training corpus segment in the first training sample is calculated based on the F1 value of the pre-trained language model and the training cost.

Among them, F1 value, also known as F1 score (Score), also known as balanced F score (balanced F Score), F1 value is defined as the harmonic mean of precision and recall, F1 value can be regarded as model precision and recall A harmonic average of the rate, its maximum value is 1 and the minimum value is 0, to evaluate the analysis effect of the binary classification model.

Specifically, the first pre-trained text error correction result determines the F1 value of the pre-trained language model, obtains the training cost of the pre-trained language model, and calculates each training corpus segment in the first training sample based on the F1 value of the pre-trained language model and the training cost The action value score of , where the action value score is calculated as follows:

若较上一轮训练后的F1值降低，则该次训练对模型拟合无益，其行动价值得分为

In the formula, S ⁱ is the action value score of the ith pre-training, F1 _i is the F1 value of the ith pre-training, and T _Costi is the training cost of the ith pre-training. It should be noted that there are positive and negative values for the action value score. If the F1 value obtained after one round of training is higher than the F1 value after the previous round of training, this training is beneficial to the model fitting, and its action value score is

If it is lower than the F1 value after the previous round of training, this training is not beneficial to the model fitting, and its action value score is

In this application, the training overhead includes but is not limited to the node's CPU operation frequency, memory capacity, GPU operation frequency, GPU memory capacity, hard disk I/O throughput rate, uplink network bandwidth and other indicators. Since the dimensions of each indicator are inconsistent, It is necessary to remove the dimension first, and then normalize each indicator. The specific method is: count the value of an indicator x in a certain historical period, and sample the three values of the indicator x, which are the highest value x _max and the average value. x _ave and minimum value x _min , normalized as follows:

Among them, x is the normalized index, and then assign different weights to each normalized index, and multiply the weights to obtain the final training cost. The weight setting can be obtained by common AHP or machine learning methods. .

Further, the steps of adjusting the proportion of each training corpus segment in the first training sample based on the action value score of each training corpus segment to obtain the second training sample specifically include:

Determine the positive or negative value of the action value score of each training corpus segment;

When the action value score is a positive number, the proportion of training corpus segments corresponding to the action value score is increased;

When the action value score is negative, the proportion of training corpus segments corresponding to the action value score is lowered.

Specifically, determine the positive or negative value of the action value score of each training corpus segment. When the action value score is positive, the proportion of the training corpus segment corresponding to the action value score is increased. When the action value score is negative, the action value score is decreased. The proportion of the corresponding training corpus segments. For example, if the adjustment ratio is 5%, when the action value score is positive, the training corpus segment corresponding to the action value score is increased to account for 5% of the training samples. When the action value score is negative, the corresponding action value score is reduced. The training corpus fragments of 5% of the training samples account for 5% of the training samples. By increasing the proportion of samples that are useful for model training and reducing the proportion of samples that are unhelpful for model training, a new set of training samples is obtained, which further improves the performance of the pre-trained language model. Error correction capability.

S206, using the second training sample to iteratively train the pre-trained language model to obtain a text error correction model.

Specifically, the second training sample is imported into the pre-training language model, the action value score of the second training sample is calculated, and the model is iteratively trained by the action value score of the second training sample and the loss function of the pre-training language model, and the text is obtained. Error correction model.

Further, the steps of using the second training sample to iteratively train the pre-trained language model to obtain a text error correction model specifically include:

Import the second training sample into the pre-training language model to obtain the error correction result of the second pre-training text;

Calculate the action value score of each training corpus segment in the second training sample based on the error correction result of the second pre-training text;

Summing the action value scores of each training corpus segment in the second training sample to obtain the final action value score;

Based on the final action value score, the pre-trained language model is iteratively trained to obtain a text error correction model.

Specifically, the second training sample is imported into the pre-training language model to obtain the error correction result of the second pre-training text, and the action value score of each training corpus segment in the second training sample is calculated according to the above calculation method of the action value score. The action value scores of each training corpus segment in the training sample are summed to obtain the final action value score. Based on the final action value score, the pre-trained language model is iteratively trained to obtain a text error correction model. The final action value score is calculated based on the following formula:

In the formula, S is the final action value score, n is the number of iterations, and γ is the value decay coefficient.

It should be noted that since the action value score after each pre-training will have an impact on the subsequent training, and this impact will gradually decay with the increase of the number of iterations, it is also necessary to calculate the potential value score after training and define the value. Decay coefficient γ, the final action value score S is calculated by the value decay coefficient γ.

This application optimizes by adopting the strategy gradient method in the field of reinforcement learning, and maximizes the value score as much as possible. The specific process is as follows: train a multi-layer neural network corresponding to M actions, take a two-layer neural network as an example, The second sample embedding vector corresponding to the second training sample is input to the multi-layer neural network as the input vector v, and the weight matrix of the first hidden layer is set to w1, the relu activation function is used, the offset is b1, and the output o1=relu(w1*v +b1); set the weight matrix of the second hidden layer to be w2, the offset to be b2, and output o2=relu(w2*o1+b2), and then obtain the neural network output o3 through the softmax layer. In practice, more hidden layers can be used. layers for better results.

S207: Receive a text error correction instruction, obtain the text to be corrected, import the text to be corrected into a text error correction model, and output a text error correction result.

Specifically, the text error correction model obtains better text error correction ability after the training is completed. After receiving the text error correction instruction, the server obtains the text to be corrected, and imports the to-be-corrected text into the text error correction model to obtain the text correction. wrong result.

In this embodiment, the electronic device (for example, the server shown in FIG. 1 ) on which the reinforcement learning-based text error correction method runs may perform text error correction instructions through wired connection or wireless connection. It should be pointed out that the above wireless connection methods may include but are not limited to 3G/4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or developed in the future .

In the above embodiment, the present application discloses a text error correction method based on reinforcement learning, which belongs to the technical field of natural language processing. This application collects training corpus, performs similar pronunciation masking and glyph similarity masking on the text in the training corpus, uses the masked training corpus to construct a first training sample, imports the first training sample into a pre-training language model, and outputs the first pre-training language model. training text error correction results, adjusting the first training sample based on the first pre-training text error correction results, generating a second training sample, using the second training sample to iteratively train the pre-training language model, obtaining a text error correction model, and obtaining For the text to be corrected, import the text to be corrected into the text error correction model, and output the text error correction result. This application introduces pronunciation information and glyph information when training the error correction model, constructs training samples with rich noise through similar pronunciation masking and similar glyph masking, and further trains the error correction model through reinforcement learning technology, so that the model can better recognize spelling error, making the model more generalizable.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. , when the computer-readable instructions are executed, the processes of the above-mentioned method embodiments may be included. The aforementioned storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM).

It should be understood that although the various steps in the flowchart of the accompanying drawings are sequentially shown in the order indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order and may be performed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution sequence is also It does not have to be performed sequentially, but may be performed alternately or alternately with other steps or at least a portion of sub-steps or stages of other steps.

Further referring to FIG. 4 , as an implementation of the method shown in FIG. 2 above, the present application provides an embodiment of a text error correction device based on reinforcement learning, which corresponds to the method embodiment shown in FIG. 2 . , the device can be specifically applied to various electronic devices.

As shown in FIG. 4 , the text error correction device based on reinforcement learning described in this embodiment includes:

The text masking module 301 is used for collecting training corpus, and performing similar pronunciation masking and similar glyph masking on the text in the training corpus according to a preset text masking ratio;

A sample construction module 302, configured to construct a first training sample by using the masked training corpus;

The vector transformation module 303 is used to perform vector transformation on the first training sample to obtain the first sample embedding vector;

The pre-training module 304 is used to import the first sample embedding vector into the pre-training language model, and output the error correction result of the first pre-training text;

A sample adjustment module 305, configured to adjust the first training sample based on the error correction result of the first pre-training text to generate a second training sample;

The iterative training module 306 is used to iteratively train the pre-trained language model by using the second training sample to obtain a text error correction model;

The text error correction module 307 is configured to receive the text error correction instruction, obtain the text to be corrected, import the text to be corrected into the text error correction model, and output the text error correction result.

Further, please refer to FIG. 5, the text masking module 301 specifically includes:

The corpus dividing unit 311 is used for collecting training corpus, and dividing the training corpus to obtain several training corpus segments;

A masking text determining unit 312, configured to determine the target masking text in the training corpus segment according to a preset text masking ratio;

Similar text determination unit 313, for determining the pronunciation similar text and the glyph similar text corresponding to the target masked text in the preset text confusion set;

The text masking unit 314 is configured to perform pronunciation similarity masking and glyph similarity masking on the target masked text based on the pronunciation similar text and the glyph similarity text.

Further, the sample building module 302 specifically includes:

The first corpus combining unit is used to combine the training corpus segments that have completed similar masking of pronunciation and the training corpus segments that have completed similar glyph masking to form a first training sample.

Further, the sample building module 302 specifically includes:

The random masking unit is used to obtain random text from the preset vocabulary, and use the random text to perform random text masking on the target masking text;

The second corpus combining unit is used to combine the training corpus segments with similar pronunciation masking, the training corpus segments with similar glyph masking, the training corpus segments with random text masking, and the training corpus segments without text masking to form the first training sample .

Further, the sample adjustment module 305 specifically includes:

a first action value score calculation unit, configured to calculate the action value score of each training corpus segment in the first training sample based on the error correction result of the first pre-training text;

The sample adjustment unit is configured to adjust the proportion of each training corpus segment in the first training sample based on the action value score of each training corpus segment to obtain the second training sample.

Further, the first action value score calculation unit specifically includes:

The F1 value calculation subunit is used to determine the F1 value of the pretrained language model based on the error correction result of the first pretrained text;

The training cost calculation subunit is used to obtain the training cost of the pre-trained language model;

The first score calculation subunit is configured to calculate the action value score of each training corpus segment in the first training sample based on the F1 value of the pre-trained language model and the training cost.

Further, the sample adjustment unit specifically includes:

The action value judgment subunit is used to judge the positive and negative value of the action value score of each training corpus segment;

The first judgment result subunit is used to increase the proportion of the training corpus segment corresponding to the action value score when the action value score is a positive number;

The second judgment result subunit is used to lower the proportion of the training corpus segment corresponding to the action value score when the action value score is negative.

Further, the iterative training module 306 specifically includes:

a training sample importing unit, used for importing the second training sample into the pre-training language model to obtain the error correction result of the second pre-training text;

The second score calculation unit is used to calculate the action value score of each training corpus segment in the second training sample based on the error correction result of the second pre-training text;

The score summation unit is used to sum the action value scores of each training corpus segment in the second training sample to obtain the final action value score;

The iterative training unit is used to iteratively train the pre-trained language model based on the final action value score to obtain a text error correction model.

Further, the first sample embedding vector includes a text embedding vector, a position embedding vector, a pronunciation embedding vector and a glyph embedding vector, and the vector conversion module 303 specifically includes:

a feature extraction unit, which is used to perform feature extraction on the first training sample to obtain text features, position features, pronunciation features and glyph features;

The vector transformation unit is used for vector transformation of text features, position features, pronunciation features and glyph features respectively, to obtain text embedding vector, position embedding vector, pronunciation embedding vector and glyph embedding vector.

In the above embodiment, the present application discloses a text error correction device based on reinforcement learning, which belongs to the technical field of natural language processing. This application collects training corpus, performs similar pronunciation masking and glyph similarity masking on the text in the training corpus, uses the masked training corpus to construct a first training sample, imports the first training sample into a pre-training language model, and outputs the first pre-training language model. training text error correction results, adjusting the first training sample based on the first pre-training text error correction results, generating a second training sample, using the second training sample to iteratively train the pre-training language model, obtaining a text error correction model, and obtaining For the text to be corrected, import the text to be corrected into the text error correction model, and output the text error correction result. This application introduces pronunciation information and glyph information when training the error correction model, constructs training samples with rich noise through similar pronunciation masking and similar glyph masking, and further trains the error correction model through reinforcement learning technology, so that the model can better recognize spelling error, making the model more generalizable.

To solve the above technical problems, the embodiments of the present application also provide computer equipment. Please refer to FIG. 6 for details. FIG. 6 is a block diagram of the basic structure of a computer device according to this embodiment.

The computer device 6 includes a memory 61 , a processor 62 , and a network interface 63 that communicate with each other through a system bus. It should be pointed out that only the computer device 6 with components 61-63 is shown in the figure, but it should be understood that it is not required to implement all of the shown components, and more or less components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, special-purpose Integrated circuit (ApplicationSpecific Integrated Circuit, ASIC), programmable gate array (Field-Programmable GateArray, FPGA), digital processor (Digital Signal Processor, DSP), embedded equipment, etc.

The computer equipment may be a desktop computer, a notebook computer, a palmtop computer, a cloud server and other computing equipment. The computer device can perform human-computer interaction with the user through a keyboard, a mouse, a remote control, a touch pad or a voice control device.

The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), Magnetic Memory, Magnetic Disk, Optical Disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6 , such as a hard disk or a memory of the computer device 6 . In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (FlashCard) and so on. Of course, the memory 61 may also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for a text error correction method based on reinforcement learning, and the like. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or will be output.

In some embodiments, the processor 62 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. This processor 62 is typically used to control the overall operation of the computer device 6 . In this embodiment, the processor 62 is configured to execute computer-readable instructions stored in the memory 61 or process data, for example, computer-readable instructions for executing the reinforcement learning-based text error correction method.

The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

In the above embodiments, the present application discloses a computer device, which belongs to the technical field of natural language processing. This application collects training corpus, performs similar pronunciation masking and glyph similarity masking on the text in the training corpus, uses the masked training corpus to construct a first training sample, imports the first training sample into a pre-training language model, and outputs the first pre-training language model. training text error correction results, adjusting the first training sample based on the first pre-training text error correction results, generating a second training sample, using the second training sample to iteratively train the pre-training language model, obtaining a text error correction model, and obtaining For the text to be corrected, import the text to be corrected into the text error correction model, and output the text error correction result. This application introduces pronunciation information and glyph information when training the error correction model, constructs training samples with rich noise through similar pronunciation masking and similar glyph masking, and further trains the error correction model through reinforcement learning technology, so that the model can better recognize spelling error, making the model more generalizable.

The present application also provides another embodiment, that is, to provide a computer-readable storage medium, where the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to The at least one processor is caused to perform the steps of the reinforcement learning based text error correction method as described above.

The present application discloses a storage medium, which belongs to the technical field of natural language processing. This application collects training corpus, performs similar pronunciation masking and glyph similarity masking on the text in the training corpus, uses the masked training corpus to construct a first training sample, imports the first training sample into a pre-training language model, and outputs the first pre-training language model. training text error correction results, adjusting the first training sample based on the first pre-training text error correction results, generating a second training sample, using the second training sample to iteratively train the pre-training language model, obtaining a text error correction model, and obtaining For the text to be corrected, import the text to be corrected into the text error correction model, and output the text error correction result. This application introduces pronunciation information and glyph information when training the error correction model, constructs training samples with rich noise through similar pronunciation masking and similar glyph masking, and further trains the error correction model through reinforcement learning technology, so that the model can better recognize spelling error, making the model more generalizable.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of this application.

The present application may be used in numerous general purpose or special purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, including A distributed computing environment for any of the above systems or devices, and the like. The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The accompanying drawings show the preferred embodiments of the present application, but do not limit the scope of the patent of the present application. This application may be embodied in many different forms, rather these embodiments are provided so that a thorough and complete understanding of the disclosure of this application is provided. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements for some of the technical features. . Any equivalent structure made by using the contents of the description and drawings of the present application, which is directly or indirectly used in other related technical fields, is also within the scope of protection of the patent of the present application.

Claims (12)

1. a text error correction method based on reinforcement learning, is characterized in that, comprises: Collect training corpus, and mask the text in the training corpus according to the preset text masking ratio for pronunciation similarity masking and glyph similarity masking; Use the masked training corpus to construct a first training sample; performing vector transformation on the first training sample to obtain a first sample embedding vector; importing the first sample embedding vector into the pre-training language model, and outputting the error correction result of the first pre-training text; Adjusting the first training sample based on the error correction result of the first pre-training text to generate a second training sample; Perform iterative training on the pre-trained language model using the second training sample to obtain a text error correction model; A text error correction instruction is received, a text to be corrected is obtained, the text to be corrected is imported into the text error correction model, and a text error correction result is output. 2. the text error correction method based on reinforcement learning as claimed in claim 1, is characterized in that, described collection training corpus, and according to preset text covering ratio to the text in described training corpus to carry out similar pronunciation masking and glyph Similar masking steps include: Collect training corpus, and divide the training corpus to obtain several training corpus segments; Determine the target masking text in the training corpus segment according to the preset text masking ratio; Determine, in a preset text confusion set, texts with similar pronunciation and similar characters corresponding to the target masked text; Pronunciation similarity masking and glyph similarity masking are performed on the target masked text based on the phonetic similarity text and the glyph similarity text. 3. The text error correction method based on reinforcement learning as claimed in claim 2, wherein the step of constructing the first training sample with the masked training corpus specifically comprises: The first training sample is formed by combining the training corpus segment with the similar masked pronunciation and the training corpus segment with the similar glyph masked. 4. The text error correction method based on reinforcement learning as claimed in claim 2, wherein the step of constructing the first training sample with the masked training corpus specifically comprises: Obtain random text from a preset vocabulary, and use the random text to perform random text masking on the target masking text; Combining the training corpus fragments with similar pronunciation and masking, the training corpus fragments with similar glyphs, the training corpus fragments with random text masking, and the training corpus fragments without text masking, to form the first training sample . 5. The text error correction method based on reinforcement learning according to claim 3 or 4, wherein the first training sample is adjusted based on the first pre-training text error correction result to generate a second The steps of training samples include: Calculate the action value score of each training corpus segment in the first training sample based on the error correction result of the first pre-training text; The second training sample is obtained by adjusting the proportion of each training corpus segment in the first training sample based on the action value score of each training corpus segment. 6. The text error correction method based on reinforcement learning as claimed in claim 5, wherein the action value of each training corpus fragment in the first training sample is calculated based on the first pre-training text error correction result The steps for scoring include: Determine the F1 value of the pre-trained language model based on the error correction result of the first pre-trained text; obtaining the training cost of the pre-trained language model; The action value score of each training corpus segment in the first training sample is calculated based on the F1 value of the pre-trained language model and the training cost. 7. The text error correction method based on reinforcement learning as claimed in claim 5, wherein the ratio of each training corpus segment in the first training sample is adjusted based on the action value score of each training corpus segment , the steps of obtaining the second training sample specifically include: Judging the positive and negative value of the action value score of each training corpus segment; When the action value score is a positive number, increase the proportion of the training corpus segment corresponding to the action value score; When the action value score is a negative number, the proportion of the training corpus segment corresponding to the action value score is lowered. 8. The text error correction method based on reinforcement learning as claimed in claim 1, wherein the step of using the second training sample to iteratively train the pre-trained language model to obtain a text error correction model, specifically comprises : importing the second training sample into the pre-training language model to obtain a second pre-training text error correction result; Calculate the action value score of each training corpus segment in the second training sample based on the error correction result of the second pre-training text; Summing the action value scores of each training corpus segment in the second training sample to obtain a final action value score; The pre-trained language model is iteratively trained based on the final action value score to obtain a text error correction model. 9. The text error correction method based on reinforcement learning according to claim 1, wherein the first sample embedding vector comprises a text embedding vector, a position embedding vector, a pronunciation embedding vector and a glyph embedding vector, and the pair of The first training sample is transformed into a vector to obtain the first sample embedding vector, which specifically includes: Perform feature extraction on the first training sample to obtain text features, position features, pronunciation features and glyph features; The text feature, position feature, pronunciation feature and glyph feature are respectively vectorized to obtain the text embedding vector, the position embedding vector, the pronunciation embedding vector and the glyph embedding vector. 10. A text error correction device based on reinforcement learning, characterized in that, comprising: A text masking module, used for collecting training corpus, and masking the text in the training corpus according to the preset text masking ratio for similar pronunciation and similar characters; a sample building module, configured to use the masked training corpus to build a first training sample; a vector transformation module, for performing vector transformation on the first training sample to obtain a first sample embedding vector; A pre-training module, used for importing the first sample embedding vector into a pre-training language model, and outputting the error correction result of the first pre-training text; a sample adjustment module, configured to adjust the first training sample based on the error correction result of the first pre-training text to generate a second training sample; an iterative training module, configured to perform iterative training on the pre-trained language model by using the second training sample to obtain a text error correction model; The text error correction module is used for receiving a text error correction instruction, obtaining the text to be corrected, importing the text to be corrected into the text error correction model, and outputting the text error correction result. 11. A computer device, comprising a memory and a processor, wherein computer-readable instructions are stored in the memory, and when the processor executes the computer-readable instructions, any one of claims 1 to 9 is implemented The steps of the reinforcement learning-based text error correction method described in item. 12. A computer-readable storage medium, characterized in that, computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, any one of claims 1 to 9 is implemented. The steps of the reinforcement learning-based text error correction method described in item.

Images