CN117708339A - ICD automatic coding method based on pre-training language model - Google Patents

Abstract

The embodiment of the invention provides an ICD automatic coding method based on a pre-training language model, which belongs to the technical field of data processing and specifically comprises the following steps: constructing an ICD automatic coding data set; forming a mapping set; constructing a prefix tree, and forming an LEDT model by combining the prefix tree form; dividing the ICD automatic coding data set into a training set and a verification set; dividing clinical texts in the training set and the verification set and corresponding ICD codes thereof respectively; training the LEDT model using the seq2seq training dataset; inputting an input text in a data set to be encoded into a target model, limiting generated characters by using a prefix tree in the decoding generation process of the target model, simultaneously reserving k prediction descriptions with highest output scores by using a bundling algorithm, and finally converting the k output prediction descriptions into corresponding ICD codes by using a mapping set to serve as prediction output. By the scheme of the invention, the coding efficiency, the precision and the adaptability are improved.

Description

An ICD automatic encoding method based on pre-trained language model

Technical field

Embodiments of the present invention relate to the field of data processing technology, and in particular, to an ICD automatic encoding method based on a pre-trained language model.

Background technique

Currently, the coding task of the International Classification of Diseases (ICD) refers to assigning corresponding ICD codes to medical primitives in medical texts. It promotes the automation of medical research practices, improves coding quality, and reduces human errors. It is of great significance in aspects such as the influence of subjective factors, auxiliary diagnosis and treatment, supporting disease-related groups (DRGs), and realizing intelligent medical insurance fee control systems.

Early coding tasks were mainly done manually. According to statistics, it takes a coding expert more than 30 minutes on average to complete the coding task of an electronic medical record, which cannot meet the actual needs of the rapid growth of medical data. In addition, manual coding tasks require experts with sufficient background knowledge to carefully read the content of medical records based on consulting relevant information. This process is expensive, inefficient, and error-prone.

It can be seen that there is an urgent need for an efficient, accurate, and adaptable ICD automatic encoding method based on a pre-trained language model.

Contents of the invention

In view of this, embodiments of the present invention provide an ICD automatic encoding method based on a pre-trained language model, which at least partially solves the problems of poor encoding efficiency, accuracy, and adaptability in the prior art.

The embodiment of the present invention provides an ICD automatic encoding method based on a pre-trained language model, including:

Step 1: Construct an ICD automatic coding data set based on electronic medical records, where the ICD automatic coding data set includes clinical texts and their corresponding ICD codes;

Step 2: Obtain the code description corresponding to the ICD code from the ICD code description library and form a mapping set;

Step 3, segment the code description, obtain the ids sequence and construct a prefix tree accordingly, adjust the input range and field of view of the encoder based on the pre-trained model, and combine the prefix tree to form an LEDT model;

Step 4: Divide the ICD automatic encoding data set into a training set and a validation set;

Step 5: Segment the clinical text and its corresponding ICD code in the training set and validation set respectively to obtain the text sequence and its corresponding ICD code sequence, and pass the ICD code sequence through the mapping set to obtain the corresponding code description, thereby forming seq2seq Training data set and seq2seq validation data set;

Step 6: Use the teacher forcing method to train the LEDT model using the seq2seq training data set, update the model parameters, and after each training round, input the seq2seq verification data set into the LEDT model, and record the model parameters when the loss is minimal;

Step 7: Select the model parameters with the smallest loss in the seq2seq verification data set to obtain the target model, input the input text in the data set to be encoded into the target model, and use the prefix tree to limit the generated characters during the decoding and generation process of the target model. The string generated by the LEDT model is a subset of the code description. At the same time, the clustering algorithm is used to retain the k prediction descriptions with the highest output scores. Finally, the mapping set is used to convert the output k prediction descriptions into the corresponding ICD codes as the prediction output.

According to a specific implementation manner of the embodiment of the present invention, step 3 specifically includes:

Step 3.1, segment the code description and convert it into an ids sequence in the pre-trained language model. Add the ids of the start symbol used in the pre-training language model generation process before the ids sequence, and add model generation at the end of the ids sequence. The ids of the end symbols used in the process, and the target code ids sequence generated by the construction model;

Step 3.2: Perform the above operations on the ids sequence described by all codes to construct a prefix tree;

Step 3.3: Expand the range of input data that the encoder in the pre-training model can process, and set the field of view of its attention, combined with the prefix tree to form the LEDT model.

According to a specific implementation manner of the embodiment of the present invention, step 6 specifically includes:

Step 6.1, at each time step, use the ICD code of the seq2seq training data set as the input of the LEDT model at the current moment, use a word segmenter to segment the clinical text and code description corresponding to the ICD code, and convert it into a corresponding clinical text and code description. The ids sequence is used as the input of the encoder and decoder in the LEDT model;

Step 6.2: The encoder encodes the ids sequence of the clinical text to obtain the context encoding vector corresponding to each time step of the clinical text;

Step 6.3, input the context encoding vector corresponding to each time step and the ids of the code description into the decoder to obtain the prediction description, calculate the loss value between the prediction description and the code description, and update the model parameters of the LEDT model through backpropagation;

Step 6.4, after each training round, input the seq2seq validation data set into the LEDT model, and record the model parameters when the loss is minimal.

According to a specific implementation manner of the embodiment of the present invention, step 6.3 specifically includes:

Step 6.3.1, initialize the hidden state of the decoder of the previous time step to the value of the context encoding vector corresponding to the current time step;

Step 6.3.2, use the decoder hidden state of the previous time step and the symbols after code description segmentation as decoder input, and update the decoder hidden state;

Step 6.3.3, send the updated decoder hidden state to a linear layer network, and calculate the output of the current time step as the prediction description;

Step 6.3.4, calculate the loss value between the prediction description of the current time step and the code description of the next time step and update the model parameters of the LEDT model through back propagation, and return to step 6.3.1 to execute the next time step until completed The prediction of all time steps determines the end of the current training round;

Step 6.3.5, after completing the current training round, input the seq2seq validation data set into the LEDT model, and record the model parameters when the loss is minimal.

According to a specific implementation manner of the embodiment of the present invention, step 7 specifically includes:

Step 7.1, input the input text in the data set to be encoded into the target model. The word segmenter of the target model will segment the input text into words and convert it into an ids sequence. Use the encoder of the target model to encode it to obtain the context vector of the input text;

Step 7.2, set the first character generated by the decoder as the start character;

Step 7.3, use the decoder hidden state and prediction description of the previous time step as decoder input, and update the decoder hidden state;

Step 7.4, send the updated decoder hidden state to a linear layer network, and calculate the output of the current time step as the prediction description;

Step 7.5, query the prefix tree and set the probability of the symbol score of the prediction description that does not belong to the prefix tree to zero;

Step 7.6, use the beam search algorithm to retain the prediction descriptions with the highest k scores;

Step 7.7, repeat steps 7.3 to 7.6 until all time step predictions are completed, obtain k prediction descriptions, and use the mapping set to convert the output k prediction descriptions into corresponding ICD codes as prediction output.

The beneficial effects of the embodiments of the present invention are:

1. Using the pre-trained language model LEDT model for ICD automatic encoding tasks can better handle the morphological transformation problem of words in electronic medical records.

2. Use a generative approach to convert the ICD coding task from a multi-label classification task to a generation task. On the basis of the efficient denoising of the generative model, the ICD coding description is also effectively used to strengthen the interaction between the input text and the ICD coding description. Finally, the prefix tree is combined to limit the generation process of the LEDT model and avoid the Out-Of-Vocabulary (OOV) problem generated during the generation process.

3. Use Longformer as the architecture of the pre-trained language model, replacing the global attention of the basic Transformer with window-based local attention, so that the model can achieve a balance between computational complexity, memory overhead and model performance in the ICD encoding task. , improving coding efficiency, accuracy and adaptability.

Description of the drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention and are not relevant to the present invention. Those skilled in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is a schematic flow chart of an ICD automatic encoding method based on a pre-trained language model provided by an embodiment of the present invention;

Figure 2 is a schematic flowchart of a specific implementation process of an ICD automatic encoding method based on a pre-trained language model provided by an embodiment of the present invention;

Figure 3 is a schematic diagram of local attention and global attention in an LEDT model provided by an embodiment of the present invention;

Figure 4 is a schematic diagram of the generation process of an LEDT model combined with a prefix tree provided by an embodiment of the present invention;

Figure 5 is a statistical diagram of the length of the data set used in the embodiment of the present invention.

Detailed ways

The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some of the embodiments of the present invention, but not all of the embodiments. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, as long as there is no conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present invention.

To illustrate, the following describes various aspects of embodiments that are within the scope of the appended claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is illustrative only. Based on this disclosure, those skilled in the art will appreciate that one aspect described herein may be implemented independently of any other aspect, and that two or more of these aspects may be combined in various ways. For example, apparatuses may be implemented and/or methods practiced using any number of aspects set forth herein. Additionally, such apparatus may be implemented and/or methods practiced using other structures and/or functionality in addition to one or more of the aspects set forth herein.

It should also be noted that the drawings provided in the following embodiments are only schematically illustrating the basic concept of the present invention. The drawings only show the components related to the present invention and are not based on the number, shape and number of components during actual implementation. Dimension drawing, in actual implementation, the type, quantity and proportion of each component can be arbitrarily changed, and the component layout type may also be more complex.

Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, one skilled in the art will understand that the described aspects may be practiced without these specific details.

Currently, the following challenges and opportunities exist in the field of ICD automatic coding:

(1) Multi-label classification task: Mainstream scholars regard the ICD automatic encoding task as a multi-label classification task, but the huge label space of ICD codes makes it difficult for the model to accurately capture the relationship between ICD codes and input text fragments.

(2) Characteristics of medical natural language processing: The ICD automatic encoding task belongs to the field of medical natural language processing and involves issues such as inconsistent word forms and differences in writing styles. Medical texts may contain a large number of abbreviations, colloquial words, synonyms, and multiple words. meaning, multiple words with one meaning, and even spelling errors. Pre-training using a large-scale general corpus can improve the model's ability to understand the text context.

(3) Challenges in long text processing: The length of clinical text input in ICD automatic encoding tasks often exceeds the processing range of basic pre-trained language models. The fully connected attention mechanism and deep network characteristics of these basic pre-trained language models make them process The computational complexity and memory overhead increases exponentially when the text is long. Therefore, current ICD automatic encoding tasks still tend to use Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) or their variants to process input text, and less use of pre-trained language models . However, CNN-based models have difficulty accurately capturing the complex relationships between input text and labels, while RNN-based models may suffer from forgetting problems when processing long texts, which leaves them with room for improvement in ICD automatic encoding tasks.

(4) Challenges in noise processing: In ICD automatic encoding tasks, input text often contains a large amount of noise, which brings challenges to the encoding task. Through the Transformer-Encoder-Decoder class generative pre-training language model (such as BART), noise can be injected into the original text during the pre-training process, and then decoded by the decoder to restore the original text, thereby learning the knowledge in it. The BART model shows superior performance in tasks such as question and answer systems, summary generation, and machine translation. Its autoregressive method of generating text can better capture the interactive relationship between input text and target sentences.

Embodiments of the present invention provide an ICD automatic encoding method based on a pre-trained language model, which method can be applied to the electronic medical record encoding process in medical scenarios.

Refer to Figure 1, which is a schematic flow chart of an ICD automatic encoding method based on a pre-trained language model provided by an embodiment of the present invention. As shown in Figures 1 and 2, the method mainly includes the following steps:

Step 1: Construct an ICD automatic coding data set based on electronic medical records, where the ICD automatic coding data set includes clinical texts and their corresponding ICD codes;

During specific implementation, electronic medical record information can be collected. For a patient's medical record i, the relevant diagnostic information includes: Admission Diagnosis: , Discharge Diagnosis: , Procedure Name:/> , Preoperative Diagnosis:/> , Postoperative Diagnosis:/> , Medical Order:/> wait. We merge the relevant diagnostic information of the i-th visit into a visit document (Document):

At the same time, collect ICD codes involved in:

when Involving/> Time/> , on the contrary,/> . /> Represented as the ICD code space in this task (if there is no special explanation in the following parts, "/> ” both represent the number of elements in A). Therefore, the ICD automatic encoding task in this embodiment can be described as: for the input text/> , predicting all its involved ICD codes/> .

in:

.

Step 2: Obtain the code description corresponding to the ICD code from the ICD code description library and form a mapping set;

During specific implementation, the corresponding text descriptions of the ICD codes involved in the data set can be obtained from the ICD code description library. Construct a one-to-one mapping set of (ICD code, ICD code description). Considering that there is a one-to-one correspondence between ICD codes and code descriptions, the mapping set is reversible.

Step 3, segment the code description, obtain the ids sequence and construct a prefix tree accordingly, adjust the input range and field of view of the encoder based on the pre-trained model, and combine the prefix tree to form an LEDT model;

Based on the above embodiment, step 3 specifically includes:

Step 3.1, segment the code description and convert it into an ids sequence in the pre-trained language model. Add the ids of the start symbol used in the pre-training language model generation process before the ids sequence, and add model generation at the end of the ids sequence. The ids of the end symbols used in the process, and the target code ids sequence generated by the construction model;

Step 3.2: Perform the above operations on the ids sequence described by all codes to construct a prefix tree;

Step 3.3: Expand the range of input data that the encoder in the pre-training model can process, and set the field of view of its attention, combined with the prefix tree to form the LEDT model.

For specific implementation, describe all the codes in step 2

Word segmentation can be performed and converted into an ids sequence in the pre-trained language model. In front of the ids sequence, add the ids of the start symbol used in the pre-training language model generation process, and at the end of the ids sequence, add the ids used in the model generation process. For the ids of the ending symbol, construct the target code ids sequence generated by the model, perform the above operations on the ids sequences described by all codes, and construct a prefix tree Trie.

On this basis, considering that no checkpoints for the LEDT model in the Chinese medical field have yet been found, the Chinese BART pre-trained model can be modified: expand the range of input data that the encoder can process, and set its attention The field of view range is used to reduce its computational complexity to form an LED, and finally combined with the above-mentioned prefix tree to form an LEDT model. As shown in Figure 3, Longformer, as the architecture of the pre-training model, gives the global attention token [CLS], so that it can pay attention to all other tokens, and all other tokens can also pay attention to it. Other tokens can only focus on nearby tokens. However, since Longformer is stacked in blocks, when the number of stacked Longformer blocks is sufficient, the receptive field of the token of the top-level Longformer block that is only given local attention is also large enough. This kind of attention calculation This method enables the pre-trained language model to achieve a balance between computational complexity overhead, memory overhead and model performance in ICD encoding tasks.

Taking the ICD code describing "secondary malignant tumor of gastric lymph nodes" as an example, the result after word segmentation is , the corresponding ids sequence is

In this pre-trained language model, the start symbol of the model generation process is " ", the end symbol of the model generation process is also "/> ”, its ids correspond to 2. Therefore, the ids sequence of the target code of “secondary malignant tumor of gastric lymph nodes” is

.

Step 4: Divide the ICD automatic encoding data set into a training set and a validation set;

During specific implementation, the data set collected in step 1 can be Randomly divide the training set and the verification set Dev set at a ratio of 9:1, and set their corresponding index sets Train set index and Devset index for subsequent operation processes.

Step 5: Segment the clinical text and its corresponding ICD code in the training set and validation set respectively to obtain the text sequence and its corresponding ICD code sequence, and pass the ICD code sequence through the mapping set to obtain the corresponding code description, thereby forming seq2seq Training data set and seq2seq validation data set;

During specific implementation, the data in the training set and validation set are converted into seq2seq training data set and seq2seq validation data set. Taking the training set as an example, for each piece of data in the training set , Proceed as follows: Right/> Split by ICD code to form a training data set split by ICD code:

when When, pass (ICD code, ICD code description) to // Convert to ICD code description to form the final seq2seq training data set seq2seqTrain set. Its data format is

.

Step 6: Use the teacher forcing method to train the LEDT model using the seq2seq training data set, update the model parameters, and after each training round, input the seq2seq verification data set into the LEDT model, and record the model parameters when the loss is minimal;

Based on the above embodiment, step 6 specifically includes:

Step 6.1, at each time step, use the ICD code of the seq2seq training data set as the input of the LEDT model at the current moment, use a word segmenter to segment the clinical text and code description corresponding to the ICD code, and convert it into a corresponding clinical text and code description. The ids sequence is used as input to the encoder and decoder in the LEDT model;

Step 6.2: The encoder encodes the ids sequence of the clinical text to obtain the context encoding vector corresponding to each time step of the clinical text;

Step 6.3, input the context encoding vector corresponding to each time step and the ids of the code description into the decoder to obtain the prediction description, calculate the loss value between the prediction description and the code description, and update the model parameters of the LEDT model through backpropagation;

Step 6.4, after each training round, input the seq2seq validation data set into the LEDT model, and record the model parameters when the loss is minimal.

Further, the step 6.3 specifically includes:

Step 6.3.1, initialize the hidden state of the decoder of the previous time step to the value of the context encoding vector corresponding to the current time step;

Step 6.3.2, use the decoder hidden state of the previous time step and the symbols after code description segmentation as decoder input, and update the decoder hidden state;

Step 6.3.3, send the updated decoder hidden state to a linear layer network, and calculate the output of the current time step as the prediction description;

Step 6.3.4, calculate the loss value between the prediction description of the current time step and the code description of the next time step and update the model parameters of the LEDT model through back propagation, and return to step 6.3.1 to execute the next time step until completed The prediction of all time steps determines the end of the current training round;

Step 6.3.5, after completing the current training round, input the seq2seq validation data set into the LEDT model, and record the model parameters when the loss is minimal.

During the specific implementation, during the model training phase, the LEDT model is trained in the seq2seq manner. In this embodiment, the teacher forcing method is used for training, that is, in the process of training the seq2seq model, each time step does not use the output of the previous moment as Instead of directly using the label of the standard answer of the training data as the input at the current moment, the model can learn the conversion ability from the input clinical document DOC to the ICD code description (desc) corresponding to the document. for , use the tokenizer to separate the documents/> and ICD description Carry out word segmentation/> , converted into ids and used as input to the LED encoder and LED decoder. The LED encoder segments the document and converts the result into ids for encoding to obtain the context encoding vector about the DOC:

Refers to (/> ) first perform word segmentation (/> ) and then convert it into ids. During the decoding process, the generated tags are compared with/> The results after word segmentation and conversion into ids are aligned, and the training method uses teacher forcing. First initialize the decoder’s hidden state to the value of the context:

hypothesis The result of word segmentation and conversion into ids is/> For each time step, /> , perform the following operations:

Step 6.1: Update the hidden state of the LED decoder, and use the hidden state of the decoder at the previous moment and the symbols after word segmentation of the standard answer (description) as the decoder input.

Step 6.2: Feed the decoder hidden state into a linear layer network and calculate the output of the current time step

Step 6.3: Compute the description using the cross-entropy function with/> between losses.

When the seq2seq training data set is fed to the model, step 6.3 calculates and After the loss between them, the parameters of the model are updated through error back propagation to optimize model training. After each training round, the seq2seq validation data set is fed to the model. Step 6.3 calculates with/> After the loss between, calculate the loss of the model on the entire seq2seq validation data set, and save the model parameters when the loss is minimal.

Step 7: Select the model parameters with the smallest loss in the seq2seq verification data set to obtain the target model, input the input text in the data set to be encoded (set its index to Test set index) into the target model, and during the decoding and generation process of the target model, Use a prefix tree to limit the generated characters so that the strings generated by the LEDT model are a subset of the code descriptions. At the same time, a clustering algorithm is used to retain the k prediction descriptions with the highest output scores. Finally, the mapping set is used to output the k prediction descriptions. Convert to corresponding ICD code as prediction output.

Based on the above embodiment, step 7 specifically includes:

Step 7.1, input the input text in the data set to be encoded into the target model. The word segmenter of the target model will segment the input text into words and convert it into an ids sequence. Use the encoder of the target model to encode it to obtain the context vector of the input text;

Step 7.2, set the first character generated by the decoder as the start character;

Step 7.3, use the decoder hidden state and prediction description of the previous time step as decoder input, and update the decoder hidden state;

Step 7.4, send the updated decoder hidden state to a linear layer network, and calculate the output of the current time step as the prediction description;

Step 7.5, query the prefix tree and set the probability of the symbol score of the prediction description that does not belong to the prefix tree to zero;

Step 7.6, use the beam search algorithm to retain the prediction descriptions with the highest k scores;

Step 7.7, repeat steps 7.3 to 7.6 until all time step predictions are completed, obtain k prediction descriptions, and use the mapping set to convert the output k prediction descriptions into corresponding ICD codes as prediction output.

During the specific implementation, as shown in Figure 4, the parameters in the model loading step 6 are the parameters when the loss of the verification set is minimal. In the model testing and generation stage, the LEDT model is combined with the prefix tree Trie to generate ICD disease code descriptions, and combined with the beam search algorithm to achieve multi-label classification tasks. for Use the tokenizer to treat documents in the encoded data set separately/> , converted into ids and used as input to the encoder in the LEDT model. In the LEDT model, the encoder segments the document and converts the result into ids for encoding to obtain the context encoding vector about the DOC:

Initialize the decoder's hidden state to the context's value:

Assume that the first token generated by LEDT is

For each time step, , perform the following operations:

Step 7.1: Update the decoder hidden state, using the decoder hidden state and prediction result at the previous moment as the decoder input

Step 7.2: Send the decoder hidden state to a linear layer network Linear to calculate the output of the current time step:

Step 7.3: Query the prefix tree Trie and set the score probability of characters that do not belong to the prefix tree to zero.

Step 7.4: Use the beam search algorithm to retain the topk most likely output sequences.

After processing steps 7.1-7.4, we get the information about Description of topk ICD codes. Use (ICD code, ICD code description) mapping to convert the generated topk ICD code descriptions into about/> Topk code predictions

.

The ICD automatic encoding method based on the pre-trained language model provided in this embodiment can better handle the morphological transformation problem of words in electronic medical records by using the pre-trained language model LEDT to perform the ICD automatic encoding task; using a generative method to encode the ICD The task is converted from a multi-label classification task to a generation task. On the basis of the efficient denoising of the generative model, the ICD coding description is also effectively used to strengthen the interaction between the input text and the ICD coding description, and finally combines the prefix tree to limit the generation of LEDs. process to avoid the problem of unregistered words generated during the generation process; using Longformer as the architecture of the pre-trained language model, replacing the global attention of the basic Transformer with window-based local attention, allowing the model to achieve computational complexity in ICD coding tasks , achieve a balance between memory overhead and model performance, improving coding efficiency, accuracy and adaptability.

The present invention will be further described below with reference to a specific embodiment.

1. Data description: The data set verified in this experiment comes from the CHIP2022 evaluation five clinical diagnosis coding tasks. The length statistics are shown in Figure 5. Given the relevant diagnostic information for a visit (including admission diagnosis, preoperative diagnosis, postoperative diagnosis, and discharge diagnosis), as well as the name of the operation, the name of the drug, and the name of the medical order, it is required to provide its information in the "National Clinical Version of Disease Classification and Code 2.0" The corresponding ICD code in the vocabulary. All medical treatment data come from real medical data and are annotated based on the vocabulary of the "Classification and Code of Diseases National Clinical Version 2.0".

1.1, the number of documents corresponding to the training set, test set, and verification set, and the maximum number of tags per document (i.e. Medium/> ), the minimum number of tags, and the statistics of the average number of tags are shown in Table 1:

Table 1

1.1. The ICD code description in this example comes from the vocabulary of the "National Clinical Edition of Disease Classification and Codes 2.0".

2. Source of pre-trained language model parameters and the structure of LED: Since we have not found public pre-trained language model parameters for Chinese LED, this embodiment will use scripts to convert and expand based on BART, which is also a seq2seq model. The longest text size it can handle and adding a local attention mechanism to it form LED. In this embodiment, the pre-trained language model comes from a preset model database.

3. Evaluation indicators: The ICD automatic encoding task is a multi-label classification task. We use the commonly used evaluation indicators for ICD automatic encoding tasks: Micro_F1, Macro_F1, Micro_AUC, Macro_AUC, P@K. In this experiment, K=1, 2, 5. In particular, in the ablation experiment, because the generation model is used as the basis of the framework, it may cause OOV problems. We add len (OOV) as an evaluation index. len (OOV) represents the number of unregistered words generated by the model during the generation process. Total quantity.

3.1, Micro_F1 and Macro_F1: The ICD encoding task is a multi-label classification task, and Micro_F1 and Macro_F1 are performance indicators for evaluating the model in multi-label classification tasks. Micro_F1 takes the precision and recall of all labels into account and unifies them into an overall evaluation metric. Macro_F1 calculates the average of precision and recall for each label. These two indicators can measure the overall classification performance of the model for all ICD tags and the classification performance of each tag.

3.2, Micro_AUC and Macro_AUC: AUC (Area Under the ROC Curve) is a commonly used evaluation metric used to measure the classification quality of the model. In the ICD encoding task, Micro_AUC combines all predicted labels into a whole and calculates its AUC value. Macro_AUC calculates the true label and predicted probability of each label, and averages the AUC of all labels. These two metrics can evaluate the model's overall prediction quality and the difference between individual labels.

3.3, P@K: For the ICD encoding task, P@K is used to evaluate the average accuracy of the model in top-k predictions. It evaluates the ranking effect of the model in the prediction set and helps select the top-ranked prediction results. This is useful for ICD coding as it provides professionals with the most relevant ICD codes to reference.

4. Comparison with other methods:

This experiment selected several advanced models in ICD automatic encoding tasks as baseline models:

4.1, CAML: Use CNN to extract information from input text, and combine it with the label attention mechanism to perform ICD automatic encoding tasks.

4.2, LAAT: In order to alleviate the inconsistent relationship between input text segments corresponding to different ICD codes, LAAT uses bidirectional LSTM to extract the features of the input text, and uses a new label attention mechanism to associate these text segments to the corresponding ICD codes. .

4.3, MVC-LDA: This model uses multi-view convolution to extract text features from multiple angles, and introduces description information constraints to improve the prediction accuracy of the model.

4.4, KAICD: KAICD uses multi-scale CNN to extract input text features. On this basis, it uses bidirectional GRU to process ICD code descriptions to establish a code description knowledge base, and combines the attention mechanism to introduce the knowledge of the knowledge base into the prediction process of the model. The model achieved excellent results in the automatic ICD coding task on both the English MIMIC-III data set and the Chinese Xiangya data set.

4.5, LD-PLAM: This model proposes a new label attention mechanism-pseudo-label attention mechanism for ICD automatic encoding tasks. It uses the same attention mode for similar ICD codes to reduce computational overhead and improve the prediction performance of the model. This model achieved excellent results in the automatic ICD coding task on both the English MIMIC-III dataset and the Chinese Xiangya dataset.

Table 2

As can be seen from Table 2, the model performance of the proposed model LEDT_ICD is overall better than the current baseline model, which verifies the effectiveness of the generative pre-training language model used in the present invention for automatic encoding of ICD.

5. Ablation experiment

table 3

MSL_1024 (Max Source Length 1024) means that the input text will be truncated and the first 1024 characters will be retained and entered into the model. It can be seen from Table 3 that when combined with Trie, MSL_2048 is better than MSL_1024 in various evaluation indicators. This is because truncating the input text of the model will lead to the loss of information, and it also verifies that the local part in Longformer Effectiveness of attention mechanisms for modeling long texts. Compared with the LED_ICD (MSL_2048) model in line 2, because the average number of labels for each record in this data set is not large, the model takes the generated topk labels (when k is relatively small) as the prediction result when calculating the F1 score. , the results generated by Macro_F1 and Micro_F1 are not much different. The performance of LEDT_ICD (MSL_2048) combined with prefix tree is slightly better. In addition, the model combined with prefix tree does more tags than the generated model without combined prefix tree. There is an outstanding advantage when classifying tasks: it does not cause OOV problems.

It should be understood that various parts of the present invention may be implemented in hardware, software, firmware, or a combination thereof.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present invention. All are covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (5)

1. An ICD automatic coding method based on a pre-training language model, comprising:

step 1, an ICD automatic coding data set is constructed according to an electronic medical record, wherein the ICD automatic coding data set comprises clinical texts and corresponding ICD codes;

step 2, obtaining the code description corresponding to the ICD code from the ICD code description library, and forming a mapping set;

step 3, word segmentation is carried out on the code description to obtain an ids sequence, a prefix tree is constructed according to the ids sequence, the input range and the visual field range of the encoder are adjusted on the basis of a pre-training model, and an LEDT model is formed by combining the prefix tree form;

step 4, dividing the ICD automatic coding data set into a training set and a verification set;

step 5, respectively segmenting clinical texts in the training set and the verification set and corresponding ICD codes thereof to obtain text sequences and corresponding ICD code sequences thereof, and obtaining corresponding code descriptions through the mapping set by the ICD code sequences to form a seq2seq training data set and a seq2seq verification data set according to the text sequences and the corresponding ICD code sequences;

step 6, training the LEDT model by using a teacher training data set by using the seq2seq training data set, updating model parameters, inputting the seq2seq verification data set into the LEDT model after each training round is finished, and recording model parameters when the loss is minimum;

and 7, selecting model parameters with minimum loss in the seq2seq verification data set to obtain a target model, inputting an input text in the data set to be encoded into the target model, and in the decoding generation process of the target model, using a prefix tree to limit generated characters, so that character strings generated by the LEDT model are subsets in code description, simultaneously using a bundling algorithm to reserve k prediction descriptions with highest output scores, and finally using a mapping set to convert the output k prediction descriptions into corresponding ICD codes as prediction output.

2. The method according to claim 1, wherein the step 3 specifically comprises:

step 3.1, word segmentation is carried out on the code description, the code description is converted into an ids sequence in a pre-training language model, ids of a start symbol used in the generation process of the pre-training language model are added in front of the ids sequence, ids of an end symbol used in the generation process of the model are added at the tail of the ids sequence, and an object code ids sequence generated by the model is constructed;

step 3.2, performing the operation on the ids sequences described by all codes, and constructing a prefix tree;

and 3.3, expanding the range of input data which can be processed by the encoder in the pre-training model, setting the field of view of the attention of the encoder, and forming the LEDT model by combining the prefix tree form.

3. The method according to claim 2, wherein the step 6 specifically comprises:

step 6.1, using ICD codes of the seq2seq training data set as input of an LED T model at the current moment in each time step, using a word segmentation device to segment clinical texts and code descriptions corresponding to the ICD codes, converting the clinical texts and code descriptions into corresponding ids sequences, and using the ids sequences as input of an encoder and a decoder in the LED T model;

step 6.2, the encoder encodes the ids sequence of the clinical text to obtain a context encoding vector corresponding to each time step of the clinical text;

step 6.3, inputting the corresponding context coding vector and ids of the code description of each time step into a decoder to obtain a prediction description, calculating a loss value between the prediction description and the code description, and updating model parameters of the LEDT model through back propagation;

at step 6.4, after each training round has ended, the seq2seq verification dataset is entered into the LEDT model, recording the model parameters at which the loss is minimal.

4. A method according to claim 3, wherein said step 6.3 comprises:

step 6.3.1, initializing the hidden state of the decoder in the previous time step to be the value of the context coding vector corresponding to the current time step;

step 6.3.2, using the decoder hidden state and the code description word-segmented symbol in the previous time step as decoder input to update the decoder hidden state;

step 6.3.3, sending the updated decoder hidden state into a linear layer network, and calculating the output of the current time step as a prediction description;

step 6.3.4, calculating a loss value between the predicted description of the current time step and the code description of the next time step, updating model parameters of the LEDT model through back propagation, and returning to step 6.3.1 to execute the next time step until the prediction of all the time steps is completed, and judging that the current training round is finished;

and 6.3.5, after the current training round is completed, inputting the seq2seq verification data set into the LEDT model, and recording model parameters when the loss is minimum.

5. The method according to claim 4, wherein the step 7 specifically includes:

step 7.1, inputting an input text in a data set to be encoded into a target model, segmenting words of the input text by a word segmentation device of the target model, converting the words into an ids sequence, and encoding the ids sequence by an encoder of the target model to obtain a context vector of the input text;

step 7.2, setting the first character generated by the decoder as a start character;

step 7.3, the decoder hidden state and the prediction description of the previous time step are used as decoder input to update the decoder hidden state;

step 7.4, sending the updated decoder hidden state into a linear layer network, and calculating the output of the current time step as a prediction description;

step 7.5, inquiring the prefix tree, and setting the symbol score probability of the predictive description which does not belong to the prefix tree to zero;

step 7.6, reserving predictive descriptions with the highest score k by using a cluster search algorithm;

and 7.7, repeating the steps 7.3 to 7.6 until all time step predictions are completed, obtaining k prediction descriptions, and converting the output k prediction descriptions into corresponding ICD codes by using a mapping set to serve as prediction output.