CN104750779A - Chinese multi-class word identification method based on conditional random field - Google Patents

Abstract

The invention discloses a method for identifying Chinese concurrent words based on a conditional random field, which includes: obtaining entries related to concurrent words, obtaining corpus from the entries; segmenting the corpus to generate language chunks, and simultaneously Generate the chunk features of each text; tag the text to obtain the part-of-speech features of the text, and use the chunk features and part-of-speech features to mark the text; randomly select a part of the corpus for training, and test the rest of the corpus to obtain the first experimental result ; Modify the feature template according to the characteristics of the corpus, continue to train and test the corpus after modification, and obtain the second experimental result; compare the performance of the metrics between the first experimental result and the second experimental result, and improve the recognition of concurrent words . The invention uses a conditional random field to identify Chinese concurrent words in the field of e-commerce, and after modifying the characteristics of the original conditional random field feature template, the accuracy rate, recall rate and f value of identifying concurrent words are all improved.

Description

A Method for Recognition of Chinese Concurrent Words Based on Conditional Random Field

technical field

The invention belongs to the field of text recognition of e-commerce products, and in particular relates to a method for recognizing Chinese concurrent words based on a conditional random field in the field of e-commerce.

Background technique

With the development of the times and the improvement of technology, ambiguous words (ambiguous words mean that the same word or words have two or more meanings, the reasons for ambiguity: unclear meaning, unfixed syntax, unclear levels, referring to Unidentified, etc.) has led to confusion in many contexts where the same word or phrases are interpreted differently by machines or humans. Therefore, whether the performance of ambiguous word recognition is accurate or not, whether it is efficient or not affects the result of processing text information. The ambiguous words are roughly divided into polysyllabic words, homonyms, polysemous words, concurrent words and anti-precepts. Previous recognition researches were limited to traditional Chinese word segmentation, and no specific field was studied. The present invention is only aimed at concurrent words in ambiguous words (concurrent words, that is, a word or word has two or more parts of speech ) in the field of e-commerce, and use the conditional random field feature template and the modified new feature template to train and test the corpus, the purpose is to optimize and improve the recognition performance of the conditional random field template in the e-commerce field.

The methods for Chinese character recognition are mainly divided into the following four categories:

1. A rule-based approach.

1a: String matching method. Match the words or phrases to be recognized with the dictionary (that is, a training set with a certain size). According to the matching direction, it can be divided into three types: forward matching, reverse matching and two-way matching. According to the principle of matching priority, it can be divided into two types: maximum matching and minimum matching.

1b: Shortest path algorithm. Dijkstra algorithm, Floyd algorithm, k-shortest path algorithm, n-shortest path algorithm and other graph theory algorithms and derived variant algorithms are used.

The above two methods are only a small part of the rule-based methods. The rule-based methods are identified according to the rules set by themselves. This method depends on whether the set rules are complete and reasonable. Any corpus is applicable, and it has poor performance and low accuracy for dealing with ambiguous words.

2. An understanding-based approach. This method analyzes syntax and semantics together, simulating people's understanding of words or words, and identifying corresponding words or words in this way. Due to the complexity of Chinese words or phrases and syntactic systems, this method requires a large amount of data and information as well as knowledge.

3. Transformation-based methods. This method is to find a corpus that has already marked the part of speech, identify the most suitable part of speech for each word or word from this corpus, and then use this as a training set, and then transform it into a new one by learning the existing rules. New rules (that is, variant transformations on some original rules).

4. Statistical-based methods. This method calculates the probability of each word and part of speech based on the compositional association and feature information before and after the word, and selects the optimal state transition probability to determine the word and part of speech. The three most representative models are Hidden Markov Model, Maximum Entropy Markov Model, and Conditional Random Field. The disadvantage of the hidden Markov model is that under the condition of a given observation sequence, the observation value only depends on the state, which makes each observation element exist independently, and in the real context, words are often not only related to the preceding and following words What is relevant is the feature information that has a certain relationship with more distant words, so it only achieves a local optimum. Although the maximum entropy Markov model takes into account the associated feature information between words that are farther away from the current word, when the state transitions, due to the unbalanced probability distribution of the number of branches, it leads to resident In a certain state, the bias problem is marked. Unlike the hidden Markov model and the maximum entropy Markov model, the state transition of the conditional random field is a directed graph. The characteristics of the undirected graph not only avoid the label bias problem of the maximum entropy Markov model, but also Taking into account the interrelated feature information between words that are farther away from the current word, it solves the situation that the words are too independent caused by hidden Markov only local normalization, and achieves global optimization.

Contents of the invention

The present invention proposes a kind of Chinese concurrent word recognition method based on conditional random field, comprises the following steps:

Step 1: Search for a Chinese concurrent word in the field of e-commerce, obtain an entry related to the concurrent word, and obtain a corpus with characteristics of the electric business field from the entry;

Step 2: Segment the corpus to generate language chunks, and simultaneously generate the language chunk features of each text in the language chunks;

Step 3: Carry out part-of-speech tagging to described text, obtain the part-of-speech feature of described text, utilize described chunk feature and described part-of-speech feature to mark described text;

Step 4: Randomly select a part of the corpus to train in the conditional random field, and test the rest of the corpus in the conditional random field to obtain the first experimental result;

Step 5: Modify the feature template in the conditional random field according to the characteristics of the corpus, and continue to train and test the corpus in the conditional random field after modification, and obtain the second experimental result;

Step 6: Comparing the performance of metrics between the first experimental result and the second experimental result to improve the recognition of concurrent words.

In the present invention's Chinese based on conditional random field and class word identification method, described step 1 comprises the following steps:

Step 1a: In the field of e-commerce, search according to the noun form of the concurrent word to obtain entries related to the noun form, classify the entries consistent with the product name as corpus, and classify the non-conforming words Items are changed into corresponding product names and classified as corpus;

Step 1b: Search according to the adjective form of the generic word, obtain the entries related to the adjective form, classify the entries that are consistent with the product name as the corpus, and modify the non-conforming entries to the corresponding products After the name, it is classified as corpus.

In the present invention's method for identifying Chinese words based on conditional random fields, in the step 2, according to the content contained in the products in the field of e-commerce, the entry is divided into a manufacturer block, a place of origin block, a brand block, and a product name blocks, and net weight blocks.

In the method for recognizing Chinese concurrent words based on conditional random fields of the present invention, in the step 2, if the language block contains more than two words, the language block feature of the first word is the initial word, and the language block feature of the remaining words is an initial word. The feature of a block is the following word; if the block contains a character, the feature of the block of the word is an independent block.

In the method for recognizing Chinese concurrent words based on conditional random fields of the present invention, in step 3, the part-of-speech features include nouns, verbs, and adjectives.

In the present invention's Chinese and class word recognition method based on conditional random field, described step 4 comprises the following steps:

Step 4a: from the corpus, randomly select the corpus containing an adjective form or a noun form of a concurrent word into the training set of the conditional random field for training; The corpus of the form is classified into the test set of the conditional random field for testing;

Step 4b: After completing the training and testing, repeat step 4a to randomly select another corpus for training and testing, until the training and testing are completed for all corpora.

In the present invention's Chinese based on conditional random field and class word recognition method, described step 5 comprises the following steps:

Step 5a: changing the combined feature associated with part of speech in the feature template of the conditional random field;

Step 5b: Return to step 4 to retrain the training set of each generic word and test the test set of each generic word to obtain the second experimental result.

In the present invention's Chinese based on conditional random field and class word recognition method, described step 6 comprises the following steps:

Step 6a: use the Conll 2000 algorithm based on Perl scripting language to carry out the performance comparison of three metric standards respectively to described first experimental result and described second experimental result; Described metric standard is precision rate, recall rate and f value;

Step 6b: If the second experimental result is lower than the first experimental result, return to step 5 to modify the feature template and obtain the second experimental result again until the second experimental result is better than the first experimental result up to the experimental results.

In the above content of the invention, the characteristics of the corpus include part of speech, semantics and the relationship between words and the like. Part-of-speech features include nouns, verbs, adjectives, etc.

The beneficial effect of the present invention is that: compared with the crf universal feature template, the modified feature template appears to be more matching when identifying concurrent words in the field of e-commerce.

Description of drawings

Fig. 1 is the flow chart of the present invention's method for recognizing Chinese concurrent words based on conditional random fields.

FIG. 2 is a specific flowchart of step 1.

FIG. 3 is a specific flowchart of step 2.

FIG. 4 is a specific flowchart of step 3.

FIG. 5 is a specific flowchart of step 4.

FIG. 6 is a specific flow chart of step 5.

FIG. 7 is a specific flow chart of step 6.

Detailed ways

The present invention will be further described in detail in conjunction with the following specific embodiments and accompanying drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the content specifically mentioned below, are common knowledge and common knowledge in this field, and the present invention has no special limitation content.

The present invention specifically comprises the following steps as shown in Figure 1:

Step 1: Search for a Chinese concurrent word in the field of e-commerce, obtain an entry related to the concurrent word, and obtain a corpus with characteristics of the electric business field from the entry;

Step 2: Segment the corpus to generate language chunks, and simultaneously generate the language chunk features of each text in the language chunks;

Step 3: Carry out part-of-speech tagging to described text, obtain the part-of-speech feature of described text, utilize described chunk feature and described part-of-speech feature to mark described text;

Step 4: Randomly select a part of the corpus to train in the conditional random field, and test the rest of the corpus in the conditional random field to obtain the first experimental result;

Step 5: modify the feature template in the conditional random field according to the part-of-speech of the corpus, semantics and the interrelationship between words, and continue to train and train the corpus in the conditional random field after modification Test to obtain the second experimental result;

Step 6: Perform a performance comparison of metrics on the experimental results to improve recognition of concurrent words.

The above-mentioned steps are explained in detail below in conjunction with specific embodiments to illustrate the technical solution of the present invention.

As shown in Figure 2, step 1 specifically completes the above through the following steps:

Step 1a: Log in to Yihaodian or the homepage of Taobao, search in the product search box according to the noun form of the said concurrent word, and obtain entries related to the noun form, words consistent with the product name on the product packaging picture The entry is classified as corpus, and the non-compliant entry is modified into the corresponding product name and classified as corpus. For example, some entries added some redundant attributive modifiers that did not appear in the product name: fresh organic pollution-free vegetables are naturally cooked in the open air without promoting red authentic tomatoes, but when you click to enter the product details page, you find that there are only fresh organic tomatoes on the product packaging.

Step 1b: Then input the adjective form of this generic word, intercept all product entries in the adjective form, and use the entries that are consistent with the product name on the product packaging picture directly as the experimental corpus, and modify the non-conforming entries into its products The product name displayed on the package is also used as the experimental corpus.

After completing the corpus to obtain the required characteristics of the e-commerce field, segment the corpus. Figure 3 shows the specific implementation process of step 2, which mainly includes the following steps:

Step 2a: Divide each intercepted commodity entry into a manufacturer block, a place of origin block, a brand block, a product name block, and a net content block. For example: Beitian Taiwan imported brown rice fruit roll milk-flavored children's biscuits 150G, which needs to be divided into: Beitian/manufacturer, Taiwan import/origin, brown rice fruit roll/brand, milk flavor children's biscuit/trade name, 150G/net content .

Step 2b: Divide each individual language chunk into words again, the language chunk feature of the first character of the individual language chunk is the initial word, marked with B, and then the language chunk features of all the words are followed words, It is marked with I, and if the word block only contains one word, then the word block feature of the word is an independent block, represented by O. For example: use B to represent the beginning word of each block, use I to represent the words that follow each block, for example: our sun we are a block I mark as B and they are marked as I and a separate block Labeled O sun for a block too labeled B yang labeled I.

After completing the segmentation of the corpus. Figure 4 shows the step 3 for the part-of-speech tagging of the corpus, as follows:

Step 3a: Carry out part-of-speech tagging on all the segmented individual words; nouns are marked as nouns, adjectives are marked as adjectives, and so on. The specific part-of-speech tagging correspondence is as follows:

Verb V V; bag M; bag/box M M; bag/group M M; bag/group bag M; bag group M; taste Z; brand N N; brand + category NL; brand + category + commodity NLC; brand + Commodity LC; brand + merchant LJ; brand + merchant + category NJL; brand + merchant + category + commodity NJLC; brand + merchant + commodity NJS; brand + color NNY; category NP; category + commodity PC; category + seasonal PT; commodity NPC; commodity + seasonal CT; merchant NJ; merchant + category JL; merchant + category + commodity JLC; merchant + commodity JC; place name NS NS; age T T; adjective A A; shape AD AD; mention M; branch M; quantity Q Q; Seasonal TG TG; Bar M; Cup M; Cup/Box M; Bucket M; Tablet M; Bottle M; Box M; Bowl M; /group M; tank group M; bag M; bag/group M; bag M; specification NG; quality NQ; conjunction H; amount NM; color NY.

For specific examples such as:

Jin N; Mai N; Lang N; Bone NP; Soup NP; Bomb NPC; Noodle NPC; Nong Z; Soup Z; Sea Z; Fresh Z;

Step 3b: For each word, combine two attributes, that is, whether the identity of the word is an initial word or a follower word or an independent block, and the part of speech marked by this word.

For specific examples such as:

Today N B-N; Mai N I-N; Lang N I-N; Bone NP B-NP; Soup NP I-NP; I-Z; Cup NPC B-NPC; Face NPC I-NPC; 7M B-M; 8M I-M; Gram M I-M;

After completing the part-of-speech tagging of the corpus. Figure 5 shows the steps for training and testing the corpus in step 4, as follows:

Step 4a: Randomly select the corpus containing the adjective form and the noun form of a certain generic word from the corpus of the correct tagged form that has met the conditions of the random field, and put it into the training set for training, while another part containing a certain generic word The corpus of adjective form and noun form is included in the test set for testing;

Step 4b: According to the method described in step 4a, continue to perform the same random training and testing on the remaining part-like words in the obtained e-commerce corpus, and obtain the first experimental result. The result of the first experiment is that the precision rate reaches 66.47%, the recall rate reaches 63.13%, and the f value reaches 65.36%.

The training method is as follows, enter the command prompt, enter the directory where the training set is stored, and enter the training command: ../crf_learn–[optional parameter] template train.data model, the test method is as follows, enter the command prompt, enter the test set storage directory, enter the test command: ../crf_test-[optional parameter]-m model test.data.

After completing the training and testing of the corpus. Figure 6 shows the step 5 to modify the feature template according to the corpus features to continue the training and testing steps, as follows:

Step 5a: On the basis of the original conditional random field feature template, continue to increase or decrease or change a certain combination of part-of-speech features. The original feature template of the conditional random field is as follows:

#Unigram

U00:%x[-2,0]

U01: %x[-1,0]

U02: %x[0,0]

U03: %x[1,0]

U04: %x[2,0]

U05: %x[-1,0]/%x[0,0]

U06: %x[0,0]/%x[1,0]

U07: %x[-2,1]

U08: %x[-1,1]

U09: %x[0,1]

U10: %x[1,1]

U11: %x[2,1]

U12: %x[-2,1]/%x[-1,1]

U13: %x[-1,1]/%x[0,1]

U14: %x[0,1]/%x[1,1]

U15: %x[1,1]/%x[2,1]

U16: %x[-2,1]/%x[-1,1]/%x[0,1]

U17: %x[-1,1]/%x[0,1]/%x[1,1]

U18: %x[0,1]/%x[1,1]/%x[2,1]

#bigram

B

After continuous experimentation and continuous modification of features, the final feature template with better performance for the experimental corpus than the original template is as follows;

#Unigram

U00:%x[-3,0]

U01: %x[-2,0]

U02: %x[-1,0]

U03: %x[0,0]

U04: %x[1,0]

U05: %x[2,0]

U06: %x[3,0]

U07: %x[-2,0]/%x[0,0]

U08: %x[-1,0]/%x[0,0]

U09: %x[0,0]/%x[1,0]

U10: %x[-2,0]/%x[-1,0]

U11: %x[1,0]/%x[2,0]

U12: %x[0,0]/%x[2,0]

Step 5b: Retrain the training set of each generic word and test the test set of each generic word according to the method described in step 4, and obtain the second experimental result. The result of this second experiment is that the precision rate reached 77.86%, the recall rate reached 73.17%, and the f value reached 74.59%.

After completing the modification of the conditional random field feature template. Figure 7 shows the performance comparison of the metrics for the experimental results in step 6, as follows:

Step 6a: Install the perl interpreter, use the Conll 2000 algorithm based on the Perl scripting language to compare the performance of the three metrics for the results of the two template tests, and automatically generate three metrics after importing the test set into the perl interpreter. Performance;

Step 6b: If the modified template is better than the original template in terms of the three performance indicators, the corpus is separated from the corpus after the first experiment, and a part of the corpus is randomly selected from the remaining corpus as the training set and part of the test set. The performance indicators under the two different template tests are compared. If it continues to be better than the original template, the corpus is also separated from the corpus after the second experiment, and the same training, testing and performance indicators are continued in the remaining corpus until Five times in a row, that is, the total number of training and testing entries is about 10,000 (that is, the total number of entries in each training set and test set is about 2,000). The results show that the new template is better than the original template, which means that the new template is in recognition. Concurrent words in the e-commerce field are better matched. As long as a certain performance index in each experiment result is not better than the original template, continue to modify the template and re-test.

The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages conceivable by those skilled in the art are all included in the present invention, and the appended claims are the protection scope.

Claims (8)

1. a kind of Chinese and class word recognition method based on conditional random field, is characterized in that, comprises the following steps: Step 1: Search for a Chinese concurrent word in the field of e-commerce, obtain an entry related to the concurrent word, and obtain a corpus with characteristics of the electric business field from the entry; Step 2: Segment the corpus to generate language chunks, and simultaneously generate the language chunk features of each text in the language chunks; Step 3: Carry out part-of-speech tagging to described text, obtain the part-of-speech feature of described text, utilize described chunk feature and described part-of-speech feature to mark described text; Step 4: Randomly select a part of the corpus to train in the conditional random field, and test the rest of the corpus in the conditional random field to obtain the first experimental result; Step 5: Modify the feature template in the conditional random field according to the characteristics of the corpus, and continue to train and test the corpus in the conditional random field after modification, and obtain the second experimental result; Step 6: Comparing the performance of metrics between the first experimental result and the second experimental result to improve the recognition of concurrent words. 2. the Chinese based on conditional random field as claimed in claim 1 is also characterized in that, described step 1 comprises the steps: Step 1a: In the field of e-commerce, search according to the noun form of the concurrent word to obtain entries related to the noun form, classify the entries consistent with the product name as corpus, and classify the non-conforming words Items are changed into corresponding commodity names and classified as corpus; Step 1b: Search according to the adjective form of the generic word, obtain the entries related to the adjective form, classify the entries that are consistent with the product name as the corpus, and modify the non-conforming entries to the corresponding products After the name, it is classified as corpus. 3. the Chinese concurrent word recognition method based on conditional random field as claimed in claim 1, is characterized in that, in described step 2, according to the contained content of product in electric business field, described entry is segmented into manufacturer's block, origin block, brand block, commodity name block, and net content block. 4. the Chinese concurrent word recognition method based on conditional random field as claimed in claim 1, is characterized in that, in described step 2, if comprise more than two words in the described language block, then the language of the first word The block feature is the initial word, and the block features of the remaining characters are the following words; if the block contains a word, the block feature of the text is an independent block. 5. the Chinese based on conditional random field as claimed in claim 1 concurrently class word recognition method is characterized in that, described step 3, described part-of-speech feature comprises noun, verb, adjective. 6. the Chinese based on conditional random field as claimed in claim 1 concurrently class word recognition method, it is characterized in that, described step 4 comprises the steps: Step 4a: from the corpus, randomly select the corpus containing an adjective form or a noun form of a concurrent word into the training set of the conditional random field for training; The corpus of the form is classified into the test set of the conditional random field for testing; Step 4b: After completing the training and testing, repeat step 4a to randomly select another corpus for training and testing, until the training and testing are completed for all corpora. 7. the Chinese based on conditional random field as claimed in claim 1 concurrently class word recognition method, it is characterized in that, described step 5 comprises the steps: Step 5a: changing the combined feature associated with part of speech in the feature template of the conditional random field; Step 5b: Return to step 4 to retrain the training set of each generic word and test the test set of each generic word to obtain the second experimental result. 8. the Chinese based on conditional random field as claimed in claim 1 concurrently class word recognition method, it is characterized in that, described step 6 comprises the steps: Step 6a: use the Conll 2000 algorithm based on Perl scripting language to carry out the performance comparison of three metric standards respectively to described first experimental result and described second experimental result; Described metric standard is precision rate, recall rate and f value; Step 6b: If the second experimental result is lower than the first experimental result, return to step 5 to modify the feature template and obtain the second experimental result again until the second experimental result is better than the first experimental result up to the experimental results.