CN106951437A - Identifying processing method and device suitable for the sensitive words and phrases of multiple Chinese - Google Patents

Abstract

The present invention provides a recognition processing method and device suitable for a plurality of Chinese sensitive words and sentences, the method includes: obtaining a plurality of preset sensitive words and sentences; establishing a suffix tree according to the sensitive words and sentences; obtaining Chinese text to be recognized; according to the The suffix tree matches the Chinese text to be identified; if the matching is successful, the sensitive words and sentences in the text to be identified are obtained and output and displayed. This method is aimed at the characteristics of Chinese, and the matching time of the pattern string on the suffix tree is determined by improve to To save time and improve the matching speed of pattern strings on the suffix tree, it is suitable for Chinese pattern string matching of multiple sensitive words and sentences.

Description

Recognition processing method and device suitable for multiple Chinese sensitive words and sentences

technical field

The invention relates to the technical field of computer processing, in particular to a recognition processing method and device suitable for multiple Chinese sensitive words and sentences.

Background technique

Identifying sensitive words and sentences refers to using the program to sniff the specified keywords in the information text, and check whether there is any behavior that violates the specified policy, which is the basis of sensitive word filtering. In order to find sensitive words quickly and accurately, some pattern matching algorithms need to be applied.

The pattern matching algorithms of pattern strings include Aho-Corasick (AC) algorithm, BM algorithm, and ACBM algorithm. Among them, the AC algorithm converts multiple pattern strings into a tree-type finite automatic state machine (DFSA) through preprocessing, and can complete all pattern string matching by scanning the text string once, and the time complexity of matching is O(n+m) . The time complexity of the BM algorithm is But it can't handle multi-pattern string matching. The ACBM algorithm combines the ideas of the AC algorithm and the BM algorithm. On average, the efficiency is better than that of the AC algorithm, and the time complexity is Although the ACBM algorithm performs well in practical applications, it is not effective for Chinese and has not fully utilized the characteristics of pattern strings and Chinese information, resulting in slow matching speed.

The reason for the inefficiency is that the basic structural unit of English is "word", while the basic structural unit of Chinese is "character". There is a big difference in the detection of sensitive words and sentences. For English, sensitive word detection is to sequentially match 26 English letters, while for Chinese it is to sequentially match tens of thousands of Chinese characters. Therefore, after the string matching algorithm changes from 26 English letters to tens of thousands of Chinese characters, it cannot achieve the expected effect of the algorithm in terms of time and space. In addition, Chinese characters are multi-byte symbols, and they also have attributes that English letters such as pinyin do not have, and have not been fully utilized in the algorithm.

Contents of the invention

The invention provides a recognition processing method and device suitable for multiple Chinese sensitive words and sentences, which are used to solve the problem of slow matching speed of Chinese sensitive words and sentences in the prior art.

In a first aspect, the present invention provides a recognition processing method suitable for multiple Chinese sensitive words and sentences, including:

Obtain multiple preset sensitive words and sentences;

Establish a suffix tree according to the sensitive words;

Obtain the Chinese text to be recognized;

Matching the Chinese text to be recognized according to the suffix tree;

If the matching is successful, the sensitive words and sentences in the Chinese text to be recognized are obtained and output for display.

Optionally, the establishment of a suffix tree according to the sensitive words and sentences includes:

S21. Establish a pattern string set P(P ₁ , P ₂ , P ₃ , P ₄ , P ₅ ... P _n ) according to multiple preset sensitive words and sentences;

S22. Setting a node, the attribute value of the root node is a first preset value, and the first preset value is an arrangement value of any pinyin letter;

S23. Select any sensitive word P _i in the set of pattern strings, the string length of the sensitive word P _i is m;

S24. Obtain the mth character of the sensitive word and sentence P _i , analyze the mth character to obtain the initial letter of the corresponding pinyin, and obtain the initial letter according to the corresponding relationship between the initial letter and the preset pinyin letter and the arrangement value the array value;

S25. Determine whether the arrangement value of the first letter is smaller than the first preset value, if it is smaller, set the node corresponding to the mth character on the left side of the root node, otherwise, set it on the left side of the root node Right;

S25. Acquire the m-1, m-2, ..., 2, 1 characters of the sensitive words and sentences P _i in turn, and the loop steps S24-S25 convert the m-1, m-2, ..., 2, 1 characters The node corresponding to the first character is set on the child nodes of the mth, m-1, ..., 2 character nodes.

Optionally, matching the Chinese text to be recognized according to the suffix tree includes: matching the Chinese text to be recognized according to the suffix tree using a BM algorithm.

Optionally, the sensitive words and sentences include single characters, phrases and sentences.

Optionally, it also includes: sending a prompt message if the matching is not successful.

In a second aspect, the present invention provides a recognition processing device suitable for multiple Chinese sensitive words and sentences, including:

The first obtaining module is used to obtain a plurality of preset sensitive words and sentences;

A processing module, configured to establish a suffix tree according to the sensitive words and sentences;

The second obtaining module is used to obtain the Chinese text to be recognized;

A matching module, configured to match the Chinese text to be recognized according to the suffix tree;

The display module is used to obtain the sensitive words and sentences in the Chinese text to be recognized and output and display after the matching is successful.

Optionally, the processing module is specifically configured to:

S21. Establish a pattern string set P(P ₁ , P ₂ , P ₃ , P ₄ , P ₅ ... P _n ) according to multiple preset sensitive words and sentences;

S22. Setting a node, the attribute value of the root node is a first preset value, and the first preset value is an arrangement value of any pinyin letter;

S23. Select any sensitive word P _i in the set of pattern strings, the string length of the sensitive word P _i is m;

S24. Obtain the mth character of the sensitive word and sentence P _i , analyze the mth character to obtain the initial letter of the corresponding pinyin, and obtain the initial letter according to the corresponding relationship between the initial letter and the preset pinyin letter and the arrangement value the array value;

S25. Determine whether the arrangement value of the first letter is smaller than the first preset value, if it is smaller, set the node corresponding to the mth character on the left side of the root node, otherwise, set it on the left side of the root node Right;

S25. Acquire the m-1, m-2, ..., 2, 1 characters of the sensitive words and sentences P _i in turn, and the loop steps S24-S25 convert the m-1, m-2, ..., 2, 1 characters The node corresponding to the first character is set on the child nodes of the mth, m-1, ..., 2 character nodes.

Optionally, the matching module is specifically configured to: use the BM algorithm to match the Chinese text to be recognized according to the suffix tree.

Optionally, the sensitive words and sentences include single characters, phrases and sentences.

Optionally, the display module is further configured to: issue a prompt message after the matching fails.

It can be known from the above technical solution that the method and device for identifying and processing multiple Chinese sensitive words and sentences of the present invention, by analyzing the acquired multiple preset sensitive words and sentences and using the arrangement values of pinyin characters to establish a suffix tree, after obtaining the Chinese text to be recognized, According to the suffix tree, the Chinese text to be identified is matched, and the letter arrangement value of the character is matched according to the branch. When the matching is successful, the sensitive words and sentences in the text to be identified are obtained and output and displayed, so as to achieve the characteristics of Chinese. The matching time of the pattern string on the suffix tree is determined by improve to To save time and improve the matching speed of pattern strings on the suffix tree, it is suitable for Chinese pattern string matching of multiple sensitive words and sentences.

Description of drawings

FIG. 1 is a schematic flow diagram of a recognition processing method suitable for multiple Chinese sensitive words and sentences provided by Embodiment 1 of the present invention;

FIG. 2 is a block diagram of a suffix tree provided by an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a recognition and processing device suitable for multiple Chinese sensitive words and sentences provided by Embodiment 2 of the present invention.

detailed description

The specific implementation manners of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

Fig. 1 shows that Embodiment 1 of the present invention provides a kind of recognition processing method suitable for multiple Chinese sensitive words and sentences, including:

S11. Obtain multiple preset sensitive words and sentences.

In this step, it should be noted that, in the embodiment of the present invention, the sensitive words and sentences are preset words and sentences. Generally, it can include words, phrases and sentences. Single words such as "stupid", "stupid". Phrases like "asshole" "violence". Statements such as "I hate China".

S12. Establish a suffix tree according to the sensitive words and sentences.

In this step, it should be noted that in the embodiment of the present invention, in order to facilitate subsequent matching of sensitive words and sentences from text information, a suffix tree needs to be established, as follows:

S21. Establish a pattern string set P(P ₁ , P ₂ , P ₃ , P ₄ , P ₅ ... P _n ) according to multiple preset sensitive words and sentences;

S22. Setting a node, the attribute value of the root node is a first preset value, and the first preset value is an arrangement value of any pinyin letter;

S23. Select any sensitive word P _i in the set of pattern strings, the string length of the sensitive word P _i is m;

S24. Obtain the mth character of the sensitive word and sentence P _i , analyze the mth character to obtain the initial letter of the corresponding pinyin, and obtain the initial letter according to the corresponding relationship between the initial letter and the preset pinyin letter and the arrangement value the array value;

S25. Determine whether the arrangement value of the first letter is smaller than the first preset value, if it is smaller, set the node corresponding to the mth character on the left side of the root node, otherwise, set it on the left side of the root node Right;

S25. Acquire the m-1, m-2, ..., 2, 1 characters of the sensitive words and sentences P _i in turn, and the loop steps S24-S25 convert the m-1, m-2, ..., 2, 1 characters The node corresponding to the first character is set to the mth, m-1, ..., 2 character nodes.

The above steps are explained with specific examples:

As shown in Figure 2, suppose the set of pattern strings is P(P ₁ , P ₂ , P ₃ , P ₄ ), P ₁ is "stupid", P ₂ is "pornographic", P ₃ is "You love France", P ₄ for "He hates France".

Set to set a node, and the attribute value of the root node is 13.

Acquire sensitive words and sentences P ₁ , the string length of the sensitive words and sentences P ₁ is 1. The initial letter "b" corresponding to the pinyin is obtained by parsing the character "stupid", and the array value "2" of the initial letter is obtained according to the correspondence between the initial letter and the preset pinyin letters and array values. If it is judged that "2" is less than the attribute value "13" of the root node, then the node with the character "stupid" is set on the left side of the root node as a child node of the root node.

Acquire sensitive words and sentences P ₂ , the string length of the sensitive words and sentences P ₂ is 2. Analyze the second character "Qing" to obtain the initial letter "q" corresponding to the pinyin, and obtain the array value "17" of the initial letter according to the corresponding relationship between the initial letter and the preset pinyin letter and the array value, and judge "17" is greater than the attribute value "13" of the root node, then the node of the character "love" is set on the right side of the root node as the child node of the root node. Analyze the first character "色" to obtain the initial letter "s" corresponding to the pinyin, and obtain the array value "19" of the initial letter according to the correspondence between the initial letter and the preset pinyin letter and the array value, and judge "19" is greater than the attribute value "17" of the character "love" node, then the character "color" node is set on the right side of the character "love" node as the child node of the root node.

The sensitive word and sentence P ₃ is obtained, and the string length of the sensitive word and sentence P ₃ is 4. Analyze the fourth character "国" to obtain the initial letter "g" corresponding to the pinyin, and obtain the array value "7" of the initial letter according to the corresponding relationship between the initial letter and the preset pinyin letter and the array value, and judge "7" is less than the attribute value "13" of the root node, then the node with the character "国" is set on the left side of the root node as the child node of the root node. Do the above-mentioned processing on "law", "love" and "you" in sequence, and will not go into details here, as shown in Figure 2.

Acquire sensitive words and sentences P ₄ , and the string length of the sensitive words and sentences P ₄ is 5. Analyze the fifth character "国" to obtain the initial letter "g" corresponding to the pinyin, and obtain the array value "7" of the initial letter according to the corresponding relationship between the initial letter and the preset pinyin letter and the array value, and judge "7" is less than the attribute value "13" of the root node, then the node with the character "国" is set on the left side of the root node as the child node of the root node. The above-mentioned processing is performed on "fa", "disgusting", "disgusting" and "he" in sequence, and will not be repeated here, as shown in Figure 2.

S13. Obtain the Chinese text to be recognized.

In this step, it should be noted that, in the embodiment of the present invention, the Chinese text to be recognized may be published articles or comment messages.

S14. Match the Chinese text to be recognized according to the suffix tree.

In this step, it should be noted that when the length of the character string P _i is greater than the character length of the text, at this time, it is impossible to find the character string P _i from the text, therefore, the character length of the Chinese text is greater than the pattern string P _i The character length of , that is, len(T)>maxlen(P _i ).

The BM algorithm is used to match the Chinese text to be recognized on the suffix tree, which may specifically include:

(1) According to the length minlen(P _i ) of the shortest pattern string P _i , select the minlen(P _i )th bit of the target string as the initial matching position, and use this tree for BM algorithm matching.

(2) If a certain character does not match, two heuristic rules are used, that is, the bad character rule and the good suffix rule.

(3) If a character is relatively matched, first judge the size of the left character and the matching character, and the size is assigned according to the pinyin of the character. If the left character is less than the matching character, then go to the left child node to search, if not, go to the right child node to search.

S15. If the matching is successful, obtain the sensitive words and sentences in the Chinese text to be recognized and output and display them.

In addition, if the matching is not successful, a prompt message can be issued to remind the user that the Chinese text can be published.

The recognition processing method suitable for a plurality of Chinese sensitive words and sentences provided by Embodiment 1 of the present invention is to obtain a plurality of preset sensitive words and sentences by analyzing and using the pinyin character arrangement value to establish a suffix tree. After obtaining the Chinese text to be recognized, according to the suffix The tree matches the Chinese text to be recognized, and matches according to the letter arrangement value of the characters. When the matching is successful, the sensitive words and sentences in the Chinese text to be recognized are obtained and output and displayed. The matching time on the suffix tree is given by improve to To save time and improve the matching speed of pattern strings on the suffix tree, it is suitable for Chinese pattern string matching of multiple sensitive words and sentences.

Figure 3 shows a recognition and processing device suitable for multiple Chinese sensitive words and sentences provided by Embodiment 2 of the present invention, including a first acquisition module 21, a processing module 22, a second acquisition module 23, a matching module 24 and a display module 25 ,in:

The first obtaining module 21 is used to obtain a plurality of preset sensitive words and sentences;

The processing module 22 is used to establish a suffix tree according to the sensitive words and sentences;

The second obtaining module 23 is used to obtain the Chinese text to be recognized;

Matching module 24, for matching the Chinese text to be recognized according to the suffix tree;

The display module 25 is configured to obtain the sensitive words and sentences in the Chinese text to be recognized and output and display after the matching is successful.

The processing module is specifically used for:

S21. Establish a pattern string set P(P ₁ , P ₂ , P ₃ , P ₄ , P ₅ ... P _n ) according to multiple preset sensitive words and sentences;

S22. Setting a node, the attribute value of the root node is a first preset value, and the first preset value is an arrangement value of any pinyin letter;

S23. Select any sensitive word P _i in the set of pattern strings, the string length of the sensitive word P _i is m;

S24. Obtain the mth character of the sensitive word and sentence P _i , analyze the mth character to obtain the initial letter of the corresponding pinyin, and obtain the initial letter according to the corresponding relationship between the initial letter and the preset pinyin letter and the arrangement value the array value;

S25. Determine whether the arrangement value of the first letter is smaller than the first preset value, if it is smaller, set the node corresponding to the mth character on the left side of the root node, otherwise, set it on the left side of the root node Right;

S25. Acquire the m-1, m-2, ..., 2, 1 characters of the sensitive words and sentences P _i in turn, and the loop steps S24-S25 convert the m-1, m-2, ..., 2, 1 characters The node corresponding to the first character is set on the child nodes of the mth, m-1, ..., 2 character nodes.

Since the principle of the apparatus described in Embodiment 2 of the present invention is the same as that of the method described in the foregoing embodiments, more detailed explanations will not be repeated here.

It should be noted that, in the embodiment of the present invention, a hardware processor (hardware processor) may be used to implement related functional modules.

The recognition processing device suitable for a plurality of Chinese sensitive words and sentences provided by Embodiment 2 of the present invention analyzes a plurality of preset sensitive words and sentences and uses pinyin character arrangement values to establish a suffix tree. After obtaining the Chinese text to be recognized, according to the suffix The tree matches the Chinese text to be recognized, and matches according to the letter arrangement value of the characters. When the matching is successful, the sensitive words and sentences in the Chinese text to be recognized are obtained and output and displayed. The matching time on the suffix tree is given by improve to To save time and improve the matching speed of pattern strings on the suffix tree, it is suitable for Chinese pattern string matching of multiple sensitive words and sentences.

Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

Those of ordinary skill in the art can understand that: the above embodiments are only used to illustrate the technical scheme of the present invention, rather than limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand : It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements to some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the claims of the present invention. range.

Claims (10)

1. A recognition processing method suitable for multiple Chinese sensitive words and sentences, characterized in that, comprising: Obtain multiple preset sensitive words and sentences; Establish a suffix tree according to the sensitive words; Obtain the Chinese text to be recognized; Matching the Chinese text to be recognized according to the suffix tree; If the matching is successful, the sensitive words and sentences in the Chinese text to be recognized are obtained and output for display. 2. The method according to claim 1, wherein said setting up a suffix tree according to said sensitive words and sentences comprises: S21. Establish a pattern string set P(P ₁ , P ₂ , P ₃ , P ₄ , P ₅ ... P _n ) according to multiple preset sensitive words and sentences; S22. Setting a node, the attribute value of the root node is a first preset value, and the first preset value is an arrangement value of any pinyin letter; S23. Select any sensitive word P _i in the set of pattern strings, the string length of the sensitive word P _i is m; S24. Obtain the mth character of the sensitive word and sentence P _i , analyze the mth character to obtain the initial letter of the corresponding pinyin, and obtain the initial letter according to the corresponding relationship between the initial letter and the preset pinyin letter and the arrangement value the array value; S25. Determine whether the arrangement value of the first letter is smaller than the first preset value, if it is smaller, set the node corresponding to the mth character on the left side of the root node, otherwise, set it on the left side of the root node Right; S25. Acquire the m-1, m-2, ..., 2, 1 characters of the sensitive words and sentences P _i in turn, and the loop steps S24-S25 convert the m-1, m-2, ..., 2, 1 characters The node corresponding to the first character is set on the child nodes of the mth, m-1, ..., 2 character nodes. 3. The method according to claim 1, wherein matching the Chinese text to be recognized according to the suffix tree comprises: matching the Chinese text to be recognized according to the suffix tree using a BM algorithm. 4. The method according to claim 1, wherein the sensitive words and sentences include single characters, phrases and sentences. 5. The method according to claim 1, characterized in that, if the matching is not successful, a prompt message is sent. 6. A recognition and processing device suitable for multiple Chinese sensitive words and sentences, characterized in that it comprises: The first obtaining module is used to obtain a plurality of preset sensitive words and sentences; A processing module, configured to establish a suffix tree according to the sensitive words and sentences; The second obtaining module is used to obtain the Chinese text to be recognized; A matching module, configured to match the Chinese text to be recognized according to the suffix tree; The display module is used to obtain the sensitive words and sentences in the Chinese text to be recognized and output and display after the matching is successful. 7. The device according to claim 6, wherein the processing module is specifically used for: S21. Establish a pattern string set P(P ₁ , P ₂ , P ₃ , P ₄ , P ₅ ... P _n ) according to multiple preset sensitive words and sentences; S22. Setting a node, the attribute value of the root node is a first preset value, and the first preset value is an arrangement value of any pinyin letter; S23. Select any sensitive word P _i in the set of pattern strings, the string length of the sensitive word P _i is m; S24. Obtain the mth character of the sensitive word and sentence P _i , analyze the mth character to obtain the initial letter of the corresponding pinyin, and obtain the initial letter according to the corresponding relationship between the initial letter and the preset pinyin letter and the arrangement value the array value; S25. Determine whether the arrangement value of the first letter is smaller than the first preset value, if it is smaller, set the node corresponding to the mth character on the left side of the root node, otherwise, set it on the left side of the root node Right; S25. Acquire the m-1, m-2, ..., 2, 1 characters of the sensitive words and sentences P _i in turn, and the loop steps S24-S25 convert the m-1, m-2, ..., 2, 1 characters The node corresponding to the first character is set on the child nodes of the mth, m-1, ..., 2 character nodes. 8. The device according to claim 6, wherein the matching module is specifically configured to: match the Chinese text to be recognized by using the BM algorithm according to the suffix tree. 9. The device according to claim 6, wherein the sensitive words and sentences include single characters, phrases and sentences. 10. The device according to claim 6, wherein the display module is further configured to: send a prompt message after the matching fails.