CN104572736A - Keyword extraction method and device based on social networking services - Google Patents

Abstract

The present invention provides a method and device for extracting keywords based on a social network. The method includes: segmenting the text to be extracted, and counting the frequency of the word and the number of texts corresponding to the word; according to the frequency of the word and the number of texts corresponding to the word, Calculate the weight of words, select the first preset value of words with higher weight as candidate keywords, and extract the second preset value of candidate keywords with higher frequency of occurrence in the text to be extracted as keywords from the candidate keywords . The present invention performs noise filtering, text deduplication, word segmentation and word weight calculation on the text to be extracted, and then extracts keywords according to the word weight. Since a large amount of historical search information is not needed, the extraction speed is improved.

Description

Keyword extraction method and device based on social network

technical field

The present invention relates to the technical field of keyword extraction, in particular to a method and device for extracting keywords based on a social network.

Background technique

Keywords, as the subject words that the majority of social users pay attention to and use, can cover a large amount of information. By extracting keyword information from massive social texts, not only can timely understand the topics that the majority of social users are concerned about, but also help social users to grasp current hot information in a timely manner. Therefore, keyword extraction can effectively deal with the problem of information overload and provide fast and convenient information services for the majority of social users.

The ubiquitous keyword extraction method is as follows: obtain historical search information of a large number of users, and extract keywords according to the historical search information of users and subject words frequently appearing in webpage content.

However, the current method relies heavily on the user's search information, and needs to obtain a large amount of historical search information to accurately extract keywords, and the extraction speed is low.

Contents of the invention

(1) Solved technical problems

The technical problem solved by the invention is: how to solve the problem of obtaining a large amount of historical search information in the process of extracting keywords.

(2) Technical solutions

In order to solve the above technical problems, the present invention provides a method for extracting keywords based on social networks, including:

Segment the text to be extracted, and count the word frequency of the word and the number of texts corresponding to the word;

According to the number of texts corresponding to the word frequency and the word, calculate the weight of the word, select the word with the larger word weight of the first preset value as the candidate keyword, and extract the second preset value in the text to be extracted from the candidate keyword Candidate keywords with higher frequency appearing as keywords.

Preferably, before performing word segmentation on the text to be extracted, further comprising: performing noise filtering on the text to be extracted, and deduplicating the filtered text;

and / or,

The step of performing word segmentation on the text to be extracted further includes: performing part-of-speech tagging on the part-of-speech, and the part-of-speech is the first part-of-speech that meets the extraction rules or the second part-of-speech that does not meet the extraction rules; Using larger words as candidate keywords includes: selecting a first preset value of words with larger weights as candidate keywords from the words whose part of speech is the first part of speech.

Preferably, the noise filtering of the text to be extracted specifically includes:

Traversing the text to be extracted according to the noise filtering rules set, matching the characters in the text to be extracted, if the characters in the text to be extracted belong to the noise filtering rules, the matching is successful, and the characters that are successfully matched are deleted;

and / or,

The deduplication of the filtered text specifically includes:

Map the currently filtered text into fingerprint information, and compare the currently filtered text with the fingerprint information base, if the number of fingerprints with differences in the comparison result is less than or equal to the third preset value, then the current filtered text The text is deleted, otherwise, the fingerprint information of the currently filtered text is added to the fingerprint information database.

Preferably, the word frequency of said statistical word and the text number corresponding to the word further include:

Assign a word index number to each word that does not repeat, and save the word index number and the characteristics of the word corresponding to the index number in the word index table;

Assign a text index number to the deduplicated text, and save the text index number of the deduplicated text and the word index number of the word in the deduplicated text to the text index according to the positional relationship of the words in the deduplicated text table;

Wherein, the features of the word include: word frequency of the word, number of texts corresponding to the word, part of speech and word weight.

Preferably, the calculation of word weight specifically includes:

The word weight of a word is calculated according to the following formula:

$\mathrm{<span\; class="notranslate">\; weight</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; tf</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; df</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ,</span>$

Among them, weight(term) is the word weight, b(term) and a(term) are empirical correction values, tf(term) is the word frequency of the word, df(term) is the number of texts corresponding to the word, |d| is the total number of texts .

In order to solve the above technical problems, the present invention also provides a keyword extraction device based on social network, including:

The word segmentation module is used to perform word segmentation on the text to be extracted, and transmit the word after word segmentation to the statistics module;

The statistical module is used to count the word frequency of the word and the text number corresponding to the word, and transmit the statistical result to the calculation module;

The calculation module is used to calculate the word weight according to the word frequency and the text number corresponding to the word, and transmit the calculation result to the selection module;

The selection module is used to select the first preset word with larger word weight as the candidate keyword, and transmit the selection result to the advance module;

The extraction module is configured to extract a second preset value of candidate keywords that appear more frequently in the text to be extracted from the candidate keywords as keywords.

Preferably, the device also includes:

A noise filtering module, configured to perform noise filtering on the text to be extracted, and transmit the filtered text to the text deduplication module;

The text deduplication module is used to deduplicate the filtered text;

and / or,

The part-of-speech tagging module is used to carry out part-of-speech tagging on the word segmentation, the part-of-speech is the first part-of-speech that meets the extraction rules or the second part-of-speech that does not meet the extraction rules, and transmits the tagging results to the selection module;

The selection module is also used to select a first preset word with a higher weight as a candidate keyword from the words whose part of speech is the first part of speech.

Preferably, the noise filtering module includes:

The setting sub-module is used to set the noise filtering rules, and transmit the set noise filtering rules to the matching sub-module;

A traversal submodule, configured to traverse the text to be extracted, and transmit the traversal result to the matching submodule;

The matching submodule is used to match the characters in the text to be extracted according to the set noise filtering rules, if the characters in the text to be extracted belong to the noise filtering rules, the matching is successful, and the successfully matched characters are transmitted to the first remove submodule;

The first deletion submodule is used to delete characters that match successfully;

and / or,

The text deduplication module includes:

The mapping submodule is used to map the currently filtered text into fingerprint information, and transmit the mapping result to the comparison submodule;

The comparison submodule is used to compare the currently filtered text with the fingerprint information database, and transmit the currently filtered text whose fingerprint number is less than or equal to the third preset value to the second deletion The submodule, and the currently filtered text whose number of fingerprints with differences in the comparison result is not less than the third preset value is transmitted to the saving submodule;

The second deletion submodule is used to delete the currently filtered text;

The saving submodule is used to add the fingerprint information of the currently filtered text into the fingerprint information database.

Preferably, the device also includes:

An allocation module is used to allocate a word index number for each non-repetitive word, and allocate a text index number for the deduplicated text, and transmit the allocation result to the preservation module;

The preservation module is used to store the index number of the word and the features of the word corresponding to the index number in the word index table, and according to the positional relationship of the word in the text after the deduplication, the text index of the text after the deduplication No. and the word index number of the word in the deduplicated text are stored in the text index table;

Wherein, the features of the word include: word frequency of the word, number of texts corresponding to the word, part of speech and word weight.

Preferably, the calculation module is used to calculate the word weight of the word according to the following formula:

$\mathrm{<span\; class="notranslate">\; weight</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; tf</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; df</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ,</span>$

Among them, weight(term) is the word weight, b(term) and a(term) are empirical correction values, tf(term) is the word frequency of the word, df(term) is the number of texts corresponding to the word, |d| is the total number of texts .

(3) Beneficial effects

The present invention provides a method and device for extracting keywords in a social network, which does not need to search for information based on a large amount of history, but directly extracts keywords from the text to be extracted, and performs word segmentation and calculation of word weights on the text to be extracted. Word weight extracts keywords, and since a large amount of historical search information is not required, the extraction speed is improved.

Description of drawings

Fig. 1 is a flow chart of the method provided by Embodiment 1 of the present invention;

Fig. 2 is a flow chart of the method provided by Embodiment 2 of the present invention;

FIG. 3 is a schematic diagram of a word index table provided by Embodiment 2 of the present invention;

FIG. 4 is a schematic diagram of a text index table provided by Embodiment 2 of the present invention;

Fig. 5 is a schematic structural diagram of the device provided by Embodiment 3 of the present invention;

FIG. 6 is a schematic structural diagram of a noise filtering module provided by Embodiment 3 of the present invention;

FIG. 7 is a schematic structural diagram of a text deduplication module provided by Embodiment 3 of the present invention.

Detailed ways

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Example 1:

In order to solve the problem in the prior art that a large amount of historical data needs to be obtained for extracting keywords, an embodiment of the present invention provides a method for extracting keywords based on a social network, as shown in FIG. 1 , the method includes:

Step 101: Segment the text to be extracted, and count the word frequency of the word and the number of texts corresponding to the word;

Step 102: According to the word frequency and the number of texts corresponding to the word, calculate the weight of the word, select a word with a higher weight of the first preset value as a candidate keyword, and extract a second preset value of words from the candidate keywords Candidate keywords that appear more frequently in the text to be extracted are used as keywords.

The embodiment of the present invention does not need to search information based on a large amount of history, but directly extracts keywords from the text to be extracted, and then extracts keywords according to the weight of words by performing word segmentation and calculating word weights on the text to be extracted. Since a large number of historical searches are not required information, thereby increasing the extraction speed.

In the embodiment of the present invention, the fairness of keyword extraction is improved by performing noise filtering and deduplication on the text to be extracted in advance. According to the tagged part of speech, select a first preset word with a higher weight as a candidate keyword from the words meeting the first part of speech of the extraction rule. Therefore, the accuracy of keyword extraction is improved.

In the embodiment of the present invention, by setting the noise filtering rules, traversing the text to be extracted, and matching the characters in the text to be extracted, if the characters in the text to be extracted belong to the noise filtering rules, the matching is successful, and the successfully matched Characters are removed, so after filtering through noise, the processing efficiency and accuracy of keyword extraction are improved. Since there are some repetitive texts in the noise-filtered text, the remaining texts are non-repetitive texts after deduplication processing, thereby reducing the complexity of the text and improving the extraction accuracy of keywords.

In the embodiment of the present invention, by establishing a word index table and a text index table, it provides convenience for subsequent keyword extraction.

Example 2

In order to solve the problems of the prior art, the second embodiment of the present invention provides a method for extracting keywords based on a social network, as shown in FIG. 2 , the method includes:

Step 201: performing noise filtering on the text to be extracted;

In the embodiment of the present invention, because the text to be extracted contains a large amount of invalid information, not only the processing efficiency is reduced, but also the effect of keyword extraction is affected. therefore,

First, set the noise filtering rules;

In the embodiment of the present invention, the set noise filtering rules include: a, emoticon (generally appearing in the form of “[text]”) noise; b, “html tag” noise; c, “username” noise; d, "//username" noise.

Secondly, traverse the text to be extracted, and match the characters in the text to be extracted according to the above noise filtering rules;

Finally, if the characters in the text to be extracted belong to one of the above extraction rules, the matching is successful, and the matching characters are deleted.

The filtered text is obtained according to the above steps, which is beneficial to the filtered text and improves the processing efficiency and the accuracy of keyword extraction.

Step 202: deduplicate the filtered text;

Due to the forwarding relationship between texts, there are a lot of repetitions in the filtered texts. In order to reduce the unfairness of word weight calculation caused by repeated content, it is necessary to deduplicate the filtered texts.

The deduplication method includes: mapping the currently filtered text into fingerprint information, and comparing the currently filtered text with the fingerprint information database, if the number of fingerprints with differences in the comparison result is less than or equal to the preset value, then The currently filtered text is deleted; otherwise, the fingerprint information of the currently filtered text is added to the fingerprint information database.

For example, the currently filtered text is mapped to 6-digit fingerprint information, and the text with a difference of 3 or less fingerprints in the comparison result is deleted; otherwise, the fingerprint information of the currently filtered text is added to the fingerprint information database.

Step 203: Perform word segmentation and part-of-speech tagging on the deduplicated text, where the part-of-speech is the first part-of-speech that meets the extraction rules or the second part-of-speech that does not meet the extraction rules;

Step 204: Count the word frequency of the word and the number of texts corresponding to the word;

Step 205: using the word's part of speech, word frequency, and the number of texts corresponding to the word, to establish a word index table and a text index table;

First, assign a word index number for each word that does not repeat, and save the index number of the word and the feature of the word corresponding to the index number in the word index table, as shown in Figure 3;

The features of the word include: word frequency of the word, number of texts corresponding to the word, part of speech and weight. Among them, the weight is the value obtained in the subsequent steps.

Secondly, assign a text index number for the deduplicated text, and save the text index number of the deduplicated text and the word index number of the word in the deduplicated text according to the positional relationship of the word in the deduplicated text. In the text index table, as shown in Figure 4.

Step 206: Calculate the word weight according to the word frequency and the number of texts corresponding to the word;

According to the different backgrounds of users, a user-focused dictionary is established.

For example, financial related, sports related, entertainment related, etc.

Calculate the word weight according to the user's attention dictionary, word frequency, the number of texts corresponding to the word, and the following formula:

$\mathrm{<span\; class="notranslate">\; weight</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; tf</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; df</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ,</span>$

Among them, weight(term) is the word weight, b(term) is the empirical correction value based on the user-focused dictionary, a(term) is the empirical correction value based on part-of-speech judgment, tf(term) is the word frequency of the word, df(term) is the number of texts corresponding to the word, |d| is the total number of texts.

The value of a(term) is:

Among them, nr is the name of the person, and nt is the name of the organization;

The value of b(term) is: when the word belongs to the word in the user-focused dictionary, then b(term) is 1.5; when the word does not belong to the word in the user-focused dictionary, then b(term) is 1.

Step 207: Sorting the word weights from large to small, and selecting a word with a first preset value of word weight as a candidate keyword from the words whose part of speech is the first part of speech;

Among them, the first part of speech includes: /a adjective, /v verb, /j abbreviation, /ns place name, /nr person name, /nt organization name, /nz proper noun.

The second part of speech is a word of another part of speech that does not belong to the first part of speech.

Step 208: Extract a second preset value of candidate keywords that appear more frequently in the text to be extracted from the candidate keywords as keywords.

This step can be extracted according to the text index table, that is, a preset number of candidate keywords whose word index numbers appear frequently in the text index table are extracted as keywords.

As shown in FIG. 4 , the word whose index number is 7 appears frequently in multiple texts, and if the word whose index number is 7 is a candidate keyword, then the word whose index number is 7 is used as a keyword.

Among them, the embodiment of the present invention is applied to all social networking platforms such as Weibo and Space.

The embodiment of the present invention provides a keyword extraction method for social networks, which does not need to search for information based on a large amount of history, but directly extracts keywords from the text to be extracted, and performs noise filtering, text deduplication, and word segmentation on the text to be extracted. And calculate the word weight, and then extract keywords according to the word weight. Since a large amount of historical search information is not required, the extraction speed is improved.

Example 3

The embodiment of the present invention also provides a social network-based keyword extraction device, as shown in Figure 5, including:

The word segmentation module 501 is used to perform word segmentation on the text to be extracted, and transmit the word after the word segmentation to the statistics module;

The statistical module 502 is used to count the word frequency of the word and the text number corresponding to the word, and transmit the statistical result to the calculation module;

The calculation module 503 is used to calculate the word weight according to the word frequency and the corresponding text number of the word, and transmit the calculation result to the selection module;

The selection module 504 is used to select a word with a larger weight of the first preset value as a candidate keyword, and transmit the selection result to the advance module;

The extraction module 505 is configured to extract a second preset value of candidate keywords that appear more frequently in the text to be extracted from the candidate keywords as keywords.

Further, the device also includes:

A noise filtering module, configured to perform noise filtering on the text to be extracted, and transmit the filtered text to the text deduplication module;

The text deduplication module is used to deduplicate the filtered text;

and / or,

The part-of-speech tagging module is used to carry out part-of-speech tagging on the word segmentation, the part-of-speech is the first part-of-speech that meets the extraction rules or the second part-of-speech that does not meet the extraction rules, and transmits the tagging results to the selection module;

The selection module is also used to select a first preset word with a higher weight as a candidate keyword from the words whose part of speech is the first part of speech.

Further, the noise filtering module is shown in Figure 6, including:

A setting submodule 601, configured to set noise filtering rules, and transmit the set noise filtering rules to the matching submodule;

The traversal submodule 602 is configured to traverse the text to be extracted, and transmit the traversal result to the matching submodule;

The matching submodule 603 is configured to match the characters in the text to be extracted according to the set noise filtering rules, if the characters in the text to be extracted belong to the noise filtering rules, the matching is successful, and the characters that are successfully matched transmitted to the first deletion submodule;

The first deletion submodule 604 is configured to delete characters that match successfully;

Further, the text deduplication module is shown in Figure 7, including:

The mapping submodule 701 is used to map the currently filtered text into fingerprint information, and transmit the mapping result to the comparison submodule;

The comparison sub-module 702 is used to compare the currently filtered text with the fingerprint information database, and transmit the currently filtered text whose fingerprint number differs in the comparison result is less than or equal to the third preset value to the second Deleting the submodule, and transferring the currently filtered text whose fingerprint number differs in the comparison result is not less than the third preset value to the saving submodule;

The second deletion submodule 703 is used to delete the currently filtered text;

The saving submodule 704 is configured to add the fingerprint information of the currently filtered text into the fingerprint information database.

Further, the device also includes:

An allocation module is used to allocate a word index number for each non-repetitive word, and allocate a text index number for the deduplicated text, and transmit the allocation result to the preservation module;

The preservation module is used to store the index number of the word and the features of the word corresponding to the index number in the word index table, and according to the positional relationship of the word in the text after the deduplication, the text index of the text after the deduplication No. and the word index number of the word in the deduplicated text are stored in the text index table;

Wherein, the features of the word include: word frequency of the word, number of texts corresponding to the word, part of speech and word weight.

Further, the calculation module is used to calculate the word weight of the word according to the following formula:

$\mathrm{<span\; class="notranslate">\; weight</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; tf</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; df</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ,</span>$

Among them, weight(term) is the word weight, b(term) and a(term) are empirical correction values, tf(term) is the word frequency of the word, df(term) is the number of texts corresponding to the word, |d| is the total number of texts .

By providing a keyword extraction device for social networks, the present invention does not need to search for information based on a large amount of history, but directly extracts keywords from the text to be extracted through the extraction module, through the noise filtering module, text deduplication module, word segmentation module , Calculation module, respectively perform noise filtering, text deduplication, word segmentation and word weight calculation on the text to be extracted, and then extract keywords according to the word weight. Since a large amount of historical search information is not required, the extraction speed is improved.

The above embodiments are only used to illustrate the present invention, but not to limit the present invention. Those of ordinary skill in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all Equivalent technical solutions also belong to the category of the present invention, and the scope of patent protection of the present invention should be defined by the claims.

Claims (10)

1. based on a keyword extracting method for social networks, it is characterized in that, comprising:

Participle is carried out to text to be extracted, and the textual data that the word frequency of adding up word is corresponding with this word;

The textual data corresponding according to described word frequency and this word, calculate word weight, choose the first preset value word that word weight is larger alternatively keyword, from candidate keywords, extract the second preset value candidate keywords that the frequency of occurrences is larger in text to be extracted as keyword.

2. the method for claim 1, is characterized in that, described participle is carried out to text to be extracted before, to comprise further: noise filtering is carried out to text to be extracted, and by the text duplicate removal after filtering;

And/or,

Described the step that text to be extracted carries out participle to be comprised further: carry out part-of-speech tagging to participle, this part of speech is meet the first part of speech of extracting rule or do not meet the second part of speech of extracting rule; Choose then the first preset value word that word weight is larger alternatively keyword comprise: be the word of the first part of speech from part of speech, choose the first preset value word that word weight is larger alternatively keyword.

3. method as claimed in claim 2, is characterized in that, describedly carries out noise filtering to text to be extracted, specifically comprises:

According to the noise filtering rule of setting, travel through text to be extracted, mate the character in text to be extracted, if the character in text to be extracted belongs to described noise filtering rule, then the match is successful, by the character deletion that the match is successful;

And/or,

Described by the text duplicate removal after filtration, specifically comprise:

Text mapping after current filter is become finger print information, and the text after this current filter and finger print information storehouse are compared, if the fingerprint number that there are differences in comparative result is less than or equal to the 3rd preset value, then by the text suppression after current filter, otherwise, the finger print information of the text after current filter is added in described finger print information storehouse.

4. the method for claim 1, is characterized in that, the word frequency of described statistics word and textual data corresponding to this word, comprise further:

For each unduplicated word distributes glossarial index number, and the feature of the call number of word and the word corresponding with call number is saved in glossarial index table;

For the text after duplicate removal distributes text index number, according to the position relationship of word in the text after duplicate removal, the glossarial index number of word in the text after the text index of the text after duplicate removal number and this duplicate removal is saved in text index table;

Wherein, the feature of institute's predicate comprises: the word frequency of word, textual data, part of speech and word weight that this word is corresponding.

5. the method for claim 1, is characterized in that, described calculating word weight specifically comprises:

Word weight according to following formulae discovery word:

$\mathrm{weight}\left(\mathrm{term}\right)=b\left(\mathrm{term}\right)*a\left(\mathrm{term}\right)*\mathrm{tf}\left(\mathrm{term}\right)*\log \frac{\left|d\right|}{1+\mathrm{df}\left(\mathrm{term}\right)},$

Wherein, weight (term) is word weight, and b (term) and a (term) is experiential modification value, the word frequency that tf (term) is word, and df (term) is textual data corresponding to word, | d| is text sum.

6. based on a keyword extracting device for social networks, it is characterized in that, comprising:

Word-dividing mode, for carrying out participle to text to be extracted, and is transferred to statistical module by the word after participle;

Described statistical module, the textual data that word frequency and this word for adding up word are corresponding, and statistics is transferred to computing module;

Described computing module, for according to described word frequency and textual data corresponding to this word, calculates word weight, and result of calculation is transferred to and chooses module;

Describedly choose module, for choosing the first preset value word that word weight is larger alternatively keyword, and result will be chosen be transferred to module in advance;

Described extraction module, for extracting the second preset value candidate keywords that the frequency of occurrences is larger in text to be extracted as keyword from candidate keywords.

7. device as claimed in claim 6, it is characterized in that, described device also comprises:

Noise filtering module, for carrying out noise filtering to text to be extracted, and by the File Transfer after filtration to text duplicate removal module;

Described text duplicate removal module, for carrying out duplicate removal by the text after filtration;

And/or,

Part-of-speech tagging module, for carrying out part-of-speech tagging to participle, this part of speech is meet the first part of speech of extracting rule or do not meet the second part of speech of extracting rule, and chooses module described in annotation results being transferred to;

Describedly choose module, also for from part of speech be the first part of speech word in, choose the first preset value word that word weight is larger alternatively keyword.

8. device as claimed in claim 7, it is characterized in that, described noise filtering module comprises:

Setting submodule, for setting noise filtering rule, and by the noise filtering regular transmission of setting to matched sub-block;

Traversal submodule, for traveling through text to be extracted, and is transferred to described matched sub-block by traversing result;

Described matched sub-block, for the noise filtering rule according to setting, the character in text to be extracted is mated, if the character in text to be extracted belongs to described noise filtering rule, then the match is successful, and the character transmission that the match is successful is deleted submodule to first;

Described first deletes submodule, for by the character deletion that the match is successful;

And/or,

Described text duplicate removal module comprises:

Mapping submodule, for the text mapping after current filter is become finger print information, is transferred to comparison sub-module by mapping result;

Described comparison sub-module, for the text after this current filter and finger print information storehouse are compared, and the File Transfer fingerprint number that there are differences in comparative result being less than or equal to the current filter of the 3rd preset value deletes submodule to second, and the File Transfer of the current filter fingerprint number that there are differences in comparative result being not less than the 3rd preset value is to preserving submodule;

Described second deletes submodule, for by the text suppression after current filter;

Described preservation submodule, for adding the finger print information of the text after current filter in described finger print information storehouse.

9. device as claimed in claim 6, it is characterized in that, described device also comprises:

Distribution module, for distributing glossarial index number for each unduplicated word, and is the text distribution text index number after duplicate removal, and allocation result is transferred to preservation module;

Described preservation module, for the feature of the call number of word and the word corresponding with call number is saved in glossarial index table, and according to the position relationship of word in the text after duplicate removal, the glossarial index number of word in the text after the text index of the text after duplicate removal number and this duplicate removal is saved in text index table;

Wherein, the feature of institute's predicate comprises: the word frequency of word, textual data, part of speech and word weight that this word is corresponding.

10. device as claimed in claim 6, is characterized in that, described computing module, the word weight for according to following formulae discovery word:

$\mathrm{weight}\left(\mathrm{term}\right)=b\left(\mathrm{term}\right)*a\left(\mathrm{term}\right)*\mathrm{tf}\left(\mathrm{term}\right)*\log \frac{\left|d\right|}{1+\mathrm{df}\left(\mathrm{term}\right)},$

Wherein, weight (term) is word weight, and b (term) and a (term) is experiential modification value, the word frequency that tf (term) is word, and df (term) is textual data corresponding to word, | d| is text sum.