JP2006350656A - Time-series document grouping method, device, and program, and recording medium storing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To collect documents handling the same subject matter with a document having a plain style, such as a diary and one's thoughts, described in an arbitrary style as a target. <P>SOLUTION: A time-series document grouping device analyzes an input document, acquires a general word that is a word in the document and the appearance position information of the general word, acquires a keyword that becomes the candidate of a main keyword and the appearance position information of the keyword in the input document, totals the keyword, selects a frequent keyword, divides a document including the selected keyword into a subset, determines whether a document in the subset describes the same topic, based on the result of the totalled general word appearing within a predetermined range around the keyword in the document in the subset, integrates the determination result for each subset, and finally determines whether the keyword is the center of the topic. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a time series document aggregation method and apparatus, a program, and a storage medium storing the program. The present invention acquires a document dealing with the same subject from a large number of documents, and represents a central group representing the acquired document group. Time-series document aggregation method, apparatus, and program for acquiring keywords accurately, especially for documents with broken writing style such as diaries and impressions, even when topics gradually change with time And a storage medium storing the program.

  Clustering is a typical technique for collecting documents with similar contents. As a clustering method, for example, there is a k-means method in which elements to be classified are expressed as vectors and elements having a small distance between the vectors are aggregated into the same cluster, which is widely used. When this method is applied to document classification, a document vector obtained by weighting each word with TF-IDF (Term Frequency & Inverse Document Frequency) or the like is used (for example, see Non-Patent Document 1).

In addition, a technique has been proposed in which word weighting is improved and clustering accuracy is increased by using a document vector in which words having particularly high importance are emphasized.
Translated by Morio Onoe, "Pattern Identification" New Technology Communications pp.528-529

  Existing document classification techniques can be expected to have a certain level of accuracy for relatively well-written sentences such as news articles and papers. However, documents such as personal diaries and impressions not only have stylistic fluctuations and differences in the length of documents, but may also include spoken words and dialects. Also, even if a document deals with the same topic, such as an impression of a movie or a document that is pleasing to achieve a great record for a certain athlete, the viewpoint differs depending on the person who describes it. There are many cases where there are various features.

  In addition, in the above-described prior art, since one document vector is given to a document in which a plurality of subjects are described in one document and classification processing is performed, the document vector may be ambiguous depending on the number of topics included. And the classification accuracy may be lowered.

  For the above reasons, the existing document classification technology does not work effectively for miscellaneous documents such as personal diaries and impressions, and it is difficult to form a set of documents that are similar in content. is there.

  The present invention has been made in view of the above points, and targets the same subject matter for enormous document groups, especially documents of broken style such as diaries and impressions written by an unspecified number of people in arbitrary style. It is an object of the present invention to provide a time-series document aggregation method and apparatus capable of aggregating handled documents, a program, and a storage medium storing the program.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) provides time information in an apparatus having a document analysis unit, a keyword extraction unit, a keyword aggregation unit, a subset generation unit, a subset determination unit, and a comprehensive determination unit. A time-series document aggregation method that aggregates documents that describe the same topic among a large number of documents that have been collected, and that also acquires axis keywords that are central keywords that represent the aggregated documents. ,
A document analysis unit analyzes the input document, acquires a general word that is a word in the document, and appearance position information of the general word, and stores it in the general word storage unit (step 1);
A keyword extraction step (step 2) in which the keyword extraction unit acquires the keyword as a candidate for the axis keyword and the appearance position information of the keyword in the input document, and stores the keyword in the keyword storage unit
A keyword selection step (step 3) in which the keyword counting means totals the keywords stored in the keyword storage means and selects high-frequency keywords;
A subset generation step (step) in which the subset generation unit acquires a document including the keyword selected in the keyword selection step from the document storage unit and divides each document into subsets based on time information. 4) and
The subset determination means obtains, from the general word storage means, general words that appear within a predetermined range around the keyword in the documents in the subset, and based on the totaled results, A subset determination step (step 5) for determining whether or not documents in the document are described on the same topic;
A comprehensive determination unit performs a comprehensive determination step (step 6) for integrating the determination results for each subset and finally determining whether or not the keyword is the topic axis.

Further, the present invention (Claim 2) provides a subset generation step (Step 4).
A subset is generated by aggregating documents having similar time information.

Further, according to the present invention (Claim 3), in the subset generation step (Step 4),
A subset is generated by aggregating documents having similar time information and similar contents.

Further, the present invention (Claim 4), in the subset determination step (Step 5),
General words that appear within a predetermined range around the selected keywords are aggregated for each subset, and the documents in the subset are based on the same topic based on the number of common words that appear frequently at a certain rate or higher. Whether or not is described is determined.

Further, according to the present invention (Claim 5), in the subset determination step (Step 5),
The number of documents X included in the subset is aggregated, and general words appearing at a high frequency Y within a predetermined range around the keyword in the X documents are acquired, and the ratio of Y to X is high When the number of general words is more than a certain value, it is determined that the documents in the subset are described on the same topic.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 6) is an axis that is a central keyword representing a group of documents that aggregates documents describing the same topic among a large number of documents to which time information is assigned. A time-series document aggregation device that acquires keywords at the same time,
Document storage means 201 for storing documents;
A document analysis unit 202 that analyzes the input document read from the document storage unit 201, acquires a general word that is a word in the document, and appearance position information of the general word, and stores the information in the general word storage unit 203;
A keyword extraction unit 204 that acquires keywords that are candidates for axis keywords and appearance position information of the keywords in the input document and stores them in the keyword storage unit 205;
A keyword totaling unit 206 that totals the keywords stored in the keyword storage unit 205 and selects high-frequency keywords;
A subset generation unit 2071 that acquires documents including keywords selected by the keyword aggregation unit 206 from the document storage unit 201 and divides each document into subsets based on time information;
Among the documents in the subset, general words appearing within a predetermined range around the keyword are obtained from the general word storage means 203, and based on the totaled results, the documents in the subset are Subset determination means 2072 for determining whether or not the same topic is described;
A judgment result for each subset is integrated to finally determine whether or not the keyword is a topic axis, and when the keyword is a topic axis, a comprehensive judgment unit 2073 for storing in the aggregated document storage unit 208;
Have.

Further, the present invention (Claim 7) is the subset generation means 2071,
Means for generating a subset by aggregating documents whose document time information is close to each other;

Further, the present invention (Claim 8) is the subset generation unit 2071,
And means for generating a subset by aggregating documents having similar time information and similar contents.

Further, the present invention (claim 9) is the subset judging means 2072,
General words that appear within a predetermined range around the selected keywords are aggregated for each subset, and the documents in the subset are based on the same topic based on the number of common words that appear frequently at a certain rate or higher. Means for determining whether or not is described.

Further, the present invention (claim 10) is the subset determining unit 2072,
The number of documents X included in the subset is aggregated, and general words appearing at a high frequency Y within a predetermined range around the keyword in the X documents are acquired, and the ratio of Y to X is high Means for determining that the documents in the subset are described for the same topic when the number of general words is greater than a certain value.

  The present invention (Claim 11) is a program that causes a computer to function as a time-series document aggregating apparatus having means according to any one of Claims 6 to 10.

  The present invention (Claim 12) is a storage medium storing a program for causing a computer to function as a time-series document aggregating apparatus having means according to any one of Claims 6 to 10.

  The present invention is a method for determining whether or not a keyword is actually an axis keyword by first finding candidate axis keywords and counting up words appearing around the keyword. For each subset obtained by dividing a document including a keyword based on time information, it is determined whether or not the keyword is an axis keyword, and the determination result is integrated so that the entire document including the keyword is the same topic. This is an aggregation method that can cope with the case where the content of a topic gradually changes with time.

  In addition, since each of the subsets is determined based on the statistical information of the axis keyword candidates and the words (general words) appearing in the vicinity, documents that contain no punctuation, spoken words, or dialects Even documents with broken style such as can be classified with high accuracy. In particular, it is possible to collect only useful documents written with the same interest by excluding documents that are not topical, such as a diary or an extremely short document that only describes individual actions.

  In addition, even when there are descriptions of a plurality of topics in one document, the topics are aggregated using words appearing around the keyword of interest. For example, a document in which the first half is written about topic A and the second half is written about topic B Is aggregated as a constituent document of topic A from the distribution of words appearing in the first half and is also aggregated as a constituent document of topic B from the distribution of words appearing in the second half. Can be classified.

  Furthermore, not only the documents including the same topic are aggregated, but also the axis keyword that is the basis for the aggregation can be acquired, so various applications such as document search and summarization using the implementation results of the present invention are possible.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 3 shows the configuration of the time-series document aggregation device according to the first embodiment of the present invention.

  The time-series document aggregation device shown in FIG. 1 includes a document storage unit 201 that stores documents to be aggregated together with time information, a document analysis unit 202 that analyzes an input document and divides the document into elements that constitute a sentence, and document analysis A general word storage unit 203 that records a general word acquired based on the analysis result of the unit 202 and a position in the document, and a keyword extraction unit that extracts a keyword characterizing each document based on the analysis result of the document analysis unit 202 205, a keyword storage unit 205 that records keywords and positions in the document, a keyword storage unit 205 that totals the keywords recorded in the keyword storage unit 205, and selects keywords that have a particularly high appearance frequency as axis keyword candidates. Aggregation determination unit 207 for determining whether or not a keyword is an axis keyword, an axis keyword, and a document constituting a topic Composed of aggregated document storage unit 208 for storing.

  FIG. 4 is an example of document data recorded in the document storage unit according to the first embodiment of the present invention. In the document storage unit 201 shown in the figure, each document is given an ID (document ID) unique to the document, and is recorded together with time information (time stamp) indicating the creation time of the document.

  The document analysis unit 202 acquires the document ID and the text data from the document data recorded in the document storage unit 201 one document at a time. The acquired text data is divided into morphemes that are grammatical minimum units constituting a document by morphological analysis processing that is an existing technology. At this time, the morpheme analysis result and the text data are collated, and the appearance position of each morpheme in the text data is checked simultaneously. Thereafter, all or part of the morphemes are stored as “general words” in the general word storage unit 203 together with the appearance position and the document ID. This process is performed for each document data recorded in the document storage unit 201.

  The “general word” may be a compound word obtained by connecting consecutive morphemes in a document.

  FIG. 5 shows an example of recording a general word in the first embodiment of the present invention. The general word storage unit 203 shown in FIG. 6 considers a morpheme determined to be a noun as a result of morphological analysis as a general word, and records it together with the appearance position in the document. Note that the appearance position is represented by the number of bytes obtained by counting the positions of the beginning (START) and the end (END) of the general word from the beginning of the document. Since the amount of information for a Japanese double-byte character is 2 bytes, if all double-byte characters are used, if the beginning of the document is 0 bytes, the first character starts at position “0” and the second character starts at “2”. "The third character starts at position" 4 ". In the example of the document with the document ID “0001”, the general word “Yesterday” appears at a position of 0 to 3 bytes, and the general word “movie” appears at a position of 16 to 19 bytes. The same applies to the case where the above compound word is regarded as a general word. Among the morphemes constituting the compound word regarded as a general word, the start position of the morpheme positioned at the head is set as the start position of the general word, and at the end. The end position of the positioned morpheme is set as the end position of the general word.

  In the following description, an example in which only nouns are registered as general words in the general word storage unit 203 as shown in FIG. 5 will be described.

  The keyword extraction unit 204 is a processing unit that extracts a keyword characterizing each document using a morpheme obtained as a result of a morpheme analysis process performed in the document analysis unit 202, its part of speech information, and an appearance position. The extraction process is performed for each document, and simultaneously recorded in the keyword storage unit 205 together with the document ID acquired from the document analysis unit 202.

  The purpose of this process is to acquire candidates for words (axis keywords) that serve as axes when documents that deal with the same topic are collected in subsequent processes. For example, in the case of a document written focusing on a certain athlete, the axis keyword is mainly a proper noun such as the name of the athlete, and in the case of a document written about a merger between companies, is there.

  To extract keywords, we use the existing technology “Hideki Isozaki: Japanese Named Expression Extraction Using Metarules and Decision Tree Learning, Journal of Information Processing Society of Japan Vol.43, No.5 pp.1481-1491 (2002)”.

  This is based on character type information such as hiragana, katakana, and kanji in accordance with the morpheme obtained as a result of the morpheme analysis process, its part-of-speech information, and the appearance position. This is a technology that automatically extracts specific expressions such as “name”, “person name”, “place name”, “artificial name” such as product name and building, “money amount”, and the like.

  In the present embodiment, among these proper expressions, only the “organization name” and “person name” acquired by the above-described existing technology are regarded as “keywords”, and the keyword is displayed together with the appearance position information and the document ID. It is assumed that it is recorded in the storage unit 205. In the subsequent processing, the keyword recorded here is treated as a candidate for the axis keyword.

  Note that the method for acquiring the appearance position of each keyword recorded simultaneously at this time is the same as the method for acquiring the appearance position of a general word in the document analysis unit 202.

  FIG. 6 shows an example of keywords recorded in the keyword storage unit in the first embodiment of the present invention. The keyword is recorded together with the document ID and the appearance position from the top of the document. Note that the appearance position is represented by the number of bytes obtained by counting the position of the beginning (START) and the end (END) of the keyword from the beginning of the document.

  For example, this indicates that the keyword “× Tanioo” extracted as a personal name was positioned 44 to 51 bytes in the document ID “0001”. Similarly, keywords extracted from documents with document IDs “0002” and “0004” are recorded. Assuming that none of the unique expressions of “organization name” and “person name” appear in the diary-like document with the document ID “0003”, no section relating to the document ID “0003” is output as shown in FIG. Not.

  The keyword totaling unit 206 acquires the document ID recorded in the keyword storage unit 205 and the keywords appearing in the document and totals the frequency of each keyword, and includes a high-frequency keyword greater than a certain value and the keyword. A list of document IDs is sent to the aggregation determination unit 207.

  The keyword frequency counting method may count the number of documents in which the keyword appears even once, or may add the total number of times a certain keyword appears in each document as the keyword frequency. For example, since the keyword “○ mountain” in FIG. 6 appears in two documents with document IDs “0002” and “0004”, the frequency of “○ mountain” is “2”.

  In general, a state in which a certain matter is a topic refers to a state in which the same subject is handled in many documents. Based on the frequency of each keyword calculated by the above aggregation method, it is possible to select a keyword that is highly likely to be established as an axis keyword by setting only a high-frequency keyword as a transmission target to the aggregation determination unit 207.

  FIG. 7 is an example of data sent from the keyword totaling unit to the aggregation determining unit according to the first embodiment of the present invention.

  The data sent from the keyword totaling unit 206 to the aggregation determining unit 207 has a frequency of “55 x Tanio” as a keyword with a total frequency of 55, and is in the document with the document ID “0001, 0005, 0032, 0035, 0052,. It was appearing in.

  The aggregation determination unit 207 obtains a keyword list as shown in FIG. 7 from the keyword aggregation unit 206, determines whether each keyword is an axis keyword, and if it is determined that it is an axis keyword, the keyword , And the document ID of the document constituting the topic is recorded in the aggregated document storage unit 208. The determination process is performed for each keyword.

  The aggregation determining unit 207 includes a subset dividing unit 2071, a subset determining unit 2072, and a comprehensive determining unit 2073.

  A flow of performing the determination process for one keyword with the aggregation determination unit 207 will be described.

  FIG. 8 is a flowchart of the process of the aggregation determination unit in the first embodiment of the present invention.

  Step 601) The subset dividing unit 2071 of the aggregation determining unit 207 acquires one keyword and a document ID list including the keyword.

  Step 602) Upon obtaining the one keyword and the document ID list including the keyword, the subset dividing unit 2071 searches the document storage unit 201 based on the document ID to acquire the document group including the keyword, A document group is divided into subsets based on time information. At this time, the document is divided so that the documents having the same time information (similar creation time) are aggregated into the same subset. For example, when the date portion of the time information is used and a subset is formed for each date, a subset of March 11, 20XX, a subset of March 12, 20XX, and so on are formed.

  For example, when focusing on the keyword “○ mountain” extracted as a personal name, the document “0002, 0004, 0021, 0021,...” Including “○ mountain” is divided into daily subsets, and each subset is set. The appearance frequency of general words appearing in a predetermined range around “○ mountain” is counted. In this example, both the documents “0002” and “0004” are divided into the same subset because the time information is both March 12, 20XX.

  Step 603) The subset determination unit 2072 counts the appearance frequency of each general word appearing in a predetermined range around each keyword in the document in each subset.

  Aggregation of general words in the subset is performed as follows.

  In FIG. 6, “○ mountain” appears at the 20th to 23rd byte positions in the document whose ID is “0002”. Therefore, this appearance position information is stored in the general word list 203 and the general word list. Collation is performed, and a general word appearing within a predetermined range around the keyword “○ mountain” in the document “0002” is acquired.

  Various settings can be made for the general word acquisition range for each keyword. FIG. 9 is a first example of a general word collection range in the first embodiment of the present invention.

  FIG. 9A shows a document “0002” including the keyword “○ mountain”. The keyword storage unit 205 stores the appearance position of the keyword “Oyama”, and the general word storage unit 203 stores the appearance position of general words such as “Meeting House” and “Dissolution”.

  For example, Example 1 shown in FIG. 5B is an example where the collection range is a general word of up to two words before and after the keyword “Oyama”. “Dissolution” and “Prime Minister” appearing before the keyword “Mt. Yama”, and “decisions” and “general elections” appearing later are general words to be collected.

  In Example 2 shown in FIG. 5C, the entire document including the keyword is an example of the collection range. In this case, “general house”, “dissolution”, “prime”, “decision”, “general election”, “melancholy” and all general words in the document “0002” are collected.

  In the example of the document “0002” shown in FIG. 11A, there are only three general words before “○ mountain”. For example, in FIG. In this case, only 3 words can be acquired from the front. In this case, all three words before the keyword and five words after the keyword may be acquired.

  In the description so far, only “document ID”, “general word (or keyword)”, and “appearance position” are recorded as in the examples of FIGS. 5 and 6, but morphological analysis performed by the document analysis unit 202 is performed. From the result of the above, it is possible to obtain a punctuation mark “.” In the document, determine a sentence break, and simultaneously record which word and which word are extracted from the same sentence. If this is used, as shown in Example 3 in FIG. 9D, it is possible to set the general word collection range for several sentences around the sentence including the keyword “○ mountain”. Example 3 in FIG. 9D is an example in which only sentences including keywords are used as the collection range, so that only four words of “lower house”, “dissolution”, “prime”, and “decision” are acquired. The range may be expanded to the preceding and following sentences, such as “two sentences before and after the sentence including the center”.

  In addition to the punctuation mark “.”, A period “.”, An exclamation mark “!”, A question mark “?”, Etc. may be regarded as sentence delimiters.

  On the other hand, FIG. 10 shows another example of a range designation method when a general word in the vicinity is acquired by paying attention to the keyword “XX newspaper” acquired as the organization name. According to the document aggregation method of the present invention, when a specific general word appears with high probability around a keyword of interest, the document group is a document describing the same topic, and the keyword is an axis keyword. It is the method of judging. Assuming that the character string “(XX newspaper issue)” in the sentence shown in FIG. 10 is a character string regularly used for the headings of all news articles issued by XX newspaper publishers, In the vicinity of “”, a general word “issue” appears with a very high probability. For this reason, all news articles issued by XX newspaper companies are mistakenly aggregated as being the same topic. The range designation example shown in FIG. 10 is one method for avoiding this. By not acquiring a general word that appears adjacent to the keyword of interest, it is possible to prevent the same issuer from mistakenly determining that this is an axis that is frequently used.

  Instead of the adjacent general word, the general word closest to the keyword of interest may be excluded from the acquisition target. In the case of FIG. 10, “inundation” and “issue” are excluded from the target.

  Not only the range setting methods shown in FIGS. 9 and 10, but various general word acquisition methods can be used. For example, when a general word is not included in a range specified by a fixed length, such as one sentence before and after, the search range is expanded, and general words of M words before and after are acquired in order from the keyword of interest, etc. You may acquire in combination with a plurality of methods.

  For any document in the subset, a general word is acquired from a predetermined range around the keyword of interest by any one of the above methods. For example, for the keyword “○ mountain”, “the House of Representatives” The number of documents from which general words are acquired is totaled, such as “dissolution” seven times.

  Step 604) When the aggregation of the documents in the subset is completed, the subset determination unit 2072 determines whether or not the documents in the subset are described for the same topic, that is, the keyword of interest “○ mountain” is the axis keyword. It is determined whether or not there is.

  Whether or not the keyword is an axis keyword is determined based on whether or not a specific word frequently appears within a predetermined range around the keyword. FIG. 11 shows an example of the top five words having a large number of appearing documents as a result of aggregating peripheral general words in each case of focusing on the keyword “Yama” and focusing on “XX newspaper”. As for the subset of 28 documents including “○ mountain”, the general words acquired from the periphery are “House of Representatives” (appearing in 25 documents), “Dissolving” (appearing in 22 documents) in descending order of frequency, It is assumed that “Prime” (appears in 21 documents), “Tomorrow” (appears in 15 documents), and “General Election” (appears in 13 documents).

  At this time, the subset determination unit 2072 counts the number of general words exceeding 14 that is half of the number of documents constituting the subset, out of the number of appearance documents of the general words up to the top five. In this example, the general words having the number of appearing documents exceeding 14 are four words of “the House of Representatives”, “Dissolution”, “Prime Minister” and “Tomorrow”. In other words, four words out of the top five general words acquired from the periphery appear at a frequency higher than a certain level.

  Similarly, in the example of “XX newspaper” shown in FIG. 11, there is no single general word that appears with a high frequency exceeding the version number 16 of the constituent document number 32 of the subset.

  The subset determination unit 2072 includes the number of general words having a high frequency when satisfying the condition among the upper words (for example, upper five words) of the general words arranged in the order of high frequency (for example, four words in the case of “○ mountain”, In the example of “XX newspaper”, 0 word) is examined, and when the ratio is 60% or more, for example, it is determined that the documents in the subset are described on the same topic. Of the top five words, “Yama” corresponds to 80% of the four words that satisfy the condition and exceeds 60%, so it is determined that the topic is the same topic. Since there is no word (0%), it is determined that the topics are not the same.

  When a group of documents containing a certain keyword is described with respect to the same subject matter, the appearance patterns of words around the keyword are similar between documents. Become. In other words, words appearing around the keyword are biased. On the other hand, a generality of a certain keyword is high, and in the case of a word used in documents in various fields, words appearing in the vicinity vary, and therefore it is difficult to increase the frequency even if the number of occurrences of each keyword is counted. In other words, the words appearing around the keyword are not biased. The above determination uses this feature, and it is automatically determined whether or not it is an axis keyword by examining the proportion of general words that frequently appear around the keyword of interest, occupying a particularly frequent word. Judgment.

  If it is the determination method which paid its attention to this characteristic, it is not necessary to be limited to said example, and various applications are possible.

  For example, it is possible to construct a database in which the possibility that all two words are used at the same time is constructed in advance, and determine whether or not high-frequency general words acquired from the periphery are related to each other. In this case, in the example of FIG. 11, general words such as “the House of Representatives”, “dissolution”, and “Prime Minister” obtained from the vicinity of the keyword “Yama” are words closely related to each other by referring to the database. It can be used as a judgment material that “○ mountain” is the topic axis. Conversely, “Arrest”, “Seismic Intensity”, and “Home Run” obtained from the vicinity of “XX Newspaper” are not related, and “XX Newspaper” is a general word, so it is not an axis keyword. It can be determined.

  Alternatively, a database that quantifies the possibility of any two words or phrases appearing together in one sentence or one document may not be built in advance, but dynamically constructed from the input document. In this case, calculate the probability that high-frequency general words such as “Senior House” and “Dissolution” obtained from the keyword “○ Mountain” will appear together in a certain range, such as in the same sentence or in the same document. If the probability is high, it can be determined that the topic is the same.

  Step 605) When the determination of each subset of the document including the keyword of interest “○ mountain” is completed, the process proceeds to Step 606, and if there is a subset that has not been determined yet, the process proceeds to Step 603.

  Step 606) The comprehensive determination unit 2073 finally uses the determination results of the subsets generated by dividing the document group including “○ mountain” to determine whether or not the attention keyword “○ mountain” is an axis keyword. Make a good judgment.

  Comprehensive determination is performed based on whether it is determined in step 604 that several subsets of the subset of documents including the keyword of interest “○ mountain” are the same topic. Assuming that there are 10 document subsets including the keyword of interest “Oyama” and 7 are determined to be the same topic in step 604, the ratio determined to be the same topic is 70%. At this time, for example, when this ratio exceeds 50%, the overall determination unit 2073 says that all documents including the keyword of interest “Oyama” are composed of the same topic, and “Oyama” is an axis keyword. Make a final decision.

  The purpose of determining whether or not the topics are the same in units of subsets and integrating the determination results to make a final determination is to acquire topics that gradually change with time. As the degree of attention of a topic centered on a certain keyword increases, there may be a transition to another topic over time. In this case, since the general words appearing around the keyword also change gradually with time, if the same determination as in step 604 is performed without dividing into subsets, it is not determined that the word is the topic axis. . By dividing the document based on the time information and performing determination processing in units of subsets, it is possible to perform aggregation corresponding to a changing topic.

  As a result of the comprehensive determination, if it is determined that the keyword of interest is an axis keyword, the process proceeds to step 607, and if not, the process ends.

  Step 607) The keyword and the ID of the document including the keyword are recorded in the aggregated document storage unit 208, and the process proceeds to the next keyword determination process.

[Second Embodiment]
The configuration of the time-series document aggregation device in the present embodiment is the same as the configuration in the first embodiment described above shown in FIG.

  Also, the processing flow is the same as in the first embodiment. Extraction of general words by the document analysis unit 202, extraction of keywords by the keyword extraction unit 204, aggregation of keyword appearances by the keyword aggregation unit 206, and selection of high-frequency keywords , The aggregation determination unit 207 determines whether or not the topic is the same and outputs the document.

  In the present embodiment, only the processing of the aggregation determination unit 207 is different from that of the first embodiment.

  The flow of processing performed for each keyword in the aggregation determination unit 207 of this embodiment will be described below.

  In the first embodiment, the determination is made in units of subsets, and then the determination is made in two stages in which the final determination is performed by integrating them together. In the present embodiment, only time information is used. Instead, after performing the first determination for each subset divided based on the relevance between documents, it is determined (second determination) whether or not it is the same topic at a certain period using the result, Finally, they are integrated to make a final determination, and a three-stage determination is performed.

  FIG. 12 is a flowchart of the operation of the aggregation determination unit according to the second embodiment of the present invention.

  Step 1001) The subset dividing unit 2071 in the aggregation determining unit 207 acquires one keyword and a document ID list including the keyword.

  Step 1002) The subset dividing unit 2071 counts general words appearing within a predetermined range around the keyword in the document including the keyword, and generates a vector of each document. The vector generation method will be described later.

  Step 1003) The subset dividing unit 2071 searches the document storage unit 201 based on the document ID including the keyword, and for each searched document, the time information recorded in the document storage unit 201 and Then, it is divided into subsets based on the similarity between documents calculated from the document vectors described below. At this time, the document is divided so that documents having close time information and high similarity belong to the same subset. For the subset division, for example, a clustering technique in which the distance between documents is defined by the following formula d (m, n) is used.

d (m, n) = (1-S (m, n)) × exp (− | t (m) −t (n) | / T)
S (m, n) is the similarity of the contents of documents m and n. exp (− | t (m) −t (n) | / T) is the time distance between these two documents, and the closer to the time information t (m) and t (n) of each document, the closer to 1 Takes a value. The time distance takes a maximum value of 1 when the time information of the two documents is the same. T is a parameter that determines the degree of attenuation of the time distance, and may be fixedly set according to the speed of the temporal transition of the topic included in the document data to be handled.

  By performing clustering using the above document distance, documents that are close in time and constitute the same topic are collected in the same cluster.

  Hereinafter, a method for generating vectors and a method for calculating similarity between documents will be described.

  The similarity S (m, n) between documents is determined by comparing the appearance frequencies of general words that appear within a predetermined range around the keyword of interest. The general word counting method is the same as the counting method in the first embodiment. The document vector is defined by the appearance frequency of the general word in the document, and for example, the cosine distance may be set as the similarity S (m, n) between the documents m and n.

  Step 1004) Subsequently, the subset dividing unit 2071 reflects whether or not the topic is the same in the subset unit divided based on the similarity and the time distance.

  Step 1005) The processing of Step 1004 is performed until the determination of all the subsets is completed, and when all the subsets are checked, the process proceeds to Step 1006.

  Step 1006) The determination results for each subset are collected and determined every certain period such as one day. For only the parts determined to be the same topic in step 1004, the time information of the centroid is calculated by averaging the time information of the documents in the subset, and the subset is classified by day according to the centroid time. The subsets determined not to be the same topic in step 1004 are rejected here. The purpose of rejection is to eliminate clustering errors. When clustering is performed, a minute cluster may be generated or an inaccurate cluster (a similarity between documents is not so high) may be generated. In order to prevent these effects, only a subset determined to be the same topic is handled.

  If a subset determined to be the same topic is included in, for example, three or more within a certain period such as one day, three topics existed simultaneously during this period, that is, the same topic for this period. It is determined that it is not.

  For example, when attention is paid to a certain keyword “XX newspaper”, it is assumed that the number of subsets classified on a certain day is four. However, as shown in FIG. 10, when “XX newspaper” is a general term included in various documents and is not suitable as an axis keyword, clustering results in “XX newspaper company”. The documents are divided according to the difference in content, such as “cluster consisting of documents of baseball game results issued by” and “cluster consisting of documents of falling stock prices issued by XX newspaper”. Even in such a case, if each cluster is composed of the same topic, a keyword such as “XX newspaper” in this example is finally determined to be the topic axis. . In order to prevent this, an aggregation determination is performed at regular intervals such as daily, for example, an intermediate determination process (second determination) as to whether or not “XX newspaper” is an axis keyword in a period of one day. )I do. Assuming that “XX newspaper” is a keyword that is commonly included in documents of various topics, a plurality of topics are present at the same time as a result of aggregation every day, and are rejected.

  Step 1007) Finally, all the determination results for a certain period are integrated, and it is determined whether the word is suitable for being called the topic axis of the keyword of interest over the entire period.

  As a result of the determination, if it is determined that the keyword of interest is an axis keyword, the process proceeds to step 1008. If not, output is not performed and the process ends.

  Step 1008) The keyword and the ID of the document including the keyword are recorded in the aggregated document storage unit 208.

  As in the final determination in the first embodiment, the final determination process totals each determination result for each day, and exceeds 50%, for example, depending on the ratio of the number of days determined as the same topic to the whole. If the same topic is determined.

  In the first and second embodiments described above, when outputting the IDs of the topic constituent documents, the IDs of all the documents including the axis keyword are not output, but are the same topic in the final determination step. Only a subset of documents determined to be output may be output.

  Alternatively, the IDs of only documents including, for example, three or more words among the top five general words acquired from the periphery of the keyword may be output. In the example of the keyword “Yama” shown in FIG. 11, only documents that include three or more words among “the House of Representatives”, “Dissolution”, “Prime Minister”, “Tomorrow”, and “General Election” are the topics centered on “Yama”. By outputting as a group of documents, it is possible to carefully select only documents that describe a specific topic such as “the general election accompanying the dissolution of the House of Representatives” without accidentally mixing documents that contain the keyword “○ mountain”. it can.

  Of course, only documents that are subset documents determined to be the same topic by a combination of the above output methods and that include three or more words in the top five general words may be output.

  As a result of the above processing, it is not a highly general keyword used for documents in all fields like “XX newspaper” in the above embodiment, but, for example, “Yama”, which is the prime minister ’s last name. It is determined that the keyword positioned at the center of the topic is an axis keyword, and at the same time, a document group constituting the topic is output. Further, even when the topic changes with time, if the keyword is suitable as the topic axis, it is determined appropriately.

  In the above-described embodiment, a word called a proper expression is regarded as a keyword, and a noun is regarded as a general word. However, the present invention is not limited to this, and various changes can be made.

  As described above, the present invention generates a subset obtained by dividing a document including a certain keyword, and determines the frequency of words (general keywords) appearing within a predetermined range around the keyword in the document in each subset. It is determined whether or not the documents in the subset are described with respect to the same topic by counting, and the entire document including the keyword is described with respect to the same topic using the determination result in each subset. It is a method of finally determining whether or not the object is a thing. For example, if you pay attention to the prime minister's surname, “Yama”, if there are general words such as “the House of Representatives”, “General Election”, and “Line” appear in the vicinity, those documents will be Judged as a group of documents on the dissolution of the House of Representatives due to the collapse of the administration.

  By this method, documents describing the same topic are aggregated, and a central keyword (axis keyword) representing the aggregated document group is simultaneously acquired.

  Note that the determination of whether or not the keyword is an axis keyword and the aggregation of topical configuration documents with the keyword as an axis are performed simultaneously. In other words, if it is determined that a document set obtained when attention is paid to a keyword, the documents in the document set describe the same topic, the keyword is simultaneously determined as the topic axis. It will be.

  3 is constructed as a program for causing a computer functioning as a time-series document aggregation device to execute the operations of the document analysis unit, the keyword extraction unit, the keyword aggregation unit, and the aggregation determination unit shown in FIG. 3 and is installed in the computer. Or can be distributed via a network.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk device, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to a technique for acquiring a document that handles the same subject from a large number of documents on a network or storage means and a central keyword of a document group.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a block diagram of the time series document aggregation apparatus in the 1st Embodiment of this invention. It is an example of the document data recorded on the document storage part in the 1st Embodiment of this invention. It is an example of the general word recorded on the general word storage part in the 1st Embodiment of this invention. It is an example of the keyword recorded on the keyword storage part in the 1st Embodiment of this invention. It is an example of the keyword list which the keyword total part in the 1st Embodiment of this invention sends out to an aggregation determination part. It is a flowchart of a process of the aggregation determination part in the 1st Embodiment of this invention. It is a 1st example of the general word collection range in the 1st Embodiment of this invention. It is a 2nd example of the general word collection | recovery range in the 1st Embodiment of this invention. It is an example of the general word acquired from the keyword periphery in the 1st Embodiment of this invention. It is a flowchart of operation | movement of the aggregation determination part in the 2nd Embodiment of this invention.

Explanation of symbols

201 Document storage unit, document storage unit 202 Document analysis unit, document analysis unit 203 General word storage unit, general word storage unit 204 Keyword extraction unit, keyword extraction unit 205 Keyword storage unit, keyword storage unit 206 Keyword totaling unit, keyword totaling unit 207 Aggregation determination unit 2071 Subset generation unit, subset division unit 2072 Subset determination unit, subset determination unit 2073 General determination unit, general determination unit 208 Integrated document storage unit, integrated document storage unit

Claims (12)

In a device having document analysis means, keyword extraction means, keyword aggregation means, subset generation means, subset determination means, and overall determination means, among many documents to which time information is given, A time-series document aggregation method that aggregates documents that describe the same topic and also acquires an axis keyword that is a central keyword representing the aggregated document group,
The document analysis unit analyzes the input document, acquires a general word that is a word in the document, and appearance position information of the general word, and stores the general word in the general word storage unit; and
A keyword extracting step in which the keyword extracting means acquires a keyword that is a candidate for the axis keyword and the appearance position information of the keyword in the input document, and stores the keyword in the keyword storing means;
A keyword selection step in which the keyword aggregation means totalizes the keywords stored in the keyword storage means to select high-frequency keywords;
The subset in which the subset generation unit acquires a document including the keyword selected in the keyword selection step from the document storage unit, and divides each document into subsets based on the degree of time information Generation step;
The subset determination means obtains, from the general word storage means, general words that appear within a predetermined range around the keyword in the documents in the subset, and based on the result of aggregation, A subset determination step for determining whether or not documents in the subset are described on the same topic;
Comprehensive determination means integrates the determination results for each subset, and finally determines whether or not the keyword is a topic axis; and
A time-series document aggregation method, characterized by: In the subset generation step,
The time-series document aggregation method according to claim 1, wherein a subset is generated by aggregating documents whose time information is close to each other. In the subset generation step,
The time-series document aggregation method according to claim 1, wherein a subset is generated by aggregating documents having similar time information and similar contents. In the subset determining step,
General words appearing within a predetermined range around the selected keyword are aggregated for each subset, and based on the number of general words appearing frequently at a certain rate or higher, documents in the subset are displayed. The time-series document aggregation method according to claim 1, wherein it is determined whether or not the same topic is described. In the subset determining step,
The number of documents X included in the subset is totaled, and general words that appear at a high frequency Y within a predetermined range around the keyword in the X documents are obtained, and the ratio of Y to X is obtained. The time-series document aggregation method according to claim 4, wherein when the number of high general words is greater than a certain value, it is determined that the documents in the subset are described for the same topic. Aggregate documents that describe the same topic among a large number of documents with time information, and collect time-series documents that simultaneously acquire axis keywords that are the central keywords that represent the aggregated documents. A device,
Document storage means for storing a large number of documents to which the time information is attached;
Analyzing the input document read from the document storage means, obtaining a general word that is a word in the document and appearance position information of the general word, and storing the general word storage means in the general word storage means,
In the input document, a keyword extraction unit that acquires the keyword that is a candidate for the axis keyword and the appearance position information of the keyword, and stores the keyword in the keyword storage unit;
A keyword counting means for counting the keywords stored in the keyword storage means and selecting high-frequency keywords;
Subset generation means for acquiring a document containing keywords selected by the keyword aggregation means from the document storage means, and dividing each of the documents into subsets based on the time information;
Of the documents in the subset, the general words appearing within a predetermined range around the keyword are acquired from the general word storage means, and the documents in the subset are calculated based on the totaled results. Are subset determination means for determining whether or not the same topic is described,
Integrated determination means for integrating the determination results for each subset, and finally determining whether or not the keyword is a topic axis;
A time-series document aggregating apparatus. The subset generation means includes
The time-series document aggregation device according to claim 6, further comprising means for aggregating documents whose time information of the documents are close to each other to generate a subset. The subset generation means includes
The time-series document aggregation device according to claim 6, further comprising means for aggregating documents having similar time information and similar contents to generate a subset. The subset determination means includes
General words appearing within a predetermined range around the selected keyword are aggregated for each subset, and based on the number of general words appearing frequently at a certain rate or higher, documents in the subset are displayed. The time-series document aggregation device according to claim 6, further comprising means for determining whether or not the same topic is described. The subset determination means includes
The number of documents X included in the subset is totaled, and general words that appear at a high frequency Y within a predetermined range around the keyword in the X documents are obtained, and the ratio of Y to X is obtained. The time-series document aggregation device according to claim 9, further comprising means for determining that the documents in the subset are described with respect to the same topic when the number of high general words is greater than a certain value.   A time-series document aggregation program for causing a computer to function as the time-series document aggregation apparatus having the means according to any one of claims 6 to 10.   A storage medium storing a time-series document aggregation program, wherein a program for causing a computer to function is stored as the time-series document aggregation device having the means according to any one of claims 6 to 10.

Images