JP3687118B2 - Related word dictionary creation device and related word dictionary creation method - Google Patents

Description

[0001]
[Industrial application fields]
The present invention relates to thesaurus creation maintenance of a text search apparatus, and more particularly to an apparatus that supports construction and maintenance of a thesaurus based on the co-occurrence relationship of keywords in a document registered in the text search apparatus.
[0002]
[Prior art]
One means for searching for a document without omission is to use a thesaurus dictionary to expand keywords and use them as search keys. The thesaurus dictionary registers words and related words, and is an effective means for searching. However, there is a problem that its construction is complicated, and conventionally, a device that supports the update operation of a personal thesaurus or a device that automatically creates a dynamic thesaurus has been proposed as follows, for example. Has been made.
[0003]
(1) Japanese Patent Laid-Open No. 4-39769 discloses a “thesaurus generation device” that performs morphological analysis and sentence structure analysis on an input document, determines a semantic relationship between words using a rule dictionary, and obtains a word space between the obtained words. Proposals have been made on automatic thesaurus generation techniques that automatically group related words into a tree structure.
[0004]
(2) Japanese Patent Laid-Open Publication No. Hei 4-64171 “Keyword Associative Generating Device” calculates a node weight on a thesaurus from a sample of a search target sentence, and assigns the node weight of a dynamic thesaurus created to the thesaurus. It has been proposed to generate associative keywords using.
[0005]
(3) Japanese Patent Application Laid-Open No. 4-123264 “Related Word Table Creation Device and Document Search Device” creates a related word table by performing a semantic expression table from a natural language sentence and then performing sentence analysis / semantic analysis and the like. Technical proposals have been made.
[0006]
(4) In Japanese Patent Application Laid-Open No. 4-222555 “Personal Thesaurus Creation Support Device”, a thesaurus or character string or word that failed to be analyzed or verified using a thesaurus at the time of search is registered as a thesaurus candidate. Proposals have been made regarding techniques for providing organized materials for determining whether to use a thesaurus or not, and for supporting thesaurus creation.
[0007]
[Problems to be solved by the invention]
However, as in the prior art (1) (Japanese Patent Laid-Open No. 4-39769) and the prior art (3) (Japanese Patent Laid-Open No. 4-123264), the sentence structure or semantic relationship is analyzed, The method of calculating the degree of association, the upper / lower relationship, or the antonym is disadvantageous in that it takes a long time when registering a large number of documents because the processing becomes heavy when it is performed at the time of document registration. As for the related art (1), related word groups are limited to construct a thesaurus with a tree structure, and the degree of freedom of the structure is also limited. Focusing on the point of creating a dynamic thesaurus for generating an associative keyword in the prior art (2) (Japanese Patent Laid-Open No. Hei 4-64171), the extracted document is not the entire search target sentence, but a part of it. However, there is a drawback that a word necessary as a thesaurus candidate may be leaked. And the method of feeding back the results at the time of retrieval as in the prior art (4) (Japanese Patent Laid-Open No. Hei 4-222555) also reflects only the one given as a retrieval formula in the thesaurus creation or should be registered as a thesaurus Even if a word can be extracted, there is a drawback that the relationship with other words in the same word group is not known.
[0008]
An object of the present invention is to make it possible to relatively easily extract and determine a keyword and its related word from data of a search target document and to automatically register it in a related word dictionary.
An object of the present invention is to make it possible to efficiently create a related word dictionary of the above keywords with performance sufficient to withstand actual use by preventing the processing from becoming heavy.
It is another object of the present invention to eliminate irrelevant keywords that are not used at all, and to select and register terms of high interest of the user as related terms.
[0009]
[Means for Solving the Problems]
The present invention provides a keyword extracting means (2) for extracting a keyword of a document to be searched, and a keyword group having the same document unit as the source of extraction from the extracted keyword group as a co-occurrence keyword, pairs keyword with determined Mel from keyword group in its co-occurrence relation, co-occurrence file means that they seek the frequency of occurrence of each keyword pair, held in the co-occurrence file (7,8), the co-occurrence Referring to file, the frequency of occurrence keyword pairs beyond the set threshold value, and while the a headword, the other as its related terms, related terms dictionary registration means for registering the related word dictionary (9) Is a related word dictionary creation device. The co-occurrence file means also holds the appearance frequency of each keyword in addition to the appearance frequency of the keyword pair, and the related word dictionary registration means refers to the co-occurrence file means to generate the occurrence of the keyword pair. When the frequency exceeds a set threshold and the appearance frequency of one keyword to be used as a headword in the keyword pair exceeds the other set threshold, it is registered in the related word dictionary.
[0010]
In one aspect of the present invention, the related word dictionary registration unit has a function of editing a related word dictionary by a user, and a function of changing a threshold value of the appearance frequency of the keyword pair or a threshold value of the appearance frequency of the keyword, And a display means for displaying related word candidates related to an arbitrary keyword from the co-occurrence file means in accordance with a user designation.
[0011]
In another aspect of the present invention, the related word dictionary registration means has a function of editing a related word dictionary by a user and a function of changing a threshold value of the appearance frequency of the keyword pair or a threshold value of the appearance frequency of the keyword. And a means (12) for displaying related word candidates relating to an arbitrary keyword from the co-occurrence file means in accordance with a user's designation.
[0012]
[Action]
The co-occurrence file means obtains a keyword pair having a co-occurrence relationship from the keyword group extracted by the keyword extracting means. It is assumed that keyword groups extracted from the same document have a co-occurrence relationship. Alternatively, when the origin of the keyword group can be determined by the unit of the component of the document as in the structured document, the keyword group extracted from the unit of the component is assumed to have a co-occurrence relationship. Thus, since the co-occurrence relationship is determined based on whether or not the document units that are the origins of extraction are the same, the determination process is simplified. Then, the frequency of appearing in the document is recorded for each keyword pair having a co-occurrence relationship. The related word dictionary registration means creates a related word dictionary with reference to the created co-occurrence file. At that time, the appearance frequency of the keyword pair in the co-occurrence file represents the strength of the relationship between the keywords of the keyword pair, so when the appearance frequency is larger than the threshold set appropriately, these are related words. Assuming that there is one, one of them is used as a headword, and the other is registered as a related word in the related word dictionary.
[0013]
The more frequently a keyword appears in a search target sentence, the more likely it is to be used as a search keyword by a user. Therefore, by calculating the frequency of appearance of each keyword alone and adding this as a criterion for determining related words in addition to the frequency of occurrence of the keyword pairs described above, keywords that are rarely used are related words. It is no longer registered as a dictionary entry word, and an efficient related word dictionary can be obtained.
[0014]
As described above, the related word dictionary registering unit has a function for editing the related word dictionary by the user and a function for allowing the user to change the setting of each threshold value, and from the co-occurrence file unit to the related word dictionary. Display means for displaying the candidates can be provided. In this case, the user can confirm the change in the range of the related word candidates to be displayed on the display means while adjusting the setting of each threshold, and determine which range is to be registered in the related word dictionary. It can be easily determined.
[0015]
【Example】
FIG. 1A to FIG. 1C are functional block diagrams showing a schematic configuration of an apparatus according to an embodiment of the present invention. FIG. 1A shows a co-occurrence file that creates an index file by extracting keywords when a document is registered, and maintains a co-occurrence relationship of keywords based on the keywords registered in the index file. The outline of the configuration necessary for creating and registering is shown. The apparatus includes an input unit 1 for giving an instruction for document registration, a keyword extraction unit 2 for extracting a keyword from a document registered in the text beta base unit 3, a text database unit 3 for holding text data of the document, An index registration unit 4 that registers an index in which an extracted keyword is associated with an identifier of a registered document in an index file, an index file unit 5 that holds the index, and another keyword group in the same text as the extracted keyword are shared. Co-occurrence keyword extraction means 6 that extracts words that are in the relationship of occurrence, co-occurrence file registration means 7 that registers the keyword co-occurrence and frequency in the co-occurrence file, and holds the keyword co-occurrence and frequency The co-occurrence file unit 8 is provided.
[0016]
FIG. 1B shows an outline of a configuration for creating an editable dictionary file of related words (thesaurus) from the co-occurrence file. The related word (thesaurus) editable dictionary file construction means 9 and related words An editable dictionary file unit 10 is provided.
[0017]
FIG. 1C shows an outline of components for displaying related word candidates when the user constructs a related word dictionary (thesaurus dictionary). Related word candidates are extracted from the co-occurrence file unit 8 and transferred to the display unit 12 in a displayable format. The display unit 12 displays the related word candidates.
[0018]
The operation of the embodiment configured as described above will be described.
First, in this embodiment, the setting criteria for synonyms (related words) are set as follows.
[0019]
For a system that extracts keywords by performing morphological analysis each time a search target document is registered in the text database,
(1) For a character string extracted by a keyword by morphological analysis, other character strings extracted from a registered document in which the character string exists are used as synonyms / related words as well as a unique ID representing the related word. Keep it in a co-occurrence file for construction.
[0020]
(2) In order to obtain the appearance frequency at the same time, the appearance frequency is incremented by one when an already extracted character string exists in the co-occurrence file or the index file.
[0021]
(3) If another character string already exists as a synonym / related word candidate of the extracted keyword of the co-occurrence file, the number of matches is incremented, and if it does not exist, the number of matches is set to 1.
[0022]
(4) Repeat the above operation every time there is a registered document.
[0023]
(5) In the co-occurrence file in which the synonym / related word candidate character strings are dynamically updated in this way, the frequency of appearance of the character strings extracted by keywords is greater than or equal to a threshold (for example, 10 locations), When the number of matches is greater than a certain threshold (for example, 5) in the same file, it is regarded as a synonym or related word and is automatically registered in the related word file as it is or presented to the user and registered in accordance with the user's instruction.
[0024]
(6) Default values are given for the appearance frequency and the threshold value of the number of matches, and the user can change them. FIG. 4 shows an example of the contents of the threshold values for the appearance frequency and the number of matches.
[0025]
The appearance frequency is the sum of the number of appearances for all registered document files for a certain keyword.
Further, the index has a structure having a keyword (character string or ID) and a document file ID entry as shown in FIG. 2A, and a document file ID and a keyword (character as shown in FIG. 2B). Some of them have a column or ID) entry.
Then, as shown in FIGS. 4A and 4B, the entry structure of the co-occurrence file for related word construction is as follows: user-defined related word ID, keyword, appearance frequency, corresponding keyword pair ([corresponding keyword, number of matches] [,], [,], ...). FIGS. 4 (a) and 4 (b) also show an example of the contents of a co-occurrence file entry for constructing a related word.
[0026]
FIG. 6 is a flowchart of processing for creating and registering a co-occurrence file that holds the relationship between the extracted keyword and the synonym / related word. First, a document registration instruction is given by the input means 1 (S60). Here, index registration is performed (S61), and the processing at this time is shown in the flow of FIG.
[0027]
In FIG. 7, keywords are extracted by the keyword extraction means 2 using morphological analysis (S 70), and the extracted keywords are converted into keyword-to-documents as shown in FIG. 2A in the index file unit 5 by the index registration means 4. It is registered in the index of file ID (S71), and is registered in the index of document file ID versus keyword as shown in FIG. 2B (S72). Next, it is checked whether or not the extracted keyword exists as a headword in the co-occurrence file portion 8 for the related word (S73). If it does not exist, the co-occurrence file registration means 7 sets the user-defined related word ID and extracts the extracted keyword. Is registered as a headword, and the appearance frequency is set to 1 (S74). If it already exists, the appearance frequency of the record of the corresponding co-occurrence file unit 8 is incremented (S75). The above operation is repeated until there are no keywords to be extracted (S76). When this is finished, the process proceeds to step S62 in FIG.
[0028]
The co-occurrence keyword extracting unit 6 extracts one keyword related to the file ID of the target document instructed for registration as a keyword of the target registered document from the document file ID versus keyword index (S62). Then, keywords related to the target registration document other than the extracted keywords are extracted as corresponding keywords having a co-occurrence relationship (S63). Next, the co-occurrence file registration unit 7 refers to the co-occurrence file unit 8 and checks whether the corresponding keyword exists in the entry of the co-occurrence file unit 8 in which the keyword of the target registration document is a headword. (S64). If it does not exist, the co-occurrence file registration means 7 adds a record of [corresponding keyword, number of matches = 1] of the entry of the corresponding co-occurrence file portion 8 (S65). If it already exists, the number of matches in the [corresponding keyword, number of matches] record of the corresponding co-occurrence file unit 8 is incremented (S66). It is checked whether or not the remaining corresponding keyword exists in the document file ID vs. keyword index (S67). If it still exists, a keyword other than the extracted keyword is extracted as the corresponding keyword (S63). Return. If it does not exist, it is determined whether or not the keyword extraction is completed (S68). If not, the process returns to the process of extracting the keyword from the index (S62). When all keywords are extracted from the registered document, it is determined whether the document registration process is completed (S69). If there is a document to be registered, the process returns to the document registration process (S60) again. When there are no more documents to register, the process ends.
[0029]
FIG. 8 is a flowchart of processing for creating an editable dictionary file of related words. First, in the configuration shown in FIG. 1B, the input unit 1 instructs the construction of an editable dictionary file of related words (S80). The related word editable dictionary file construction means 9 confirms with the user whether the threshold value of the appearance frequency may be a default value (S81), and if it is desired to select a value other than the default value, it is changed to the threshold value of the appearance frequency specified by the user. (S82) If the default value is acceptable, the threshold is held as it is. Next, the related word editable dictionary file construction means 9 confirms with the user whether the threshold value of the number of matches may be a default value (S83), and if it is desired to select a value other than the default value, the appearance frequency specified by the user is determined. The threshold value is changed (S84), and if the default value is acceptable, the threshold value is held as it is. The related word editable dictionary file construction means 9 extracts an entry of a co-occurrence file satisfying the condition that is equal to or greater than the set appearance frequency and the threshold value of the number of matches (S85), and the related word editable dictionary is extracted from the extracted entry. A file is created (S86).
[0030]
FIG. 9 is a flowchart of processing for displaying related word candidates. First, in the configuration of FIG. 1 (c), the input means 1 instructs the related word candidate display (S90). The related word candidate display means 11 confirms with the user whether the threshold value of the appearance frequency may be a default value (S91), and if it is desired to select a value other than the default value, it changes to the threshold value of the appearance frequency specified by the user (S92). ) If the default value is acceptable, the threshold value is held as it is. Next, the related word candidate display means 11 confirms with the user whether the threshold value for the number of matches may be a default value (S93), and if it is desired to select a value other than the default value, it changes to the appearance frequency threshold value specified by the user. (S94) If the default value is acceptable, the threshold value is held as it is. The related word candidate display means 11 extracts a co-occurrence file entry that satisfies the condition that is equal to or greater than the set appearance frequency and the matching frequency threshold (S95), and displays related word candidates from the extracted entry (S96). ).
[0031]
Hereinafter, a specific operation example of the present embodiment will be described with reference to FIGS. 2A to 6 as appropriate.
[0032]
The keyword-to-document fileID index in FIG. 2A, the document fileID-to-keyword index in FIG. 2B, and the co-occurrence file for related word construction in FIG. 3A already exist. It is assumed that a document including “Panna Cotta”, “Nata de Coco”, and “Tiramisu” is registered. In step S61 of FIG. 6, the index registration unit 4 records “Pannacotta”, “Natadecoko”, “Tiramisu”, and the fileID 20222 of the document to be registered in the keyword-to-document fileID index as shown in FIG. to add. If “Panna Cotta”, “Nata de Coco”, and “Tiramisu” have already been registered, only the file ID 20222 of the corresponding document is added to the field of the document file ID. Next, the index registration means 4 adds a file ID 20222 record of the document to be registered as shown in FIG. 3B to the document file ID vs. keyword index, and adds “Panna Cotta”, “Nata deco”, “Tiramisu” to the keyword field, and records Update. As shown in FIG. 4A, “Panna Cotta” has already been registered with an appearance frequency of 9 times, and “Nata Deco” has already been registered with an appearance frequency of 4 times. Since “tiramisu” is not yet registered, user-defined related word ID N3 is set for “tiramisu” as shown in FIG. 4B, and “tiramisu” is registered as a headword, and the appearance frequency is set to 1. Set. Also, the user-defined related word ID N1 is already registered for “Panna Cotta”, and the user-defined related word ID N2 is already registered for “Natacococo”, and the appearance frequency is incremented by 1 to 10 and 5, respectively.
[0033]
Next, in S62 to S67 of FIG. 6, the keywords “Pannacotta”, “Natadecoko”, and “Tiramisu” in the fileID 20222 are sequentially extracted from the document fileID versus keyword index of FIG. The co-occurrence file registration means 7 checks whether the extracted keyword is an entry word in the co-occurrence file portion 8 for related words, and “Panna Cotta”, “Nata de Coco”, and “Tiramisu” exist according to the above settings. The remaining keywords existing in the file ID 20222 are registered together with the number of matches. In the case of “Panna Cotta”, the remaining keywords are “Nata de Coco” and “Tiramisu”, and “Nata de Coco” and “Tiramisu” exist in the corresponding keyword pair fields with the appearance frequencies = 4 and 1, respectively. ), Each pair is incremented by 1 (Natadecoko, 5) and updated to (Tiramisu, 2). In the case of “Natadecoko”, the remaining keywords are “Panna Cotta” and “Tiramisu”, and “Panna Cotta” exists in the corresponding keyword pair field with the appearance frequency = 1, as shown in FIG. The “Pannacotta” pair is incremented by 1 (Pannacotta, 2) and updated, and there is no corresponding keyword pair “Tiramisu”. Therefore, in the case of “Tiramisu”, the remaining keywords are the keywords “Pannacotta” (Tiramisu, 1) is additionally registered. In the case of “Tiramisu”, since there is no corresponding keyword pair, (Panna Cotta, 1) and (Nata Deco, 1) are newly registered in the field of the corresponding keyword pair of “Tiramisu”.
[0034]
Next, FIG. 5A shows the default values of the threshold values for the appearance frequency and the number of matches in the present embodiment. When the input unit 1 instructs the construction of an editable dictionary file of related words, it is determined that the appearance frequency threshold is 10, the matching frequency threshold is 5, and the default value may be left as it is. Focusing on “Pannacotta”, “Natadecoko”, and “Tiramisu”, the related word editable dictionary file construction means 9 uses the keywords “Pannacotta”, “Natadecoko”, and “Tiramisu” in FIG. Since only “Panna cotta” is extracted, only “Panna cotta” is extracted, and “confectionery” and “Nata de coco” having a match count of 5 or more are extracted from the corresponding keyword pair of the extracted “Panna cotta” entry, as shown in FIG. Register the related words in the editable dictionary file. By changing the threshold value of the appearance frequency and the threshold value of the number of matches, the keyword of each entry to be extracted also changes.
[0035]
Also in the case of related word candidate display, if the case similar to the example of the above-mentioned related word editable dictionary file is taken as an example, the related word candidate display means 11 having received the candidate display instruction, “Sweets” and “Natadecoko” exceeding the respective set threshold values are extracted, and the display unit 12 displays related word candidates.
[0036]
【The invention's effect】
In the present invention, since the related word dictionary is constructed using the keywords extracted from the document that the user is searching for, it is possible to construct a useful related word dictionary that meets the user's request.
[0037]
In addition, according to the present invention, the keyword co-occurrence relationship determination is performed based on whether or not the extracted source has appeared in the same document unit, and the keyword pair having the co-occurrence relationship is generated. Related words can be easily obtained without omission, and the frequency of occurrence is recorded and used as a material for judging the degree of association between keywords, so the related word judgment process is simple and reliable. Become. Therefore, even if the amount of data increases, it can be implemented with performance sufficient to withstand actual use.
[0038]
According to the configuration of the present invention in which related words to be registered are determined based on the appearance frequency of each extracted keyword, it is possible to omit irrelevant keywords that are rarely used.
[0039]
In addition, according to the configuration of the present invention in which the co-occurrence file is displayed according to the user's instruction and the related word dictionary is registered by the user's editing, “synonymous” or “relevant” presented to the user from the co-occurrence file The keyword group is likely to contain keywords that the user is interested in or treats as specialized, so that it is easy for users to build their own related word dictionaries, A useful related word dictionary can be constructed. At that time, some unnecessary related words that may appear may be suppressed more in accordance with the user's request by appropriately changing the threshold value of the appearance frequency according to the user's judgment.
[Brief description of the drawings]
FIG. 1A is a diagram showing an outline of a configuration for creating and registering a co-occurrence file that holds a relationship between a synonym and a related word with an extracted keyword, and FIG. 1B is an editing of a related word. The figure which shows the outline of the structure for creating a possible dictionary file, (c) is the figure which shows the outline of the structure for displaying a related word candidate, when a user constructs | assembles a related word itself. Is the contents of the index of the keyword vs. document file ID, (b) is a diagram showing the contents of the index of the keyword of the file file ID vs. the keyword (both before implementation in the embodiment)
FIGS. 3A and 3B show the contents of an index of keyword vs. document file ID, and FIG. 3B shows the contents of an index of document file ID vs. keyword (both after implementation in the embodiment).
4A is a content of a co-occurrence file entry for constructing a related word (state before implementation in the embodiment), and FIG. 4B is a content of a co-occurrence file entry for constructing a related word (in the embodiment). FIG. 5A shows an example of the contents of the appearance frequency and the threshold value of the number of matches, and FIG. 5B shows an example of the contents of an editable dictionary file of related words. Flowchart of processing for creating and registering a co-occurrence file that maintains the relationship between extracted keywords and synonyms / related words [FIG. 7] Flow chart of index registration processing [FIG. 8] Creating an editable dictionary file of related words Flowchart of Processing FIG. 9 is a flowchart of processing for displaying related word candidates.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Input means, 2 ... Keyword extraction means, 3 ... Text database part, 4 ... Index registration means, 5 ... Index file part, 6 ... Co-occurrence keyword extraction means, 7 ... Co-occurrence file registration means, 8 ... Co-occurrence file , 9 ... Related word editable dictionary file construction means, 10 ... Related word editable dictionary file section, 11 ... Related word candidate display means, 12 ... Display section.

Claims (3)

A keyword extracting means for extracting a keyword of a document to be searched;
From the extracted keyword group, a keyword group having the same document unit as the origin of extraction is set as a keyword having a co-occurrence relationship, and all keyword pairs are obtained from the keyword group having the co-occurrence relationship. A co-occurrence file means for determining the appearance frequency of the pair and holding it in the co-occurrence file;
Referencing the co-occurrence file, and a related word dictionary registering means for registering a keyword pair whose appearance frequency exceeds a set threshold value in a related word dictionary, with one of them as a head word and the other as a related word. A related word dictionary creation device comprising:
The co-occurrence file means holds the appearance frequency of each keyword in addition to the appearance frequency of the keyword pair,
The related word dictionary registration means refers to the co-occurrence file means, and the appearance frequency of the keyword pair exceeds the set threshold, and the appearance frequency of one keyword to be used as a headword in the keyword pair is set to the other A related word dictionary creation device, wherein a related word dictionary is registered when a specified threshold value is exceeded. The related word dictionary registration means has a function of editing a related word dictionary by a user, and a function of changing a threshold value of the appearance frequency of the keyword pair or a threshold value of the appearance frequency of the keyword. 2. The related word dictionary creating apparatus according to claim 1 , further comprising display means for displaying candidate related words related to an arbitrary keyword from the co-occurrence file means. The keyword extraction means for extracting the keywords of the document to be searched, and the keyword group having the same document unit as the origin of extraction from the extracted keyword group are set as co-occurrence keywords, and the co-occurrence relationship is established. Obtain all keyword pairs from the keyword group, obtain the appearance frequency of each keyword pair, store the co-occurrence file means in the co-occurrence file, and refer to the co-occurrence file, and the threshold at which the appearance frequency is set A related word dictionary creating method by a related word dictionary creating device provided with a related word dictionary registering means for registering in a related word dictionary, with one of the keyword pairs exceeding as a head word and the other as a related word,
Extracting a keyword of a document to be searched by the keyword extracting means;
The co-occurrence file means obtains a keyword group having the same document unit as a source of extraction from a keyword group extracted from a document to be searched as a keyword having a co-occurrence relationship from the keyword group having the co-occurrence relationship. Creating and registering co-occurrence files that hold keyword pairs, the frequency of occurrence of each keyword pair, and the frequency of occurrence of each keyword;
The related word dictionary registration means refers to the co-occurrence file, the appearance frequency of the keyword pair exceeds a set threshold value, and the appearance frequency of one keyword to be used as a headword in the keyword pair is another setting. Registering a keyword pair that exceeds the threshold value in a related word dictionary, with one of the keyword pairs as a headword and the other as a related word;
A related word dictionary creation method comprising:

Images