JP5597653B2 - Word extraction apparatus, method, and program - Google Patents

Description

  The present invention relates to a word extraction method, apparatus, and program, and more particularly, to a word extraction method, apparatus, and program for extracting a place name, word, and phrase pair from a document set.

  Specifically, a word extraction method, apparatus, and program for use in document search that accepts geographical conditions, classification of documents from a geographical perspective, or recommendation of search keywords that are useful in a specific geographic range, etc. About.

  2. Description of the Related Art Conventionally, a technique for outputting a document related to a keyword by inputting a keyword for a large document such as information obtained from the Internet is known.

  In recent years, with the spread of map services on the Internet, and the spread of devices that can acquire the current terminal position by GPS, etc., keywords and geographic conditions are added as part of the search conditions, adding the location and map range. There is known a technique (geographic information search) for outputting a document related to both (see, for example, Non-Patent Document 1).

  In addition, the above search can be used to search information related to a keyword of interest within a range while browsing a map. At that time, it is conceivable that the system side recommends a word or phrase that allows the user to obtain characteristic information in the map range without recalling the keyword. In that case, a keyword will be selected using a pair in which a specific region and a specific word appear significantly more than others in the document set.

  In order to implement the above-described geographic information search, search keyword recommendation, etc., a pair of a place name and a word or phrase appearing in a document is required.

  There are two major methods for extracting pairs of place names and words or phrases that appear in such documents. One is a method of using co-occurrence in a document, in which a place name and a word or phrase appear as a pair if they appear in the same document. For example, a method of processing a document in which both a place name and a designated keyword appear as shown in Non-Patent Document 1 indirectly uses this intra-document co-occurrence. Another extraction method is a method that uses smaller units, for example, units such as sentences and paragraphs, instead of co-occurrence in a document.

Yoshihito Yasuda and Hiroyuki Toda, "Geographical information search for the immediate vicinity of search location", Transactions of the Japanese Society for Artificial Intelligence, Vol. 23, No.5, pp. 364-373, July 2008.

  However, when pairs are extracted and processed for co-occurrence in a document by the conventional methods, there is a problem that even if there is no semantic connection or a semantic connection is rare, a pair is extracted. was there. For example, consider the following sentence:

“There was a Sepak Takraw match yesterday in Yokosuka. It was a defeat. However, since the yakisoba launched at home was delicious, I was satisfied! ]
In this sentence, the {Yokosuka-Yakisoba} pair is not connected semantically, but it is extracted as a pair when co-occurrence in the document is targeted.

  On the other hand, if the co-occurrence is limited to a narrow range such as in the same sentence, there is a problem that handling pairs are extremely reduced. This is because place names do not appear frequently in documents, but are often used to relate over a long range with a single appearance.

  For example, consider the following sentence:

“I have really liked this city since I moved to Yokosuka three years ago. It is a mountainous terrain, there are many tunnels, and roads tend to be congested. Sometimes it is a scar, but it is thanks to being surrounded by the sea on all sides. It has a mild climate and does not require much heating even in winter. It ’s also easy to go to Tokyo with Keikyu. ”
In this sentence, the {Yokosuka-warm} and {Yokosuka-Keikyu} pairs are considered to be deeply connected, but if they are limited to co-occurrence within a sentence, they will not be extracted as pairs. .

  To sum up these problems, if the number of pairs to be extracted is increased, the number of pairs having no semantic connection is increased. On the other hand, if the number of pairs is narrowed down, a sufficient amount of pairs cannot be extracted.

  In addition, depending on the usage, it is not necessary to use arbitrary words as long as they are related to the region, but there are cases where it is desired to search only for preferred words that match the purpose or to recommend search words. For example, when it is used to search for information centered on information about local tourism and recommend search terms, a pair that includes words that are unlikely to be related to tourism, such as the name of a candidate who was running for an election in those areas. Do not want to extract.

  However, according to the prior art, it was not possible to select a pair suitable for such a purpose.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a word extraction method, apparatus, and program capable of extracting a place name and word pair from a document according to the purpose of use.

In order to solve the above problems, the present invention (Claim 1) is a word extraction device that extracts a pair of a place name and a word or phrase from an input document set,
Document analysis means for determining word breaks in the documents of the document set for selecting the input pair extraction target document and pair candidates, and outputting the words;
The place name expression is extracted from the document set for selecting the pair extraction target document and the pair candidate, the position information where the place name exists in the document is specified, and the position information, the latitude / longitude of the representative point of the place name , Geographic information giving means for outputting a range implied by the place name;
With respect to the document set for selecting the pair candidate, the information output from the document analysis unit and the geographic information adding unit, and an exemplary group of typical pairs that are considered to be input (hereinafter referred to as “typical pairs”). A pair of words and place names that are similar to the typical pair of examples in the document set, and a suitable pair determination unit that obtains a suitability score of the pair and stores it in the preferred pair storage unit; ,
With respect to the extraction target document, if the preferred pair storage means includes a pair of a word and a place name of information output from the document analysis unit and the geographic information providing unit, the word in the extraction target document The distance to the place name is obtained, and an appropriateness score for the pair is obtained. A determination score having a value that decreases as the distance increases and increases as the appropriateness score increases is calculated. Pair extraction means for extracting pairs of values or more.

In the present invention (Claim 2), in the preferred pair determining means,
When calculating the suitability score of the pair, a bootstrapping algorithm is used only when one of the pair is a place name.

In the present invention (Claim 3), in the pair extraction means,
Means in which the distance between the word w _i and the place name g _j in the document to be extracted is the number of sentences included between the sentence in which the word W _i appears and the sentence in which the place name g _j appears including.

In the present invention (Claim 4), in the pair extraction means,
Means for calculating the judgment score to be paired even if the score in the document is less than a predetermined value if the distance in the document is within one sentence;
Or
If the determination score is higher than a predetermined value, the determination score includes any means that calculates a pair even if the distance is longer than the predetermined value.

  According to the present invention, a typical group of examples according to the purpose of use is given to a document set, and a pair of place name and word similar to the typical group of examples and a score (preferability score) are obtained by existing technology. Store a word (word) and place name from the input document, calculate a determination score with a value that decreases as the distance between the word and the place name increases and increases as the word suitability score increases, If the judgment score is greater than or equal to a predetermined value, outputting pairs of words and place names avoids extracting pairs with a weak semantic connection in the document and allows the user to It is possible to extract a large number of pairs in a document limited to only examples that are considered suitable.

It is a block diagram of the word extraction apparatus in one embodiment of this invention. It is an example of the suitable pair memory | storage part in one embodiment of this invention. It is a flowchart of the outline | summary operation | movement in one embodiment of this invention. It is a flowchart of the process of the pair extraction part in one embodiment of this invention.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a configuration of a word extraction device according to an embodiment of the present invention.

  A word extraction device 10 shown in FIG. 1 includes a document analysis unit 11, a geographic information addition unit 12, a suitable pair determination unit 13, a pairing preference storage unit 14, and a pair extraction unit 15.

  It should be noted that a document set for selecting a suitable pair candidate is not given each time, but may be registered in advance in a database or the like.

  As shown in FIG. 2, the paired moderate storage unit 14 stores a suitable pair (place name and word) obtained by the suitable pair determining unit 13 and the suitability score of the pair.

  Hereinafter, processing in the above configuration will be described.

  FIG. 3 is a flowchart of an outline operation in one embodiment of the present invention.

  Step 100) The document analysis unit 11 inputs a document to be extracted and a document set for selecting a suitable pair candidate, and determines word breaks in the document. For this, morphological analysis techniques known to those skilled in the art can be used. Furthermore, as necessary, not only words but also chunking for extracting phrases as a syntactic unit, extraction of proper expressions (extraction of proper nouns such as names of person names and time expressions), and the like may be performed. A technique known to those skilled in the art can also be used for these.

  Hereinafter, these separated words, chunks, specific expressions, and the like are collectively referred to as “words”.

  If the document set for selecting a suitable pair candidate does not change each time, the result of performing the process only once may be stored in a database or the like.

  Further, the geographic information giving unit 12 receives each document analyzed by the document analysis unit 11 as input, specifies an expression that seems to be a place name described in the document, and the position of the place name in the document, the representative point of the place name Output the range implied by the latitude / longitude and place name. To specify the place name and the latitude and longitude of the representative point of the place name, reference 1 “Toru Hirano, Yoshihiro Matsuo, Genichiro Kikui,“ Resolving ambiguity of place names using geographical distance and famousness ”, Information Processing Society of Japan The conventional method shown in the National Convention Proceedings, 2008 ", etc. can be used. For the implications of place names, the conventional method as shown in Non-Patent Document 1 or existing numerical maps can be used. If the document set for selecting a suitable pair candidate does not change each time, the result of performing the process only once may be stored in a database or the like.

Step 200) The suitable pair determination unit 13 selects a suitable pair candidate, the analyzed document set provided by the document analysis unit 11 and the geographic information addition unit 12, and a typical example that is considered preferable. Take a pair of examples as input. In the preferred pair determination unit 13, Reference 2 "Patrick Pantel and Marco Pennacchiotti. 2006. Espresso: leveraging generic patterns for automatically harvesting semantic relations. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics (ACL-44). Associations for Computational Linguistics, Stroudsburg. PA, USA, 113-120. A known technique of the kind called can be used, which gives a small amount of suitable pairs as seed data and determines the patterns in which these seed data pairs appear in the document set, and then matches pairs that match this pattern. This is an algorithm that applies these repeatedly.Instead of simply using the bootstrapping algorithm, processing is performed only when one of the pairs is a place name. Next, the preferred pair determination unit 13 stores the pair obtained by bootstrapping as a preferred pair. Storing to 14. At this time, the resulting pairs confidence score (e.g., r _{i (i)} in reference 2), and stores the preferred scores. Determination of a preferred pair to elect a suitable pair candidates Even if the set of documents for change does not change, it is executed every time the group of typical pairs considered as preferable changes.

Step 300) The pair extraction unit 15 receives the processing result of the document analysis unit 11 and the geographic information addition unit 12 for the document to be extracted, and inputs all pairs of each word w _i and each location name g _j in the document. 4 is determined by the procedure shown in FIG.

  Step 301) The pair extraction unit 15 first initializes an output pair set on a memory (not shown).

  Step 302) The following processing is repeated for all place names g-words w in the document.

Step 303) Referring to the preferred pair storage unit 14, if the pair of the word w _i and the place name g _j is included, the process proceeds to Step 304. If the pair is not included, the pair is not included in the output. Transition to paired processing.

Step 304) word _{w i} and place names _{g j} the distance _{d (w} i in the document, the _{g j),} the sentence word _{w i} has appeared, the statements contained between sentences places _{g j} appeared It is a number. In particular, if w _i and g _j appear in the same sentence, d (w _i , g _j ) = 0. If w _i or g _j appears multiple times in the document, the one with the smallest distance is defined as d (w _i , g _j ).

Step 305) The suitability score p (w _i , g _j ) of the pair in the preferred pair storage unit 14 is acquired.

Determined by step 306) following expression score _{s (w} i, determining the _{g j).}

ここで、αは事前に設定した定数パラメータである。

Here, α is a constant parameter set in advance.

The calculation of the determination score s (w _i , g _j ) is not limited to the above formula, and qualitatively, if the distance in the document is shorter than a predetermined value such as within one sentence, the degree of preference Even if the score is lower than a predetermined value, it is recognized as a pair, and if the suitability score is higher than a predetermined value, the pair may be recognized even if the distance in the document is larger than the predetermined value.

  For example, in addition to the method obtained above, the determination may be made by using a constant parameter β set in advance and a step-like function as defined below.

ステップ３０７） 判定スコアｓ（ｗ_ｉ，ｇ_ｊ）が１以上であれば、抽出対象とし、ステップ３０８に移行し、１未満であれば当該対は抽出対象ではないとし、ステップ３０９に移行する。

Step 307) If the determination score s (w _i , g _j ) is 1 or more, it is determined as an extraction target, and if it is less than 1, it is determined that the pair is not an extraction target, and the process proceeds to Step 309.

Step 308) Add the pair of judgment scores s (w _i , g _j ) ≧ 1 to the output pair set of the memory (not shown).

  Step 309) If the processing after step 303 is completed for all place names g-words w, the process proceeds to step 310. If not yet completed, the process proceeds to step 303.

  Step 310) Output an output pair set of a memory (not shown).

  As described above, the present invention gives a typical group of examples according to the purpose of use to a document set, and obtains a pair of place name and word similar to the typical group of examples and its term moderate score. When a document is input, a word and place name are extracted from the document, and a determination score is calculated so that the smaller the distance between the word and the place name is, the larger the term appropriate score of the word is. If the score is greater than or equal to a predetermined value, a pair of word and place name is output. This makes it possible to select a pair suitable for the purpose of use.

  It is also possible to construct the operation of each component of the word extraction device shown in FIG. 1 as a program, install it on a computer used as the word extraction device, execute it, or distribute it via a network.

  The present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the scope of the claims.

DESCRIPTION OF SYMBOLS 10 Word extraction apparatus 11 Document analysis part 12 Geographic information provision part 13 Suitable pair determination part 14 Pair suitable degree memory | storage part 15 Pair extraction part

Claims (8)

A word extraction device for extracting a pair of a place name and a word or phrase from an input document set,
Document analysis means for determining word breaks in the documents of the document set for selecting the input pair extraction target document and pair candidates, and outputting the words;
The place name expression is extracted from the document set for selecting the pair extraction target document and the pair candidate, the position information where the place name exists in the document is specified, and the position information, the latitude / longitude of the representative point of the place name , Geographic information giving means for outputting a range implied by the place name;
With respect to the document set for selecting the pair candidate, the information output from the document analysis unit and the geographic information adding unit, and an exemplary group of typical pairs that are considered to be input (hereinafter referred to as “typical pairs”). A pair of words and place names that are similar to the typical pair of examples in the document set, and a suitable pair determination unit that obtains a suitability score of the pair and stores it in the preferred pair storage unit; ,
For the extraction target document, if a pair of each word and each name of information output from the document analysis unit and the geographic information providing unit is included in the suitable pair storage unit, A distance between a word and the place name is obtained, and a suitable score for the pair is obtained. A determination score having a value that decreases as the distance increases and increases as the suitable score increases is calculated. A pair extracting means for extracting a pair having a predetermined value or more;
A word extraction device characterized by comprising: The suitable pair determining means includes
The word extraction device according to claim 1, wherein when calculating the suitability score of the pair, a bootstrapping algorithm is used by limiting to one of the pairs being a place name. The pair extraction means includes
Means in which the distance between the word w _i and the place name g _j in the document to be extracted is the number of sentences included between the sentence in which the word W _i appears and the sentence in which the place name g _j appears The word extraction device according to claim 1, comprising: The pair extraction means includes
Means for calculating the judgment score to be paired even if the score in the document is less than a predetermined value if the distance in the document is within one sentence;
Or
The word extraction device according to claim 1, further comprising: means for calculating the determination score so that if the suitability score is higher than a predetermined value, the determination score is paired even if the distance is longer than the predetermined value. A word extraction method for extracting a pair of a place name and a word or phrase from an input document set,
A document analysis step in which the document analysis means determines word breaks in the documents of the document set for selecting the input pair extraction target document and pair candidates, and outputs the words;
Geographic information adding means extracts a place name expression from the document set for selecting the pair extraction target document and the pair candidate, specifies the position information in which the place name exists in the document, the position information, the place name Geographic information giving step for outputting the latitude / longitude of the representative point and the range implied by the place name;
When a typical pair of examples is input, the preferred pair determination unit uses the information output in the document analysis step and the geographic information addition step for the document set for selecting the pair candidates, and bootstrapping A suitable pair determination step of obtaining a word similar to the typical pair group in the document set, a pair of place names, and a suitability score of the pair by an algorithm, and storing the pair in a suitable pair storage unit;
If the pair extraction unit includes a pair of each word and each name of the information output by the document analysis step and the geographic information addition step for the extraction target document in the suitable pair storage unit, the extraction target The distance between the word and the place name in the document is obtained, and a suitable score for the pair is obtained, and a determination score having a value that decreases as the distance increases and increases as the suitable score increases is calculated. A pair extraction step for extracting a pair whose determination score is equal to or greater than a predetermined value;
A word extraction method characterized by: In the pair extraction step,
The distance between the word w _i and the place name g _j in the document to be extracted is the number of sentences included between the sentence in which the word W _i appears and the sentence in which the place name g _j appears. Item 6. The word extraction method according to Item 5. In the pair extraction step,
If the distance in the document is within one sentence, the determination score is calculated as a pair even if the suitability score is lower than a predetermined value.
Or
The word extraction method according to claim 5, wherein the determination score is calculated so as to be paired even if the distance is longer than a predetermined value if the suitability score is higher than a predetermined value. Computer
The word extraction program for functioning as each means of the word extraction apparatus of any one of Claims 1 thru | or 4.

Images