JPH1185797A - Automatic document classification device, learning device, classification device, automatic document classification method, learning method, classification method and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an automatic document classification device which can form a vector space where topics are precisely reflected and which can appropriately execute classification. SOLUTION: The automatic document classification device selects a valid word from a learning document (valid word selection part 103). The number of the valid words contained in respective paragraphs is obtained by referring to the learning document and the valid word (intra-paragraph valid word number calculation part 105). The intra-paragraph cooccurrence frequency of the group of the respective valid words is obtained by using the number of intra-paragraph valid words (intra-paragraph cooccurrence calculation part 107). The valid word vectors of the respective valid words are obtained from obtained intra- paragraph cooccurrence frequency, and the document vectors are obtained on the learning document and the document being a classification object by referring to the valid word vectors. The folder vectors of the respective categories, which are obtained from the document vector of the learning document, are compared with the document vector of the document being the classification object. The category to which the document being the classification object belongs is decided in accordance with the compared result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an automatic document classification device for classifying documents to be classified according to the user's intention, a learning device and a classification device used therefor, an automatic document classification method, a learning method and a classification used therefor. The present invention relates to a method and a storage medium for constructing an automatic document classification device.

[0002]

2. Description of the Related Art As one of methods for classifying documents to be classified according to a user's intention, there is a method using a vector space model. In this vector space model, words, documents, and categories useful for classification are represented by vectors, and the category to which the document belongs is determined from the direction of the vectors. The automatic document classification processing using this vector space model is divided into a learning phase and a classification phase. In the learning phase, words useful for classification (hereinafter, referred to as effective words) are selected from learning documents that have been correctly classified in advance, and each effective word is expressed as a vector. This vector is called the effective word vector,
The components of the effective word vector are obtained from the appearance frequency, the word co-occurrence probability, and the like. In addition, a folder vector representing the characteristics of each category is calculated by expressing the learning document in a vector. In the classification phase, a document to be classified is represented as a vector using the effective word dictionary obtained in the learning phase (hereinafter, referred to as a document vector), the document vector is compared with a folder vector, and classification is performed according to the comparison result. Determine the category to which the target document belongs.

[0003] The configuration of an automatic document classification apparatus employing this method will be described with reference to FIGS. 7 to 9. FIG. 7 is a block diagram showing the configuration of a conventional automatic document classification device.
FIG. 8 is a flowchart showing the processing procedure of the learning phase in the automatic document classification apparatus of FIG. 7, and FIG. 9 is a flowchart showing the processing procedure of the classification phase in the automatic document classification apparatus of FIG.

[0004] As shown in FIG.
A learning document holding unit 501 that holds a learning document, a classification target document holding unit 502 that holds a classification target document, an effective word selection unit 503 that selects an effective word from the learning document, and a selected valid word. An effective word holding unit 504 to hold, an effective word number calculation unit 505 for calculating the number of effective words included in each document by referring to the learning document and the effective word, and an effective word in each obtained document. A document effective word count holding unit 506 for holding the word count.

The number of effective words in the document held in the effective word number storage unit 506 is given to a co-occurrence frequency calculation unit 507 in the document. Find the co-occurrence frequency of a word set in a document. The obtained co-occurrence frequency in the document is stored in the co-occurrence frequency in document storage unit 50.
8 is provided to the effective word vector calculation unit 509. The effective word vector calculation unit 509 obtains an effective word vector of each effective word using the co-occurrence frequency in the document. Here, assuming that the co-occurrence probability of the valid word Ti and the valid word Tj is ci, j and the number of valid words is N, the valid word vector Ti of the valid word Ti
Is given by the following equation (1): Ti = (ci, 1, ci, 2, ..., ci, N) (1) The co-occurrence probability ci, j is defined by the following equation (2).

Ci, j = (the number of documents including both Ti and Tj) / (the number of documents including Ti) (2) The effective word vector obtained by the effective word vector calculation unit 509 is an effective word vector holding unit. After being stored in 510, it is provided to the document vector calculation unit 511. The document vector calculation unit 511 obtains a document vector by referring to the effective word vector for each of the learning document and the classification target document. The document vector obtained for each of the learning document and the classification target document is a document vector holding unit. 512. The document vector of the learning document held in the document vector holding unit 512 is a folder vector calculation unit 513.
And the folder vector calculation unit 513 obtains a folder vector of each category using the document vector of the learning document. The obtained folder vector of each category is stored in the folder vector storage unit 514.

[0007] The folder vector of each category stored in the folder vector storage unit 514 is provided to the classification determination unit 515 together with the document vector of the classification target document stored in the document vector storage unit 512. The document vector of the document is compared with the folder vector of each category, and the category to which the document to be classified belongs is determined according to the comparison result. The determined category of the classification target document is stored in the classification result storage unit 516.

Next, the processing procedure of the learning phase in the automatic document classification device will be described with reference to FIG.

First, in step S601, words useful for classification are selected as effective words from the words included in the document requiring learning, and in step S602, the number of selected effective words included in each document is determined. Ask for.

Next, the process proceeds to step S603, in which the co-occurrence frequency of each set of effective words in the document is obtained from the number of effective words in the document. In step S604, an effective word vector is calculated from the co-occurrence frequency in the document. Then, in step S605, an effective word is extracted from the learning document with reference to the effective word vector,
In the next step S606, the average of the effective word vectors of the extracted effective words is calculated to obtain the document vector of the learning document.

Next, the process proceeds to step S607, in which the average of the document vectors of the documents belonging to each category in the learning document is calculated, the folder vector is obtained from the average of the vectors of the documents, and the process ends.

When the learning phase ends, a classification phase starts. The processing procedure of this classification phase will be described with reference to FIG.

In the classification phase, first, step S701
In step, an effective word is extracted from the document to be classified with reference to the effective word vector obtained in step S604, and an average of the effective word vectors extracted in step S702 (effective word vector obtained in step S604) is calculated. A document vector of the document to be classified is obtained from the average of the vectors.

Next, in step S703, the document vector of the document to be classified is compared with the folder vector obtained in the learning phase, the category to which the document to be classified belongs is determined according to the comparison result, and the process ends. I do.

[0015]

However, in the conventional document automatic classification apparatus described above, the effective word vector is obtained from the co-occurrence frequency of the effective word in the learning document in the document. Appearing valid words are also determined to co-occur, and a vector space that accurately reflects the topic may not be formed, and classification cannot be performed properly.

An object of the present invention is to provide an automatic document classification apparatus, an automatic document classification method, and a storage medium that can form a vector space that accurately reflects a topic and can perform classification appropriately. .

Another object of the present invention is to provide a learning apparatus and a classifying apparatus capable of forming a vector space accurately reflecting a topic and realizing an automatic document classification system capable of appropriately performing classification. , A learning method, a classification method, and a storage medium.

[0018]

According to the first aspect of the present invention,
An automatic document classification apparatus for classifying a document to be classified according to a user's intention using a learning document and an effective word selected from the learning document. Means for calculating the number of effective words in each sentence included in each unit, and means for calculating the co-occurrence frequency in each sentence for each set of effective words by referring to the number of effective words Means, an effective word vector calculation means for obtaining an effective word vector of each effective word by referring to the co-occurrence frequency in the text unit, and for each of the learning document and the classification target document, the effective word vector Document vector calculating means for obtaining a document vector by reference, and folder vector calculating means for obtaining a folder vector of each category using the document vector obtained for the learning document Classifying means for comparing a document vector obtained for the classification target document with a folder vector of each category, and determining a category to which the classification target document belongs according to the comparison result. .

According to a second aspect of the present invention, there is provided an automatic document classification system for classifying documents to be classified according to a user's intention, and a folder of each category serving as a reference for determining a category to which the document to be classified belongs. In a learning device for obtaining a vector, a learning document holding unit that holds a learning document, an effective word selecting unit that selects an effective word from the learning document, and a reference to the effective word for the learning document. Means for calculating the number of effective words included in each sentence unit for each sentence unit; and co-occurrence within the sentence unit for obtaining the co-occurrence frequency within the sentence unit for each set of effective words with reference to the number of effective words A frequency calculating means, an effective word vector calculating means for obtaining an effective word vector of each effective word by referring to the co-occurrence frequency in the sentence unit, and a document vector referring to the effective word vector A document vector calculating means for calculating a,
Folder vector calculating means for obtaining a folder vector of each category using the document vector.

According to a third aspect of the present invention, there is provided an automatic document classification system for classifying a document to be classified according to a user's intention together with the learning apparatus according to the second aspect, for determining a category to which the document to be classified belongs. A classification target document holding unit that holds the classification target document; a document vector calculation unit that obtains a document vector for the classification target document by referring to an effective word vector obtained by the learning device; Classification determining means for comparing a document vector obtained for the classification target document with a folder vector of each category obtained by the learning device, and determining a category to which the classification target document belongs according to the comparison result. It is characterized by.

According to a fourth aspect of the present invention, there is provided an automatic document classification method for classifying a document to be classified according to a user's intention using a learning document and an effective word selected from the learning document. A step of obtaining the number of valid words included in each sentence unit by referring to the valid words for the document; anda step of calculating a co-occurrence frequency in the sentence unit of each set of valid words by referring to the number of valid words. Determining the effective word vector of each effective word by referring to the co-occurrence frequency within the sentence unit;
A step of obtaining a document vector by referring to the effective word vector for each of the learning document and the classification target document; and a step of obtaining a folder vector of each category using the document vector obtained for the learning document And comparing the document vector obtained for the classification target document with the folder vector of each category, and determining a category to which the classification target document belongs according to the comparison result.

According to a fifth aspect of the present invention, there is provided an automatic document classification system for classifying documents to be classified according to a user's intention, and a folder of each category serving as a reference for determining a category to which the document to be classified belongs. In the learning method for obtaining a vector, a step of holding a learning document, a step of selecting an effective word from the learning document, and a step of selecting the effective word from the learning document, the effective word being included in each sentence unit with reference to the effective word. Determining the number of valid words to be calculated, and calculating the number of valid words by referring to the number of valid words, and calculating the co-occurrence frequency within the sentence unit of each set of valid words. Determining the effective vector of the category, obtaining the document vector with reference to the effective vector, and obtaining the folder vector of each category using the document vector. Characterized in that it comprises a.

According to a sixth aspect of the present invention, there is provided an automatic document classification system for classifying a document to be classified according to a user's intention together with the learning method according to the fifth aspect, for determining a category to which the document to be classified belongs. In the classification method, the step of holding the classification target document, the step of obtaining a document vector for the classification target document by referring to the effective word vector obtained by the learning method, and the step of obtaining the classification target document Comparing the document vector with the folder vector of each category obtained by the learning method, and determining a category to which the classification target document belongs according to the comparison result.

According to a seventh aspect of the present invention, there is provided an automatic document classifying apparatus for classifying a document to be classified according to a user's intention using a learning document and an effective word selected from the learning document. In the storage medium storing the program,
The program includes a module for calculating the number of valid words included in each sentence unit for each sentence unit with reference to the valid words for the learning document, and a module for calculating each valid word with reference to the number of valid words. A set of sentence co-occurrence frequencies within a sentence unit, a co-occurrence frequency calculation module within a sentence unit, an effective word vector calculation module for obtaining an effective word vector for each effective word by referring to the co-occurrence frequency within a sentence unit, For each of the learning document and the classification target document, a document vector calculation module that obtains a document vector by referring to the effective word vector, and a folder vector of each category is obtained using the document vector obtained for the learning document. A folder vector calculation module to be obtained, a document vector obtained for the classification target document, and a folder of each category. Comparing the vector, characterized in that it comprises a classification decision module that determines the category of the classifying target document belongs in accordance with the comparison result.

According to an eighth aspect of the present invention, there is provided an automatic document classification system for classifying documents to be classified according to a user's intention, and a folder of each category serving as a reference for determining a category to which the document to be classified belongs. In a storage medium storing a learning program for constructing a learning device for obtaining a vector, the learning program includes a learning document holding module for holding a learning document, and an effective word for selecting an effective word from the learning document. A word selection module, a valid word count calculation module within a sentence unit for obtaining the number of valid words included in each sentence unit with reference to the valid words for the learning document, and a valid word number referring to the valid word number. A sentence-unit co-occurrence frequency calculating module for obtaining a sentence-unit co-occurrence frequency of a set of words, and An effective word vector calculation module for obtaining an effective word vector of an effective word; a document vector calculation module for obtaining a document vector by referring to the effective word vector; and a folder vector module for obtaining a folder vector of each category using the document vector. And characterized in that:

According to a ninth aspect of the present invention, there is provided an automatic document classification system for categorizing a document to be classified according to a user's intention, together with the storage medium according to the eighth aspect, for determining a category to which the document to be classified belongs. 9. A storage medium storing a classification program for constructing a classification device according to claim 8, wherein the classification program stores a classification target document holding module for holding the classification target document and the classification target document. 9. A document vector calculation module for obtaining a document vector by referring to an effective word vector obtained by the learning program of claim 8, and a document vector obtained for the classification target document.
And a classification determining module that compares the folder vector of each category obtained by the learning program of the storage medium described above and determines a category to which the classification target document belongs according to the comparison result.

[0027]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing a functional configuration of an embodiment of the automatic document classification device of the present invention, and FIG. 2 is a block diagram showing a hardware configuration of the automatic document classification device of FIG.

As shown in FIG. 1, the automatic document classifying apparatus
A learning document holding unit 101 that holds a learning document, a classification target document holding unit 102 that holds a classification target document, an effective word selection unit 103 that selects an effective word from the learning document, and a selected valid word A valid word holding unit 104 for holding, a valid word count calculating unit 105 for calculating the number of valid words included in each paragraph with reference to the learning document and the valid words, A valid word count holding unit 106 for holding the number of valid words.

The number of effective words in each paragraph held in the effective word number holding section 106 is given to the intra-paragraph co-occurrence frequency calculation section 107, and the intra-paragraph co-occurrence frequency calculation section 107 uses the number of effective words in the paragraph to calculate each word. Find the co-occurrence frequency of a set of valid words in a paragraph. The obtained intra-paragraph co-occurrence frequency is stored in the intra-paragraph co-occurrence frequency holding unit 1.
After being stored in 08, it is provided to the effective word vector calculation unit 109. The effective word vector calculation unit 109 obtains an effective word vector of each effective word using the intra-paragraph co-occurrence frequency. Here, assuming that the co-occurrence probability of the valid word Ti and the valid word Tj is c′i, j and the number of valid words is N, the valid word vector T′i of the valid word Ti
From the following equation (3), T′i = (c′i, 1, c′i, 2,..., C′i, N) (3) The co-occurrence probability ci, j is defined by the following equation (4).

C′i, j = (number of paragraphs including both Ti and Tj) / (number of paragraphs including Ti) (4) The effective word vector calculated by the effective word vector calculation unit 109 is an effective word vector After being held in the holding unit 110, it is given to the document vector calculation unit 111. The document vector calculation unit 111 obtains a document vector for each of the learning document and the classification target document by referring to the effective word vector. The document vector obtained for each of the learning document and the classification target document is a document vector holding unit. 112. The document vector of the learning document held in the document vector holding unit 112 is a folder vector calculation unit 113
And the folder vector calculation unit 113 obtains a folder vector of each category using the document vector of the learning document. The obtained folder vector of each category is stored in the folder vector storage unit 114.

The folder vector of each category stored in the folder vector storage unit 114 is provided to the classification determination unit 115 together with the document vector of the classification target document stored in the document vector storage unit 112. The document vector of the document is compared with the folder vector of each category, and the category to which the document to be classified belongs is determined according to the comparison result. The determined category of the document to be classified is stored in the classification result storage unit 116.

In the hardware configuration of the automatic document classification apparatus, as shown in FIG. 2, a control program stored in a ROM 201 is executed to execute control (to be described later) (FIG. 3).
And a central processing unit 203 for performing the control shown in FIG. 4). The RAM 202 is used as a work area for the arithmetic processing of the central processing unit 203.
2 is a valid holding unit 104 and a paragraph co-occurrence frequency holding unit 10
8. Document vector storage unit 112, classification result storage unit 116
To provide storage space for

The central processing unit 203 includes a hard disk drive 204 together with a ROM 201 and a RAM 202.
Is connected via the bus 205 and the hard disk drive 2
Reference numeral 04 constitutes the learning document storage unit 101, the classification target document storage unit 102, the effective word vector storage unit 110, and the folder vector storage unit 114. Note that, instead of the hard disk drive 204, another storage medium is used to store the learning document holding unit 101, the classification target document holding unit 102, the valid word vector holding unit 110, and the folder vector holding unit 11.
4 can also be configured.

Next, the processing executed by the automatic document classification apparatus will be described with reference to FIGS. FIG.
4 is a flowchart showing a processing procedure of a learning phase in the automatic document classification apparatus of FIG. 1, and FIG. 4 is a flowchart showing a processing procedure of a classification phase in the automatic document classification apparatus of FIG.

The processing in the automatic document classification apparatus is divided into a learning phase and a classification phase. First, a processing procedure in the learning phase will be described with reference to FIG.

In the learning phase, as shown in FIG. 3, first, in step S301, a word useful for classification is selected as an effective word from words included in the document requiring learning, and in step S302, each word is included in each paragraph. Find the number of selected valid words included.

Then, the process proceeds to step S303, where the co-occurrence frequency in each paragraph of the set of effective words is obtained from the number of effective words in each paragraph.
In a succeeding step S304, an effective word vector is calculated from the intra-paragraph co-occurrence frequency. Then, in step S305, an effective word is extracted from the learning document with reference to the effective word vector. In step S306, the effective vector of the extracted effective words is averaged to obtain a document vector of the learning document.

Next, the process proceeds to step S307, where the average of the document vectors of the documents belonging to each category in the learning document is obtained, the folder vector is obtained from the average of the vectors of the documents, and this processing is completed.

When the learning phase ends, the classification phase starts. The processing procedure of this classification phase will be described with reference to FIG.

In the classification phase, as shown in FIG. 4, first, in step S401, an effective word is extracted from the document to be classified with reference to the effective word vector calculated in step S304. The average of the (effective word vector calculated in step S304) is taken, and the document vector of the document to be classified is determined from the average of this vector.

Next, the process proceeds to step S403, where the document vector of the document to be classified is compared with the folder vector obtained in the learning phase, the category to which the document to be classified belongs is determined according to the comparison result, and this processing ends. I do.

As described above, in the present embodiment, different topics have been described by obtaining an effective word vector from a co-occurrence frequency in a paragraph using a paragraph structure provided according to a change in the content of a document. Effective words appearing in one paragraph are not judged to co-occur, and a vector space whose meaning accurately reflects topics based on word co-occurrence can be formed, and classification must be performed properly. Can be.

In the present embodiment, the setting is made such that the number of valid words in a paragraph is obtained after the selection of valid words from the learning document is completed. The number of appearances in a paragraph may be calculated.

Further, in the present embodiment, in the learning phase, the effective word vector of each effective word is obtained after obtaining the co-occurrence frequency for the set of effective words. The calculation of the word vector may be performed in parallel.

Further, in the present embodiment, the co-occurrence frequency in a paragraph is obtained for each paragraph. However, the present invention is not limited to this, and it can be handled for other sentences such as sentences and sections. It is.

Further, in the present embodiment, a program for executing the above-described processing (function of each block) is
Although the example in which the program is stored in the OM is shown, it is also possible to use another storage medium to supply the program. Further, the present device can be configured by a circuit configuration having the function of each block.

Further, the present apparatus can be constructed on an information processing apparatus such as a computer. In this case, a storage medium storing a program for executing the above-described processing (the function of each block) is prepared, and the CPU or the like reads out the program from the storage medium and executes the program. Is done. As a storage medium for supplying the above program, a floppy disk, hard disk, optical disk, magneto-optical disk, C
DROM, CD-R, magnetic tape, nonvolatile memory card, ROM, and the like can be used. Note that when the automatic document classification apparatus is configured by executing the above-described program, the case where the OS running on the computer is configured to execute a part or all of the processing included in the above-described program is also included. It is. Further, after the program supplied from the storage medium is written to the extension function board mounted on the computer or the connected peripheral extension unit, the CPU provided on the extension function board or the peripheral extension unit executes the written program. It is also included.

Further, the principle of the present invention can be applied to any of a system including a plurality of devices and an apparatus including a single device.

Further, in the present embodiment, an example in which the learning phase and the classification phase are performed on one device has been described. However, the present invention is not limited to this. And a device that performs document classification, and classifies documents using the respective devices. In this case, a method for obtaining a valid word vector and a folder vector by a device that performs a learning phase, and supplying the valid word vector and the folder vector to a device that performs a classification phase by a portable storage medium or communication, and performing classification. Is used.

An apparatus for performing the learning phase and an apparatus for performing the classification phase will be described with reference to FIGS. FIG. 5 is a block diagram showing a configuration of an embodiment of the learning device of the present invention, and FIG. 6 is a block diagram showing a configuration of an embodiment of the classification device of the present invention.

As shown in FIG. 5, an apparatus for performing a learning phase includes a learning document holding unit 801 for holding a learning document.
And an effective word selecting unit 80 for selecting an effective word from the learning document
2 and a valid word holding unit 803 holding the selected valid word
And the number of effective words included in each paragraph with reference to the learning document and the effective words, and calculates the number of effective words included in each paragraph.
And a number-of-effective-words-in-paragraph holding unit 805 that holds the calculated number of effective words in each paragraph.

The number of effective words in each paragraph held in the effective number of words in paragraph holding section 805 is given to the intra-paragraph co-occurrence frequency calculation section 806. Find the co-occurrence frequency of a set of valid words in a paragraph. The obtained intra-paragraph co-occurrence frequency is stored in the intra-paragraph co-occurrence frequency holding unit 8.
After being held at 07, it is provided to the effective word vector calculation unit 808. The effective word vector calculation unit 808 obtains an effective word vector of each effective word using the intra-paragraph co-occurrence frequency.

The valid word vector obtained by the valid word vector calculation unit 808 is provided to the document vector calculation unit 810 after being stored in the valid word vector storage unit 809. The document vector calculation unit 810 obtains a document vector for the learning document by referring to the effective word vector, and the obtained document vector for the learning document is stored in the document vector storage unit 811. Document vector holding unit 81
The document vector of the learning document held in 1 is given to the folder vector calculation unit 812, and the folder vector calculation unit 812 obtains a folder vector of each category using the document vector of the learning document. The obtained folder vector of each category is stored in a folder vector holding unit 813.
Is held.

The folder vector of each category stored in the folder vector storage unit 813 and the effective word vector stored in the effective word vector storage unit 809 are stored in a portable storage medium and supplied to a device that performs a classification phase. , Or by communication to a device that performs the classification phase.

As shown in FIG. 6, the apparatus for performing the classification phase includes a classification target document holding unit 90 for holding classification target documents.
1, an effective word vector holding unit 902 for holding an effective word vector supplied from a device performing the learning phase via a portable storage medium or communication, and a valid word vector holding unit 902 via a portable storage medium or communication from the device performing the learning phase A folder vector holding unit 905 for holding the supplied folder vector,
A document vector calculation unit 903 that obtains a document vector by referring to an effective word vector for a classification target document, and a document vector holding unit 904 that stores a document vector obtained for the classification target document.

The document vector of the document to be classified held in the document vector holding unit 904 is given to the classification determining unit 906 together with the folder vector of each category held in the folder vector holding unit 905. The document vector of the document is compared with the folder vector of each category, and the category to which the document to be classified belongs is determined according to the comparison result. The determined category of the classification target document is stored in the classification result storage unit 907.

[0058]

As described above, according to the automatic document classification apparatus of the first aspect, the number of effective words included in each sentence unit is obtained for each sentence unit by referring to the effective words in the learning document. Means for calculating the number of effective words in a sentence unit; means for calculating the co-occurrence frequency in a sentence unit for each set of effective words with reference to the number of effective words; and An effective word vector calculating means for obtaining an effective word vector of an effective word; a document vector calculating means for obtaining a document vector by referring to an effective word vector for each of the learning document and the classification target document; Vector vector calculating means for obtaining a folder vector of each category using the document vector obtained, and comparing the document vector obtained for the document to be classified with the folder vector of each category. And, from and a classification determining means for determining a category of classification target document belongs in accordance with the comparison result, the topic to be able to form a vector space that accurately reflects, it is possible to properly carry out the classification.

According to the second aspect of the present invention, a learning document holding unit for holding a learning document, an effective word selecting unit for selecting an effective word from the learning document, and referring to the effective word for the learning document. Means for calculating the number of effective words contained in each sentence unit for each sentence unit, and a co-occurrence frequency for each set of effective words with reference to the number of effective words. A word frequency calculating means, an effective word vector calculating means for obtaining an effective word vector of each effective word by referring to a co-occurrence frequency in a sentence unit, a document vector calculating means for obtaining a document vector by referring to an effective word vector, Since a folder vector calculating means for obtaining a folder vector of each category using a vector is provided, a vector space accurately reflecting a topic can be formed, and classification can be performed properly. It is possible to realize a written automatic classification system.

According to the third aspect of the present invention, a classifying target document holding means for holding a classifying target document, and a document for obtaining a document vector for the classifying target document by referring to an effective word vector obtained by a learning device. Vector calculation means,
A classification vector determining unit that compares the document vector obtained for the classification target document with the folder vector of each category obtained by the learning device and determines a category to which the classification target document belongs according to the comparison result; A vector space that accurately reflects a topic can be formed, and a document automatic classification system that can appropriately perform classification can be realized.

According to the automatic document classification method of the fourth aspect, a step of referring to the effective words in the learning document to determine the number of effective words included in each sentence unit, and referring to the number of effective words. A step of obtaining the co-occurrence frequency in the sentence unit of each set of effective words, a step of obtaining an effective word vector of each effective word by referring to the co-occurrence frequency in the sentence unit, and a process for each of the learning document and the classification target document. A step of obtaining a document vector by referring to an effective word vector, a step of obtaining a folder vector of each category using a document vector obtained for a learning document, and a step of obtaining a document vector obtained for a classification target document and And comparing the folder vector with the folder vector and determining a category to which the document to be classified belongs according to the comparison result, thereby forming a vector space that accurately reflects the topic. It can be, it is possible to properly perform the classification.

According to the fifth aspect of the present invention, there is provided a step of holding a learning document, and a step of referring to an effective word in the learning document to determine the number of effective words included in each sentence unit. , Selecting valid words from the learning document,
A step of obtaining the co-occurrence frequency in the sentence unit of each set of effective words by referring to the number of effective words, a step of obtaining an effective word vector of each effective word by referring to the co-occurrence frequency in the sentence unit, and referring to the effective word vector To obtain a folder vector of each category using the document vector, so that a vector space that accurately reflects the topic can be formed, and classification can be performed properly. A simple document automatic classification system can be realized.

According to the classifying method of the sixth aspect, a step of holding a document to be classified, a step of obtaining a document vector of the document to be classified by referring to an effective word vector obtained by a learning method, Comparing the document vector obtained for the document with the folder vector of each category obtained by the learning method, and determining a category to which the document to be classified belongs according to the comparison result. It is possible to form a reflected vector space, and realize an automatic document classification system capable of appropriately performing classification.

According to the storage medium of the present invention, the program refers to the effective word in the learning document and calculates the number of effective words included in each sentence unit for each sentence unit by referring to the effective word. , A co-occurrence frequency within a sentence unit for obtaining a co-occurrence frequency within a sentence unit of each set of valid words with reference to the number of effective words, and an effective word vector of each effective word with reference to a co-occurrence frequency within a sentence unit An effective word vector calculation module, a document vector calculation module for obtaining a document vector by referring to an effective word vector for each of the learning document and the classification target document, and each category using the document vector obtained for the learning document. A folder vector calculation module for obtaining a folder vector of
A classification decision module for comparing the document vector obtained for the document to be classified with the folder vector of each category and determining a category to which the document to be classified belongs according to the comparison result; A space can be formed, and classification can be performed appropriately.

According to the storage medium of the present invention, the learning program includes a learning document holding module for holding a learning document, an effective word selecting module for selecting an effective word from the learning document, and a learning document. A module for calculating the number of valid words included in each sentence unit for each sentence unit with reference to the effective words, and a co-occurrence frequency for each set of valid words within the sentence unit for each set of valid words with reference to the number of effective words Co-occurrence frequency calculation module within sentence unit, effective word vector calculation module for obtaining effective word vector of each effective word by referring to co-occurrence frequency within sentence unit, and document vector calculation for obtaining document vector by referring to effective word vector Module and a folder vector module that obtains a folder vector for each category using document vectors. It is possible to form a space, it is possible to realize the automatic document classification system which can properly perform classification.

According to the storage medium of the ninth aspect, the classification program is obtained for the classification target document holding module for holding the classification target document and the classification target document by the storage medium learning program of the eighth aspect. A document vector calculation module for obtaining a document vector by referring to an effective word vector, and comparing a document vector obtained for a document to be classified with a folder vector of each category obtained by a storage medium learning program according to claim 8. And a classification determining module for determining a category to which the document to be classified belongs according to the comparison result, so that a vector space that accurately reflects the topic can be formed, and the automatic document classification can be performed appropriately. A classification system can be realized.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a functional configuration of an embodiment of an automatic document classification device of the present invention.

FIG. 2 is a block diagram showing a hardware configuration of the automatic document classification device shown in FIG. 1;

FIG. 3 is a flowchart showing a processing procedure of a learning phase in the automatic document classification device of FIG. 1;

FIG. 4 is a flowchart showing a processing procedure of a classification phase in the automatic document classification device of FIG. 1;

FIG. 5 is a block diagram showing a configuration of an embodiment of the learning device of the present invention.

FIG. 6 is a block diagram illustrating a configuration of an embodiment of a classification device according to the present invention.

FIG. 7 is a block diagram showing a configuration of a conventional automatic document classification device.

8 is a flowchart showing a processing procedure of a learning phase in the automatic document classification device of FIG. 7;

9 is a flowchart showing a processing procedure of a classification phase in the automatic document classification device of FIG. 7;

[Explanation of symbols]

 101, 801 learning document holding unit 102, 901 classification target document holding unit 103, 802 valid word selecting unit 104, 803 valid word holding unit 105, 804 effective word number calculation unit 106, 805 effective word number holding unit 107, paragraph 806 Intra-paragraph co-occurrence frequency calculation unit 108,807 In-paragraph co-occurrence frequency storage unit 109,808 Effective word vector calculation unit 110,809,902 Effective word vector storage unit 111,810,903 Document vector calculation unit 112,811,904 Document vector storage unit 113, 812 Folder vector calculation unit 114, 813, 905 Folder vector storage unit 115, 906 Classification determination unit 116, 907 Classification result storage unit 201 ROM 202 RAM 203 Central processing unit 204 Hard disk device

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Takaya Ueda 3-30-2 Shimomaruko, Ota-ku, Tokyo Inside Canon Inc. (72) Inventor Yuji Ikeda 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inside the corporation

Claims (9)

[Claims] An automatic document classification apparatus for classifying a document to be classified according to a user's intention using a learning document and an effective word selected from the learning document. Means for calculating the number of effective words included in each sentence unit by referring to each sentence unit; and a sentence unit for obtaining the co-occurrence frequency within each sentence of each set of effective words by referring to the number of effective words. Inner co-occurrence frequency calculating means, effective word vector calculating means for obtaining an effective word vector of each effective word with reference to the textual unit co-occurrence frequency, and for each of the learning document and the classification target document, A document vector calculating means for obtaining a document vector by referring to the effective word vector; and a folder vector for obtaining a folder vector of each category using the document vector obtained for the learning document. And a classification determining means for comparing a document vector obtained for the classification target document with a folder vector of each category, and determining a category to which the classification target document belongs according to the comparison result. Document automatic classification apparatus characterized by the above-mentioned. 2. Learning for obtaining a folder vector of each category, which is used as a reference for determining a category to which the classification target document belongs, which is used in an automatic document classification system for classifying the classification target document according to a user's intention. In the apparatus, learning document holding means for holding a learning document,
An effective word selecting means for selecting an effective word from the learning document; and a valid word number calculating means within a sentence unit for obtaining the number of effective words included in each sentence unit with reference to the effective word in the learning document. And a sentence-unit co-occurrence frequency calculating means for obtaining the intra-sentence co-occurrence frequency of each set of effective words by referring to the number of effective words, and the validity of each effective word by referring to the intra-sentence co-occurrence frequency. Effective vector calculating means for obtaining a word vector; document vector calculating means for obtaining a document vector by referring to the effective word vector; and folder vector calculating means for obtaining a folder vector of each category using the document vector. A learning device, characterized in that: 3. A classification device for determining a category to which the classification target document belongs, which is used together with the learning device according to claim 2 in an automatic document classification system for classifying the classification target document according to a user's intention. Classification target document holding means for holding a classification target document; document vector calculation means for obtaining a document vector by referring to the effective word vector obtained by the learning device for the classification target document; A classification vector determining unit that compares the document vector obtained with the folder vector of each category obtained by the learning device, and determines a category to which the classification target document belongs according to the comparison result. . 4. An automatic document classification method for classifying a document to be classified according to a user's intention using a learning document and an effective word selected from the learning document. Referring to each sentence unit to determine the number of effective words contained therein, and referring to the number of effective words to determine the co-occurrence frequency of each set of effective words within the sentence unit; Obtaining an effective word vector of each of the effective words by referring to the occurrence frequency; obtaining a document vector by referring to the effective word vector for each of the learning document and the classification target document; Obtaining a folder vector of each category using the document vector obtained for the document for use, and a document vector obtained for the classification target document and a folder vector of each category. And determining a category to which the classification target document belongs in accordance with the comparison result. 5. A learning method for determining a folder vector of each category, which is used as a reference for determining a category to which the classification target document belongs, which is used in an automatic document classification system for classifying the classification target document according to a user's intention. The method comprises the steps of: retaining a learning document; selecting valid words from the learning document; and referring to the valid words for the learning document, the number of valid words included in each sentence unit for each sentence unit. Determining the effective word vector of each set of effective words by referring to the number of effective words, and determining the co-occurrence frequency within the sentence unit of each set of effective words with reference to the number of effective words. And a step of obtaining a document vector by referring to the effective word vector; and a step of obtaining a folder vector of each category using the document vector. Learning method to do. 6. A classification method for determining a category to which the classification target document belongs, which is used together with the learning method according to claim 5 in an automatic document classification system for classifying the classification target document according to a user's intention. Holding a document to be classified, for the document to be classified, determining a document vector by referring to an effective word vector obtained by the learning method, and a document vector obtained for the document to be classified and the learning method. And comparing the folder vector of each category obtained in the step (a) with each other, and determining a category to which the document to be classified belongs according to the comparison result. 7. A storage medium storing a program for constructing an automatic document classification apparatus for classifying a document to be classified according to a user's intention using a learning document and an effective word selected from the learning document. In the program, the effective word number calculation module in the sentence unit to find the number of effective words included in each sentence unit for each sentence unit by referring to the effective words for the learning document, A sentence unit co-occurrence frequency calculation module for obtaining a sentence unit co-occurrence frequency of a set of effective words, and an effective word vector calculation module for obtaining an effective word vector of each effective word by referring to the sentence unit co-occurrence frequency; A document vector calculation module for obtaining a document vector by referring to the effective word vector for each of the learning document and the classification target document; And a folder vector calculation module that obtains a folder vector of each category using the document vector obtained for the above, and compares the document vector obtained for the classification target document with the folder vector of each category, and according to the comparison result, A classification determination module for determining a category to which the classification target document belongs; 8. Learning for obtaining a folder vector of each category, which is used as a reference for determining a category to which the classification target document belongs, which is used in an automatic document classification system for classifying the classification target document according to a user's intention. In a storage medium storing a learning program for constructing an apparatus, the learning program includes a learning document holding module for holding a learning document, an effective word selecting module for selecting an effective word from the learning document, A module for calculating the number of effective words in a sentence unit that determines the number of effective words contained in each sentence unit with reference to the effective words for the learning document, and a sentence unit for each set of effective words with reference to the number of effective words An intra-unit co-occurrence frequency calculating module for obtaining an intra-unit co-occurrence frequency, and an effective word vector of each of the effective words by referring to the intra-sentence unit co-occurrence frequency. , A document vector calculation module for obtaining a document vector by referring to the effective word vector, and a folder vector module for obtaining a folder vector of each category using the document vector. Storage medium. 9. A classification apparatus for determining a category to which the classification target document belongs, which is used together with the storage medium according to claim 8 in an automatic document classification system for classifying the classification target document according to a user's intention. 9. A storage medium storing a classification program for storing a classification target document, wherein the classification program stores a classification target document holding module for holding the classification target document and the classification target document.
9. A storage medium learning program according to claim 8, wherein a document vector calculation module for obtaining a document vector by referring to an effective word vector obtained by the storage medium learning program described above, and a document vector obtained for the classification target document. And a classification determination module that compares the folder vector of each category obtained by the above with the folder vector and determines the category to which the classification target document belongs according to the comparison result.