JP6059486B2 - Teacher data verification device, teacher data creation device, image classification device, teacher data verification method, teacher data creation method, and image classification method - Google Patents

Description

  The present invention relates to a technique for verifying teacher data used for learning of a classifier that classifies images, a technique for creating teacher data using the technique, and a technique for classifying images.

  In the manufacture of semiconductor substrates, glass substrates, printed wiring boards, and the like, appearance inspection is performed using an optical microscope, a scanning electron microscope, or the like in order to inspect defects such as foreign matters, scratches, and etching defects. In addition, the cause of the defect is specified by performing detailed analysis on the defect detected in such an inspection process, and countermeasures against the defect are taken. In recent years, as the pattern on the substrate becomes more complex and finer, the types and quantities of detected defects tend to increase, and automatic classification that automatically classifies defects detected in the inspection process is also used. Automatic classification enables defect analysis to be performed quickly and efficiently.

  In automatic classification, a classifier using a neural network, a decision tree, discriminant analysis, or the like is used. In order for the classifier to perform automatic classification, it is necessary to prepare the teacher data including a signal indicating the defect image and its category (that is, the type of the defect image) and to learn the classifier. Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique for creating high-quality teacher data by determining whether or not a category previously assigned to a defect image is a category to which the defect image belongs. Specifically, a feature amount range including a representative value of the feature amount is set for the type based on the distribution of the feature amount of each type of the plurality of defect images belonging to each category, A process of voting for a category that includes a type of feature quantity in the type of feature quantity range is performed for all types of feature quantities. Then, when the category with the largest number of votes out of a plurality of categories is different from the category to which the defect image belongs, this is output.

  In Patent Document 2, a plurality of feature amounts are calculated for a plurality of teaching defect images, and a classification parameter indicated by a discriminant function between categories in a feature space based on the feature amount is calculated for each teaching defect. Diagnose the statistical significance of teaching data showing the correspondence between teaching defects and the category corresponding to the type of defect, display teaching defect images on the screen that may reduce performance, and display teaching data A technique for correcting is disclosed.

JP 2010-91401 A JP 11-344450 A

  By the way, the category determination by statistical processing in Patent Document 1 can be considered that the distribution of each type of feature amount of a plurality of defect images (teacher images) belonging to each category follows a normal distribution (or follows a normal distribution). It is particularly effective in the case where the distribution of the feature quantity shows multimodality, and the result (category) in Patent Document 1 does not agree with the operator's judgment when the distribution does not follow the normal distribution, or The operator may not be able to reasonably interpret the result. In this case, there is a possibility that the creation of highly reliable teacher data may be hindered. In the method of Patent Document 2, it is necessary to calculate a discriminant function between categories by complicated processing when the category of the teacher image is presented to the operator. Therefore, there is a need for a new technique that can easily identify appropriate category candidates based on a plurality of types of feature amounts for each teacher image.

  The present invention has been made in view of the above problems, and an object thereof is to easily specify an appropriate category candidate based on a plurality of types of feature amounts for each teacher image.

The invention according to claim 1 is a teacher data verification device for verifying teacher data used for learning of a classifier that classifies images, and a plurality of teacher images each assigned to one of a plurality of categories , And a storage unit for storing teacher data indicating categories of the plurality of teacher images, and a plurality of types of feature amounts, each representing a validity when each teacher image belongs to each of the plurality of categories. In each of the plurality of categories in the evaluation value acquisition unit that acquires an evaluation value, and for each of the plurality of categories, a representative value of the evaluation value in the plurality of types of feature values is obtained, and a plurality of representative values in the plurality of categories are obtained. A category candidate specifying unit for specifying a category determined to belong to each teacher image as a category candidate based on the teacher data; The teacher image with categories and the category candidate is different as a category difference image, and a display control unit for displaying the category difference picture image on the display unit together with the category candidates, the evaluation value acquiring unit, each of the feature With respect to the type, a histogram of the feature amount of the teacher image allocated to each of the plurality of categories is generated, and the attention is included in the plurality of histograms for the plurality of categories, with a section including the feature amount of each teacher image as the attention interval. Based on the frequency of the section, the evaluation value for each of the plurality of categories of each teacher image is acquired, and the evaluation value acquiring unit and the category candidate specifying unit are processed while removing the category difference image from the teacher data Depending on the number of times the process is repeated until it is identified as the category difference image The plurality of teacher images Ru is divided into a plurality of groups.

Invention of Claim 2 is the teacher data verification apparatus of Claim 1 , Comprising: The said evaluation value acquisition part is a some area adjacent to both sides of the said attention area and the said attention area in these histograms The evaluation value for each of the plurality of categories of each teacher image is acquired based on the frequency of the above.

The invention described in claim 3 is the teacher data verification apparatus according to claim 1 or 2, the plurality of histograms, the frequency of the section of interest in the histogram for one of the categories is equal to or less than a predetermined number, When the frequency of the attention section in the histogram for each of all other categories is 0, the evaluation value acquisition unit does not acquire the evaluation value.

The invention described in claim 4 is the teacher data verification apparatus according to any one of claims 1 to 3, wherein the display control unit, before the hear categories difference picture image, category showing the teacher data and It is displayed on the display unit together with the category candidates.

The invention according to claim 5 is a teacher data creation device for creating teacher data used for learning of a classifier that classifies images, and the teacher data verification device according to any one of claims 1 to 4 , and an input unit for accepting an input of the final categories against before asked categorical difference images.

The invention according to claim 6 is an image classification device for classifying images, and learning is performed using the teacher data creation device according to claim 5 and the teacher data created by the teacher data creation device. And a classifier for classifying the images.

The invention according to claim 7 is a teacher data verification method for verifying teacher data used for learning of a classifier that classifies images, and a) a plurality of each of which is assigned to one of a plurality of categories A step of preparing teacher images and teacher data indicating the categories of the plurality of teacher images; and b) appropriateness when each teacher image belongs to each of the plurality of categories with respect to each of a plurality of types of feature values. A step of obtaining an evaluation value indicating sex; c) a step of obtaining representative values of evaluation values of the plurality of types of feature amounts for each of the plurality of categories in each of the teacher images; and d) the plurality of categories. Identifying a category to which each of the teacher images should belong as a category candidate based on a plurality of representative values in e), e) The teacher image and to categories and the category candidate is different as a category difference image, and a step of displaying a category different picture images on the display unit together with the category candidates, in the b) step, for each type of feature quantity , A histogram of the feature amount of the teacher image allocated to each of the plurality of categories is generated, and a section including the feature amount of each teacher image is set as the attention section, and the attention section of the plurality of histograms for the plurality of categories Based on the frequency, the evaluation value for each of the plurality of categories of each teacher image is acquired, and the steps b) to d) are repeated a plurality of times while removing the category difference image from the teacher data, and the category Depending on the number of times the process is repeated until the images are identified as different images, the plurality of teacher images Ru is divided into groups of a few.

The invention according to claim 8 is the teacher data verification method according to claim 7 , wherein in the step b), the attention section in the plurality of histograms and a plurality of sections adjacent to both sides of the attention section. Based on the frequency, the evaluation value for each of the plurality of categories of each teacher image is acquired.

The invention according to claim 9 is the teacher data verification method according to claim 7 or 8 , wherein, in the step b), the frequency of the attention section in the histogram for one category among the plurality of histograms. The evaluation value is not acquired when the frequency is equal to or less than the predetermined number and the frequency of the attention section in the histogram for each of all other categories is zero.

The invention described in claim 10 is the teacher data verification method according to any one of claims 7 to 9, in the step e), before hear categorical difference picture image, category indicates the teacher data and It is displayed on the display unit together with the category candidates.

The invention described in claim 11 is a teacher data generation method for generating teacher data used for learning of a classifier that classifies images, and the teacher data verification method according to any one of claims 7 to 10 , and a step of determining the final category against before asked categorical difference images.

The invention according to claim 12 is an image classification method for classifying images, the step of learning a classifier using teacher data created by the teacher data creation method according to claim 11 , and the classifier And a step of classifying the images.

  According to the present invention, it is possible to easily identify appropriate category candidates based on a plurality of types of feature amounts for each teacher image. Further, by displaying the category difference image together with the category candidates on the display unit, highly reliable teacher data can be easily created.

It is a figure which shows the structure of an image classification device. It is a figure which shows the flow of a classification | category of a defect image. It is a figure which shows the structure of a host computer. It is a figure which shows the function structure of a host computer. It is a figure which shows the function structure of a teacher data creation part. It is a figure which shows the flow of the process which produces teacher data and learns a classifier. It is a figure which shows the some teacher image displayed on a display. It is a figure which shows several histograms with respect to several categories. It is a figure which shows the some teacher image displayed on a display. It is a figure for demonstrating the degree of heterogeneity. It is a figure which shows a some teacher image. It is a figure which shows a some teacher image.

  FIG. 1 is a diagram showing a schematic configuration of an image classification apparatus 1 according to an embodiment of the present invention. In the image classification apparatus 1, a defect image indicating a defect on a semiconductor substrate 9 (hereinafter simply referred to as “substrate 9”) is acquired, and the defect image is classified. The image classification device 1 picks up a defect into a category to which the defect should belong when the defect is detected based on the image data from the image pickup device 2 and the image pickup device 2 that picks up the inspection target area on the substrate 9. It has an inspection / classification device 4 for automatic classification, and a host computer 5 that controls the overall operation of the image classification device 1 and generates a classifier 421 used for defect classification in the inspection / classification device 4. The types (categories) of defects present on the substrate 9 are, for example, defects, protrusions, disconnections, shorts, and foreign matters. In addition, the imaging device 2 is incorporated in the production line of the substrate 9, and the image classification device 1 is a so-called inline system. The image classification device 1 can also be regarded as a device in which a function of automatic defect classification is added to a defect inspection device.

  The imaging device 2 captures an inspection target area on the substrate 9 to acquire image data, a stage 22 that holds the substrate 9, and a stage that moves the stage 22 relative to the imaging unit 21. A drive unit 23 is included. The imaging unit 21 electrically outputs an illumination unit 211 that emits illumination light, an optical system 212 that guides the illumination light to the substrate 9 and receives light from the substrate 9, and an image of the substrate 9 formed by the optical system 212. An imaging device 213 that converts the signal into a signal is included. The stage driving unit 23 includes a ball screw, a guide rail, a motor, and the like, and the inspection area on the substrate 9 is imaged by the host computer 5 controlling the stage driving unit 23 and the imaging unit 21.

  The inspection / classification apparatus 4 includes a defect detection unit 41 that detects defects while processing image data of an inspection target area, and an automatic defect classification unit 42 that classifies defect images. The defect detection unit 41 has a dedicated electric circuit for processing image data of the inspection target area at high speed, and performs defect inspection of the inspection target area by comparing the captured image with a reference image having no defect or by image processing. Do. The automatic defect classification unit 42 includes a CPU that performs various arithmetic processes, a memory that stores various information, and the like, and classifies defects (that is, defect images) using a classifier 421 that uses a neural network, a decision tree, discriminant analysis, and the like. Classification).

  FIG. 2 is a diagram showing a flow of defect image classification by the image classification apparatus 1. First, when the imaging device 2 shown in FIG. 1 images the substrate 9, the defect detection unit 41 of the inspection / classification device 4 acquires image data (step S11). Next, when the defect detection unit 41 performs the defect inspection of the inspection target region and a defect is detected (step S12), the image of the defective portion (that is, the defect image) is transmitted to the automatic defect classification unit 42. The The automatic defect classification unit 42 calculates a plurality of types of feature values of the defect image (step S13), and the feature values of the defect image are input to the classifier 421 of the automatic defect classification unit 42 and the classification result is output. That is, the defect image is classified into one of a plurality of categories by the classifier 421 (step S14). In the image classification device 1, the feature amount is calculated in real time every time a defect is detected by the defect detection unit 41, and automatic classification of a large number of defect images is performed at high speed.

  Next, classifier learning by the host computer 5 will be described. FIG. 3 is a diagram showing the configuration of the host computer 5. The host computer 5 has a general computer system configuration in which a CPU 51 that performs various arithmetic processes, a ROM 52 that stores basic programs, and a RAM 53 that stores various information are connected to a bus line. The bus line further includes a fixed disk 54 that stores information, a display 55 that is a display unit that displays various types of information such as images, a keyboard 56a and a mouse 56b that accept input from an operator (hereinafter referred to as "input unit 56"). A signal is transmitted / received between the reading device 57 for reading information from the computer-readable recording medium 8 such as an optical disk, a magnetic disk, a magneto-optical disk, and the other configuration of the image classification device 1. The communication unit 58 to be connected is appropriately connected through an interface (I / F) or the like.

  In the host computer 5, the program 80 is read in advance from the recording medium 8 via the reading device 57 and stored in the fixed disk 54. Further, the program 80 is copied to the RAM 53 and the CPU 51 follows the program in the RAM 53. Arithmetic processing is executed.

  FIG. 4 is a block diagram showing a functional configuration for learning the classifier realized by the CPU 51, ROM 52, RAM 53, fixed disk 54, etc. of the host computer 5, and also shows the inspection / classification device 4. The host computer 5 includes a teacher data creating unit 61 that creates teacher data used for learning the classifier, and a learning unit 62 that learns the classifier using the teacher data.

  The teacher data includes a teacher image data that is a defect image, a feature amount of the teacher image, and a teaching signal that is information indicating the category of the defect. As the feature amount of the teacher image, for example, the defect area, the brightness average, Peripheral length, flatness, inclination of major axis when defect is approximated to ellipse, and the like are employed. In the learning unit 62, the feature amount of the teacher image read from the teacher data is input to a classifier (not shown) in the host computer 5, and the output of the classifier is the same as the teaching signal indicating the defect category. Learning is performed, and the learning result, that is, the learned classifier 421 (more precisely, information indicating the structure of the classifier 421 and the value of the variable) is transferred to the automatic defect classification unit 42.

  FIG. 5 is a block diagram showing a functional configuration of the teacher data creation unit 61 of the host computer 5, and also shows a learning unit 62. The teacher data creation unit 61 includes a data calculation unit 610, a display 55, and an input unit 56. The data calculation unit 610 includes an evaluation value acquisition unit 611, a category candidate identification unit 612, a display control unit 613, a storage unit 614, and a feature amount calculation unit 615. Details of the processing of the data calculation unit 610 will be described later. Note that the functions of the data calculation unit 610 and the learning unit 62 may be constructed by a dedicated electric circuit, or a dedicated electric circuit may be partially used.

  FIG. 6 is a diagram showing a flow of processing for creating teacher data and learning a classifier. In the teacher data creation unit 61, first, data of a large number of teacher images (for example, thousands of teacher images), which are defect images for teacher data creation, are stored in the storage unit 614 of the teacher data creation unit 61 shown in FIG. To be prepared (step S21). The teacher image may be acquired using the imaging device 2 and the defect detection unit 41 illustrated in FIG. 1 or may be separately prepared.

  The feature amount calculation unit 615 calculates a plurality of types (for example, 100 to 200 types) of feature amounts of all the teacher images (step S22). The calculated feature amount is stored in the storage unit 614. Further, each teacher image is displayed on the display 55 and a display prompting the user to input a category of the teacher image is displayed, and the input of the category by the operator is accepted by the input unit 56 (step S23). In the following description, the category input by the operator for each teacher image in the process of step S23 is referred to as “initial category”. Note that the initial category of each teacher image may be automatically determined by a predetermined algorithm.

  FIG. 7 is a diagram illustrating a plurality of teacher images displayed on the display 55. A category display area 72 is provided below each of the teacher images 71a, 71b, 71c, and 71d. In the teacher images 71a to 71d in FIG. ”,“ Middle protrusion ”, and“ middle protrusion ”. In this way, a plurality of teacher images each assigned to one of a plurality of categories and teacher data 710 indicating the categories of the plurality of teacher images are prepared and stored in the storage unit 614. In the present embodiment, a plurality of feature amounts in each teacher image is also included in the teacher data 710. Also, a number is assigned to each of a plurality of types of categories, and the value of the categorical variable associated with the teacher image is changed to the number of the initial category by determining the initial category for each teacher image.

  When the teacher data 710 is prepared, the evaluation value acquisition unit 611 generates a histogram of the feature amount of the teacher image assigned to each of a plurality of categories for each type of feature amount. FIG. 8 is a diagram illustrating a plurality of histograms for a plurality of categories in one type of feature amount. In the example of FIG. 8, 5348 teacher images are assigned to any of the four types of categories 1 to 4, the number 1 category includes 1578 teacher images, and the number 2 category is 2849. The number 3 category includes 688 teacher images, and the number 4 category includes 133 teacher images. Further, in FIG. 8, the feature amount of each teacher image is normalized and then quantized into 100 sections, and the frequency in each section is shown. In FIG. 8, symbols H1, H2, H3, and H4 are attached to the histograms of categories numbered 1 to 4, respectively. It should be noted that the feature value histogram in the evaluation value acquisition unit 611 may be one that shows a frequency distribution, and the table showing the frequency distribution is substantially equivalent to the histogram.

The evaluation value acquisition unit 611 acquires an evaluation value indicating the validity (may be regarded as a probability) when each teacher image belongs to each of a plurality of categories for each type of feature amount (step S24). ). For example, a certain teacher image is an attention teacher image, an interval including the feature amount of the attention teacher image regarding one type of feature amount is an attention interval, and there are x ₁ teacher images of category No. 1 in the attention interval and number 2. category teacher image are _{two x,} a teacher image of the number 3 category _{3 x,} a teacher image of the number 4 category is to be present _{four x} (i.e., interest in the histogram H1, H2, H3, H4 _{1 x} frequency intervals, respectively, _{two x, 3} pieces _x, a _{four x.).} It should be noted that, even if in the case where the initial category of the attention the teacher image is the number 2 categories, one of the _two x is the target teacher image. In this case, the evaluation value P _n indicating the validity when the teacher image of interest belongs to the category of number n (here, n is any one of 1 to 4) is ((x _n / (x ₁ + x ₂ + X ₃ + x ₄ )) × 100) [%].

Specifically, the principal feature of the teacher image is included in the section denoted by a reference sign A in in FIG. 8, the histogram H1, H2, frequently each _{one a} of the target interval A in H3, ₂ pieces _{a, 3} pieces _a (The frequency in the histogram H4 is 0), the evaluation value P ₁ indicating the validity when the teacher image of interest belongs to the category of number ₁ is ((a ₁ / (a ₁ + a ₂ + A ₃ )) × 100) [%]. Further, the evaluation value P ₂ indicating the validity when the attention teacher image belongs to the category of number 2 is ((a ₂ / (a ₁ + a ₂ + a ₃ )) × 100) [%], and the number 3 The evaluation value P ₃ indicating the validity when it belongs to the category is ((a ₃ / (a ₁ + a ₂ + a ₃ )) × 100) [%]. Note that the evaluation value P4 indicating the validity when the teacher image of interest belongs to the category of number ₄ is 0%. In addition, in FIG. 8, the frequency of the said point is shown with the code | symbol attached | subjected to a black point.

Further, the feature quantity of the target teacher image is included in the section denoted by a reference sign B in in Fig. 8, the histogram H1, H2, frequently _{one b} each section of interest B in H3, ₂ pieces _b, a _{three b} (The frequency in the histogram H4 is 0.), the evaluation value P ₁ indicating the validity when the teacher image of interest belongs to the category of number ₁ is ((b ₁ / (b ₁ + b ₂ + b ₃ )) × 100) [%]. In addition, the evaluation value P ₂ indicating the validity when the teacher image of interest belongs to the category of number 2 is ((b ₂ / (b ₁ + b ₂ + b ₃ )) × 100) [%], and the number 3 The evaluation value P ₃ indicating the validity in the case of belonging to the category is ((b ₃ / (b ₁ + b ₂ + b ₃ )) × 100) [%]. Note that the evaluation value P4 indicating the validity when the teacher image of interest belongs to the category of number ₄ is 0%.

Further, the feature quantity of the attention teacher image is included in the section denoted by reference character C in FIG. 8, and the frequencies of the attention section C in the histograms H3 and H4 are c ₃ and c ₄ respectively (in the histograms H1 and H2). The frequency is 0.) In this case, the evaluation value P ₃ indicating the validity when the teacher image of interest belongs to the category of number ₃ is ((c ₃ / (c ₃ + c ₄ )) × 100) [ %]. Further, the evaluation value P ₄ indicating the validity when the focused teacher image belongs to the category of number 4 is ((c ₄ / (c ₃ + c ₄ )) × 100) [%]. Incidentally, the subject supervisor image evaluation value indicating the validity of the case of as belonging to the category number 1 P _1, and the evaluation value P ₂ that indicates the validity in the case of as belonging to the No. 2 categories are both 0% is there.

  As described above, the evaluation value acquisition unit 611 uses a section including the feature amount of each teacher image as an attention section, and based on the frequency of the attention section in a plurality of histograms for a plurality of categories, Evaluation values for each are obtained. Thus, in each teacher image, for one type of feature value, a combination of four evaluation values in four categories (for example, “number 1 category is 25%, number 2 category is 72%, number 3 is Is obtained as an evaluation value group, and in the case where there are N types of feature values, in principle, N evaluation value groups are acquired. The The above process is performed for all teacher images, and (in principle) N evaluation value groups are acquired for all teacher images. In the above processing, it is possible to acquire an evaluation value corresponding to the unevenness of the histogram even when the histogram has multimodality. Note that an evaluation value indicating validity when each teacher image belongs to each of a plurality of categories may be obtained by a calculation other than the above.

  On the other hand, for example, in the region on the right side of the horizontal axis in FIG. 8, only the histogram H4 of category No. 4 is distributed. In this case, even when the frequency of the histogram H4 in the attention interval is extremely low as 1 or 2, the evaluation value of the category of number 4 is always 100%, and the evaluation value of the category of number 4 obtained in the processing described later is This will unduly push up the representative value. Therefore, among the plurality of histograms of a plurality of categories related to one type of feature amount, the frequency of the attention section in the histogram for one category is equal to or less than a predetermined number (for example, 1 or 2), and each of all other categories When the frequency of the attention section in the histogram is 0, no evaluation value is acquired for the type of feature amount.

  Subsequently, the category candidate specifying unit 612 obtains representative values of evaluation values for a plurality of types of feature values (more precisely, all types for which evaluation values have been acquired) for each category in each teacher image (steps). S25). Here, the representative value of the evaluation value is a value indicating the vicinity of the center of the range in which the evaluation value is distributed, such as an average value or a median value. The representative value of the evaluation value of each category in each teacher image is a value indicating validity in consideration of a plurality (all) of feature amounts when the teacher image belongs to the category.

  When a plurality of representative values in a plurality of categories are obtained for each teacher image, a category determined to belong to the teacher image based on the plurality of representative values is specified as a category candidate (step S26). . Specifically, in each teacher image, a category in which the maximum value among a plurality of representative values is obtained is specified as a category candidate. However, when the maximum value is less than a predetermined threshold (for example, 50%), it is preferable that “not classified” is given as a category candidate to the teacher image.

  In the display control unit 613, a plurality of category-difference images are displayed on the display 55 using a teacher image in which the initial category indicated by the teacher data 710 is different from the category candidate as a category-difference image (step S27). FIG. 9 is a diagram illustrating a plurality of teacher images displayed on the display 55. In the present embodiment, teacher images 71b and 71c other than the category difference images 71a and 71d are also displayed on the display 55. In the category display area 72 for the teacher images 71a to 71d in FIG. 9, the category candidates (names thereof) are written as “Classification not possible”, “Medium defect”, “Medium protrusion”, and “Classification not possible”, respectively. In the category display area 72, both initial categories and category candidates are displayed. Furthermore, the background color of the category display area 72 for the category difference images 71a and 71d is different from the background color of the category display area 72 for the other teacher images 71b and 71c that are not category difference images, and the category difference images 71a and 71d. Is emphasized.

  Subsequently, the operator determines the final category by referring to the category difference image while considering the initial category and the category candidate for each category difference image (step S28). Then, the input of the final category from the operator is accepted by the input unit 56, and the category (value of the categorical variable) associated with the teacher image that is the category difference image is changed or maintained. In this way, the teacher data 710 is updated. Of course, a final category may be input for a teacher image other than the category difference image, but the current category (initial category) may be maintained for a teacher image in which the initial category and the category candidate match. preferable.

  When the final category of each teacher image is determined, all the teacher image data and updated teacher data 710 including information on the feature amount and the category are transferred from the storage unit 614 to the learning unit 62. The learning unit 62 learns the classifier using the teacher data 710 (step S29). That is, the values of variables constituting the classifier (see FIG. 4) are determined, and the structure is determined to generate the classifier 421. Thereby, the process of FIG. 6 for creating the teacher data and learning the classifier is completed.

  Here, in the process of FIG. 6, category candidates with high validity are presented for each category-difference image in consideration of all types of feature values. Therefore, the teacher data for which the final category has been determined is less likely to include teacher images associated with inconsistent categories in the feature amount space. That is, the reliability of teacher data is improved. Therefore, a learning result (classifier) using such teacher data can be expected to have high classification performance.

  In addition, when the process described later is repeated a plurality of times while removing the category difference image from the original teacher data, all the teacher images assigned to a certain category as the initial category are specified as the category difference image. It can be divided into a plurality of groups according to the number of times the process is repeated. Each group can be regarded as a collection of teacher images having approximately the same degree of heterogeneity for the category, and the plurality of groups have different degrees of heterogeneity. FIG. 10 abstractly shows a plurality of groups having different degrees of heterogeneity. For example, the teacher image specified as the category difference image in the first process is included in the outermost annular region Z1. It is perceived as “a heterogeneous thing”. The teacher images specified as the category difference images in the second to fourth processes are included in the annular regions Z2, Z3, and Z4 where the degree of heterogeneity gradually decreases. A teacher image that is not specified as a category-difference image in the fourth process is included in the central circular area Z0, and is captured as a typical teacher image of the category.

  Next, the superiority of the above process for specifying category candidates will be described. The image classification apparatus 1 can repeat the processing of steps S24 to S26 a plurality of times while removing the category difference image from the original teacher data in the process of FIG. 6, and here, while removing the category difference image from the teacher data. A result obtained when the processes in steps S24 to S26 are repeated four times (hereinafter, referred to as “result in this process example”) is compared with a result in the process of the comparative example. Here, the processing of the comparative example is the same as the method of Japanese Patent Application Laid-Open No. 2010-91401 (Patent Document 1), and based on the distribution of the feature amounts of each type of a plurality of teacher images belonging to each category, A feature amount range including a representative value of the feature amount is set for the type, and the process of voting for a category including one type of feature amount of each teacher image in the feature amount range of the type Done about types. The category having the largest number of votes among a plurality of categories is set as a category candidate. When the difference in the number of votes between the category with the largest number of votes and the category with the second largest number of votes is equal to or less than a predetermined value, such a teacher image is determined to be an inappropriate teacher image.

  FIG. 11 is a diagram showing a result in this processing example, and FIG. 12 is a diagram showing a result in a comparative example. 11 and 12, the same plurality of teacher images 71a to 71d are shown together with category candidates. In addition, in the category display area 72 for the category difference image shown in FIGS. 11 and 12, a parallel oblique line with a narrower width is added as the number of repetitions of the process until the category difference image is specified (actually, the category display area 72 is displayed). The background color of the display area 72 is different.) Actually, in the result of the present processing example, the teacher images 71a and 71d shown in FIG. 11 are identified as category difference images in the first processing of steps S24 to S26, and the teacher image 71c is the fourth time of steps S24 to S26. In this process, the teacher image 71b is not specified as a category difference image in any of the first to fourth processes (that is, the initial category is correct). In the result of the comparative example, the teacher image 71b shown in FIG. 12 is specified as a category difference image in the first comparative example process, and the teacher image 71c is specified as a category difference image in the second comparative example process. The teacher images 71a and 71d are not specified as category difference images in any of the first to fourth comparative examples.

  In the process of the comparative example, as shown in FIG. 12, “large deficit” is determined as a category candidate in the same manner as the initial category with respect to the teacher image 71a indicating the pattern disconnection. That is, nothing is pointed out that both “large defect” and “disconnection”, which are considered to be greatly different in the feature space, are treated as “large defect”. In contrast, in the present processing example, as shown in FIG. 11, “not classified” that is different from the initial category “large deficit” is determined as a category candidate for the teacher image 71a indicating the disconnection of the pattern. It has been pointed out that a large defect and a disconnection are mixed. Further, in this processing example, “not classified” that is different from the initial category “medium protrusion” is determined as the category candidate for the teacher image 71d, and the teacher image 71d is either “medium protrusion” or “large protrusion”. It is pointed out that this is confusing. From the above, it can be understood that the result in this processing example is more in line with the human sense than the result in the comparative example.

  As described above, in the image classification device 1, a plurality of teacher images and teacher data indicating categories of a plurality of teacher images are stored in the storage unit 614, and each teacher is associated with a plurality of types of feature values. The evaluation value acquisition unit 611 acquires an evaluation value indicating validity when the image belongs to each of a plurality of categories. Then, the category candidate specifying unit 612 obtains, for each teacher image, representative values of evaluation values for a plurality of types of feature amounts for each category, and further, each teacher image belongs based on a plurality of representative values in a plurality of categories. A category determined to be a power is specified as a category candidate. Thereby, it is possible to easily identify appropriate category candidates based on a plurality of types (all types) of feature amounts for each teacher image by a simple calculation.

  Also, a category difference image, which is a teacher image in which the initial category and the category candidate are different, is displayed on the display 55 together with the category candidate by the display control unit 613. The operator can easily create highly reliable teacher data by referring to the category difference image and the category candidate on the display 55 and inputting the final category. As a result, the image classification device 1 can classify defect images with high accuracy using the classifier 421 that performs learning using the teacher data.

  Further, by displaying both the initial category and the category candidate side by side on the display 55 for each category difference image, the operator can easily interpret the category difference image. Furthermore, in the image classification device 1, the frequency of the attention section in the histogram for one category is (1 or more) a predetermined number or less among the plurality of histograms for each type of feature amount, and the histogram for each of all other categories. When the frequency of the section of interest in is 0, the evaluation value acquisition unit 611 does not acquire the type of evaluation value of the feature amount. As a result, the type of inappropriate feature quantity that unreasonably pushes up the representative value of the evaluation value without performing a process of eliminating the type of unnecessary feature quantity by analysis of variance (so-called ANOVA (analysis of variance)) or the like ( Evaluation value) can be eliminated.

By the way, when the number of teacher images is small, many teacher images exist in isolation in any section (become isolated in the feature space), and the histogram H4 of category No. 4 in FIG. Thus, the frequency is low in most of the sections and shows a flat distribution. In such a case, in the histogram of the category of number n, evaluation is performed by setting a value that includes not only the frequency of the target section in the target teacher image but also the frequencies of a plurality of sections adjacent to both sides of the target section as x _n . The value P _n is preferably determined by ((x _n / (x ₁ + x ₂ + x ₃ + x ₄ )) × 100) [%]. In this way, the evaluation value acquisition unit 611 acquires evaluation values for each of the plurality of categories of each teacher image based on the attention section in the plurality of histograms and the frequencies of the plurality of sections adjacent to both sides of the attention section. This makes it possible to appropriately determine category candidates even when the number of teacher images is small.

Various modifications can be made to the above apparatus .

  In the above embodiment, the superiority of the process of FIG. 6 for identifying category candidates is described by referring to the result of repeating the processes of steps S24 to S26 of FIG. 6 a plurality of times while removing the category difference image from the teacher data. However, final teacher data may be generated by repeating the processes in steps S24 to S26 a plurality of times while removing the category difference image from the teacher data. As described with reference to FIG. 10, such teacher data can be regarded as including only typical teacher images of each category.

  The display 55 does not necessarily display a plurality of category difference images, and only one category difference image may be displayed. In addition, only the category candidates may be displayed together with the category difference image. As described above, in the process of step S27 in FIG. 6, each of at least one category difference image is displayed on the display 55 together with the category candidate.

  In the image classification device 1, the feature amounts of a large number of teacher images used for teacher data may be calculated by the inspection / classification device 4. The feature amount may be omitted from the teacher data, and the learning unit 62 may obtain the feature amount based on the teacher image data.

  In the above-described embodiment, the image classification apparatus 1 in which the function of automatic defect classification is added to the defect inspection apparatus has been described. However, an observation apparatus that observes defects on the substrate detected by other defect inspection apparatuses (both review apparatuses) The teacher data creation unit 61 may be used in a device in which a function of automatic defect classification is added to (referred to as an image classification device). In such an image classification device, an image with a higher resolution than that of the image pickup device 2 in FIG. 1 is acquired in order to analyze defects more highly.

  In the image classification apparatus 1 of FIG. 1, inspection of a glass substrate (for example, a glass substrate for a flat display device), a printed wiring board, or a mask substrate used for exposure of the substrate may be performed instead of the semiconductor substrate.

  Further, the image classification device 1 may be used for the purpose of classifying a cell image obtained by imaging cells in a predetermined liquid such as blood or a culture solution. As described above, the image classification device 1 can be used to classify images showing various objects. Furthermore, in the image classification device 1, in addition to images captured with visible light, images captured with an electron beam, an X-ray, or the like may be classified.

  In the above embodiment, the teacher data verification device that verifies teacher data by specifying category candidates for each teacher image includes the storage unit 614 provided in the teacher data creation unit 61, the evaluation value acquisition unit 611, the category candidates. Although implemented by the specifying unit 612 and the display control unit 613, the teacher data verification device may be implemented as a separate device from the teacher data creation unit 61. In addition, the teacher data creation unit 61 which is a teacher data creation device may be realized as a separate device from the image classification device 1.

  The configurations in the above-described embodiments and modifications may be combined as appropriate as long as they do not contradict each other.

DESCRIPTION OF SYMBOLS 1 Image classification device 55 Display 56 Input part 61 Teacher data creation part 71a-71d Teacher image 421 Classifier 611 Evaluation value acquisition part 612 Category candidate specific part 613 Display control part 614 Storage part 710 Teacher data H1-H4 Histogram S14, S21- S29 step

Claims (12)

A teacher data verification device for verifying teacher data used for learning of a classifier that classifies images,
A plurality of teacher images each assigned to one of a plurality of categories, and a storage unit that stores teacher data indicating categories of the plurality of teacher images;
An evaluation value acquisition unit that acquires an evaluation value indicating validity when each teacher image belongs to each of the plurality of categories for each of a plurality of types of feature amounts;
In each of the teacher images, for each of the plurality of categories, a representative value of the evaluation value in the plurality of types of feature quantities is obtained, and each teacher image should belong based on the plurality of representative values in the plurality of categories. A category candidate identifying unit that identifies a category to be determined as a category candidate;
The teacher image and the category showing the teacher data and the category candidate is different as a category different from the image, a display control unit for displaying the category difference picture image on the display unit together with the category candidates,
Equipped with a,
The evaluation value acquisition unit generates a histogram of the feature amount of the teacher image assigned to each of the plurality of categories for each type of feature amount, and sets a section including the feature amount of each teacher image as an attention interval. Based on the frequency of the attention interval in a plurality of histograms for the plurality of categories, the evaluation values for each of the plurality of categories of the teacher images,
The processing of the evaluation value acquisition unit and the category candidate specifying unit is repeated a plurality of times while removing the category difference image from the teacher data, and the plurality of the plurality of times are determined according to the number of times the processing is repeated until the category difference image is specified. teacher data verifying device teacher image is characterized Rukoto is divided into a plurality of groups. The teacher data verification device according to claim 1 ,
The evaluation value acquisition unit acquires the evaluation value for each of the plurality of categories of each teacher image based on the attention interval in the plurality of histograms and the frequencies of a plurality of intervals adjacent to both sides of the attention interval. A teacher data verification device characterized in that: The teacher data verification device according to claim 1 or 2 ,
Of the plurality of histograms, when the frequency of the attention interval in the histogram for one category is a predetermined number or less and the frequency of the attention interval in the histogram for each of all other categories is 0, the evaluation value A teacher data verification apparatus, wherein an acquisition unit does not acquire the evaluation value. The teacher data verification device according to any one of claims 1 to 3 ,
Wherein the display control unit, pre-listen categories the difference image image, the teacher data verifying device and displaying on the display unit together with the category and the category candidate the teacher data indicates. A teacher data creation device for creating teacher data used for learning of a classifier that classifies images,
A teacher data verification device according to any one of claims 1 to 4 ,
An input unit for accepting an input of the final categories against before asked categorical difference picture image,
A teacher data creation device comprising: An image classification device for classifying images,
A teacher data creation device according to claim 5 ;
Learning is performed using the teacher data created by the teacher data creation device, and a classifier that classifies images;
An image classification apparatus comprising: A teacher data verification method for verifying teacher data used for learning a classifier that classifies images,
a) preparing a plurality of teacher images each assigned to any of a plurality of categories, and teacher data indicating the categories of the plurality of teacher images;
b) obtaining an evaluation value indicating validity when each teacher image belongs to each of the plurality of categories for each of the plurality of types of feature values;
c) obtaining a representative value of the evaluation value in the plurality of types of feature amounts for each of the plurality of categories in each teacher image;
d) identifying a category determined as a category candidate to which each teacher image should belong based on a plurality of representative values in the plurality of categories;
The teacher image e) and the category indicated supervisor data and the category candidate is different as a category different from the image, and displaying the category difference picture image on the display unit together with the category candidates,
Equipped with a,
In the step b), for each type of feature quantity, a histogram of the feature quantity of the teacher image allocated to each of the plurality of categories is generated, and a section including the feature quantity of each teacher image is set as an attention section. Based on the frequency of the attention interval in a plurality of histograms for a plurality of categories, the evaluation value for each of the plurality of categories of each teacher image is acquired,
While removing the category difference image from the teacher data, the steps b) to d) are repeated a plurality of times, and the plurality of teacher images are a plurality of times depending on the number of repetitions of the process until the category difference image is specified. teacher data verification method according to claim Rukoto are divided into groups. The teacher data verification method according to claim 7 ,
In the step b), the evaluation value for each of the plurality of categories of each teacher image is acquired based on the attention section in the plurality of histograms and the frequencies of a plurality of sections adjacent to both sides of the attention section. The teacher data verification method characterized by the above-mentioned. The teacher data verification method according to claim 7 or 8 ,
In the step b), among the plurality of histograms, the frequency of the attention interval in the histogram for one category is not more than a predetermined number, and the frequency of the attention interval in the histogram for each of all other categories is 0. In this case, the teacher data verification method is characterized in that the evaluation value is not acquired. The teacher data verification method according to any one of claims 7 to 9 ,
Wherein e) in the step, prior to hear categorical difference picture image is teacher data verification method characterized in that it is displayed on the display unit together with the category and the category candidate the teacher data indicates. A teacher data creation method for creating teacher data used for learning a classifier that classifies images,
A teacher data verification method according to any one of claims 7 to 10 ,
Determining a final category against before asked categorical difference picture image,
A method for creating teacher data, comprising: An image classification method for classifying images,
Learning a classifier using the teacher data created by the teacher data creation method according to claim 11 ;
Classifying images by the classifier;
An image classification method comprising:

Images