KR20200039049A - Inspection method for appearance badness and inspection system for appearance badness - Google Patents

Abstract

An appearance defect inspection method of the present invention comprises the steps of: acquiring an inspection image; recovering the inspection image by means of a trained denoising auto-encoder (dA), and extracting a difference between the recovered image and the inspection image to a recovery error map; converting the recovery error map to a binary error map; sizing a defect area in the binary error map by means of morphology image processing and blob processing, and detecting a defect candidate whose size of defect area is greater than a predetermined size; extracting a region of interest (ROI) patch of the defect candidate, and scaling the ROI patch to a predetermined size; and calculating a defect probability of the scaled ROI patch of the defect candidate by means of a learned convolution neural network (CNN), and determining the defect candidate to be a defective product if the defect probability is greater than a predetermined value, or determining the defect candidate to be an acceptable product if the defect probability is equal to or less than the predetermined value.

Description

Inspection method and exterior defect inspection system {INSPECTION METHOD FOR APPEARANCE BADNESS AND INSPECTION SYSTEM FOR APPEARANCE BADNESS}

The present invention relates to an external defect inspection method and an external defect inspection system.

The technology for inspecting appearance defects is a relatively old technology, and it began to appear as product production was automated. In the meantime, the productivity has been improved by the automated inspection system using cameras and image processing technology, replacing the ones that were inspected by humans, and the quality has been improved through consistent inspection.

However, with the rapid development of IT technology and the explosive demand for IT, the trend of semiconductor, display, and pcb processes is also rapidly changing. In particular, due to the technological trend called light and compact, the process is complicated and refined, leading to new inspection for defect inspection. Although the demand is increasing, the types of defects are diversified and the boundary with good products becomes ambiguous.

In the existing appearance defect inspection system, a skilled expert had to perform a pre-treatment and feature extraction process based on experience, and it was difficult to automate the exterior inspection system in that proper pre-treatment and feature extraction was required according to the inspected object.

Therefore, it is possible to extract high-level abstract features through optimized image pre-processing and optimal processes, and thus, there is a need for an appearance defect inspection method and an exterior defect inspection system capable of improving inspection accuracy, automation, and productivity.

An object of the present invention is to provide a visual defect inspection method and a visual defect inspection system with improved inspection accuracy.

Another object of the present invention is to provide an automated defect inspection method and an appearance defect inspection system capable of automation and improved productivity.

The above and other objects of the present invention can be achieved by the present invention described below.

One aspect of the present invention relates to a defect inspection method.

According to one embodiment, the appearance defect inspection method comprises the steps of obtaining an inspection image; Restoring the inspection image to a learned dA (Denoising Auto-Encoder), and extracting a difference between the restored image and the inspection image into a restoration error map; Converting the restored error map into a binary error map; Calculating a size of a defective area by morphology image processing and blob processing on the binary error map, and detecting a defect candidate whose size is larger than a preset size; Extracting the defective candidate region of interest (ROI) patch and scaling to a preset size; And calculating a defective probability using the learned convolution neural network (CNN) of the scaled defective candidate ROI patch, and determining that the defective probability is a defective product when the defective probability is greater than a predetermined value, and determining that the defective product is a good product. Includes.

In addition, the dA (Denoising Auto-Encoder) may be to learn only with images of good products.

을 문턱값으로 사용할 수 있다.In addition, the step of converting to the binary error map of Equation 1 below

Can be used as a threshold.

[Equation 1]

,

,

는 입력 영상의 (i, j) 위치에 해당하는 픽셀의 밝기,

는 복원된 영상의 픽셀 밝기,

는 입력 영상의 폭,

는 입력 영상의 높이 이며,

은 학습 시 사용된 영상의 총 개수임).(In the above formula 1,

Is the brightness of the pixel corresponding to the (i, j) position of the input image,

Is the pixel brightness of the restored image,

Is the width of the input image,

Is the height of the input image,

Is the total number of images used in learning).

In addition, the scaling of the defective candidate ROI (region of interest) patch may be scaled to a size of 128x128.

According to another embodiment, the Convolution Neural Network (CNN) may be composed of a Convolutional Auto-Encoder (CAE) and a Convolutional Fully-connected Net (CFN).

In addition, the appearance defect inspection method detects the location of the defect by using an activation map from the CAE, and the determination of the quality and defect of any pixel (i, j) of the activation map is as shown in Equation 2 below. can do.

[Equation 2]

는 (i,j) 픽셀 위치에서의 활성화 맵(activation map)의 활성화 값(activation value)을 나타내며

는 알고 있는 R개의 양품 영상 패치(patch)를 CNN에 투입했을 때 CAE(Convolutional Auto-Encoder)로부터 얻은 활성화 맵(activation map)에서의 활성화 값(activation value)들의 전체 평균값이며, H는 양품 영상 패치의 높이, W는 양품 영상 패치의 폭, m은 채널 깊이임).(Equation 2 above,

Denotes the activation value of the activation map at the (i, j) pixel position

Is a total average value of activation values in an activation map obtained from a convolutional auto-encoder (CAE) when a known R quality patch is supplied to CNN, and H is a quality video patch. Is the height, W is the width of the quality video patch, and m is the channel depth).

Another aspect of the present invention relates to a defect inspection system.

According to one embodiment, the appearance defect inspection system includes a camera for photographing the appearance of the inspected object; A control unit for detecting a defect candidate with a learned denoising auto-encoder (dA), and determining a defective product by using the learned convolution neural network (CNN); And a notification unit outputting a signal according to the signal of the control unit when it is determined that the control unit is a defective product, and the control unit may determine a defect by the inspection method of any one of claims 1 to 6.

The present invention has an effect of providing an inspection method for appearance defects and an inspection system for appearance defects with improved inspection accuracy, automation, and improved productivity.

1 is a simplified diagram showing a flow chart of a defect inspection method according to an embodiment of the present invention.
FIG. 2 is a schematic diagram showing a CNN configuration of an external defect inspection method according to another embodiment of the present invention.

Hereinafter, the present invention will be described in more detail.

In describing the present invention, when it is determined that detailed descriptions of related known technologies may unnecessarily obscure the subject matter of the present invention, detailed descriptions thereof will be omitted.

When 'include', 'have', 'consist of', etc. mentioned in this specification are used, other parts may be added unless '~ man' is used. When a component is expressed as a singular number, the plural number is included unless otherwise specified.

In addition, in analyzing the components, it is interpreted as including an error range even if there is no explicit description.

In addition, in this specification, 'X to Y' representing a range means 'X or more and Y or less'.

Referring to Figure 1 will be described the appearance defect inspection method according to an embodiment of the present invention. 1 is a simplified diagram showing a flow chart of a defect inspection method according to an embodiment of the present invention.

Referring to Figure 1, the appearance defect inspection method according to an embodiment of the present invention comprises the steps of obtaining an inspection image; Restoring the inspection image to a learned dA (Denoising Auto-Encoder), and extracting a difference between the restored image and the inspection image into a restoration error map; Converting the restored error map into a binary error map; Calculating a size of a defective area by morphology image processing and blob processing on the binary error map, and detecting a defect candidate whose size is larger than a preset size; Extracting the defective candidate region of interest (ROI) patch and scaling to a preset size; And calculating a defective probability using the learned convolution neural network (CNN) of the scaled defective candidate ROI patch, and determining that the defective probability is a defective product when the defective probability is greater than a predetermined value, and determining that the defective product is a good product. Includes.

Hereinafter, each step will be described in detail.

In the obtaining of the inspection image, a means for obtaining the image may be used. For example, an image for inspection may be acquired by using a camera to photograph the appearance of the subject, but is not limited thereto.

The dA (Denoising Auto-Encoder) applied in the present invention randomly adds noise to an image of a good product, and then removes this noise to extract a good image before adding the noise. As a model, we learn by checking whether the extracted image is the image value of pure good products before adding noise. In addition, the learning may be performed by adding noises of various types and degrees (for example, noise ranges from 25% to 50%), and the dA thus learned may be made to learn more distinct features. Therefore, in the present invention, dA was not trained as labeled data containing a defective product, but can have a sufficient effect to detect a defective candidate, extracting a defective candidate before the CNN step described later, and ROI of a specific size or less CNN By providing to, there is an advantage that the judgment speed is also excellent. In addition, it is easy to learn dA in that it is practically difficult to sufficiently secure labeled data containing defective products.

For the above learning, for example, a random image or a whole region is filled with a certain value in a random image, and the original image is converted to a value of 0 to 1.0 through normalization, and dA is not learned. Input to get a prediction of dA. The dA prediction is a reconstructed image having a value between 0 and 1.0, which is different from the original image. This difference is defined as the cost (restore error) (Equation 3 below), and the cost (restore error) value is not changed above the preset criteria using the back propagation method, or the weight and bias inside the dA are no longer. Learning will end when it does not change or reaches a predetermined number of iterations.

[Equation 3]

는 입력 영상의 (i, j) 위치에 해당하는 픽셀의 밝기,

는 복원된 영상의 픽셀 밝기,

는 입력 영상의 폭,

는 입력영상의 높이 이며

은 한 묶음(batch) 의 영상의 개수이다(Equation 3 above,

Is the brightness of the pixel corresponding to the (i, j) position of the input image,

Is the pixel brightness of the restored image,

Is the width of the input image,

Is the height of the input image

Is the number of images in a batch

The inspection image is restored to the learned dA (Denoising Auto-Encoder). Then, a step of extracting a difference between the restored image and the inspection image into a reconstruction error map is performed.

을 문턱값으로 사용할 수 있다. 이 경우, 검사 정확도가 개선되는 효과가 있다. 한편,

값 정도의 에러는 정상 상태의 복원에서도 발생할 수 있는 에러임을 의미할 수 있다.The restoration error map is converted to a binary error map, in which case Equation 1 below

Can be used as a threshold. In this case, there is an effect that the inspection accuracy is improved. Meanwhile,

An error of a value degree may mean that an error may occur even in a normal state restoration.

[Equation 1]

,

,

는 입력 영상의 (i, j) 위치에 해당하는 픽셀의 밝기,

는 복원된 영상의 픽셀 밝기,

는 입력 영상의 폭,

는 입력 영상의 높이 이며,

은 학습 시 사용된 영상의 총 개수임).(In the above formula 1,

Is the brightness of the pixel corresponding to the (i, j) position of the input image,

Is the pixel brightness of the restored image,

Is the width of the input image,

Is the height of the input image,

Is the total number of images used in learning).

The binary error map is not defective due to system noise, but morphology processing is performed to remove small-sized data that can be recognized as defective, and blob processing is performed to calculate the location and size of the defective area ( blob processing). If the size of the defective area calculated by the above process is larger than the preset size (which can be appropriately determined according to the inspection environment and inspection purpose), it is determined as a defective candidate.

For a blob determined as a bad candidate, a region of interest (ROI) patch may be extracted and scaled to a preset size in consideration of the calculation speed of the CNN and the size of the activation map described later. For example, the step of scaling the defective candidate ROI (region of interest) patch may be scaled to a size of 128x128. In this case, the inspection speed is excellent, which has the advantage of improving productivity.

The CNN (Convolution Neural Network) applied to the present invention is for the purpose of feature extraction and classification of good and bad products. Any type of deep neural net and support vector machine (SVM) that can be used for classification purposes can be used. Normally, several layers of convolution layers can be used by attaching several layers of fully connected layers. In this specification, a general neural net having such a structure is commonly referred to as a convolutional neural net (CNN). Due to its structural characteristics, CNN can extract features using multiple stage convolution kernels (3x3, 5x5, 7x7, etc.). By using a mechanism called max pooling, the size of the input image is gradually reduced, and meaningful information can be extracted regardless of the position of the object in the image. Therefore, the deeper the layer of CNN, the more abstract feature extraction is possible. Since this high level of abstraction is made to the level that even experts with domain knowledge in the related field can not imagine, if the deep neural net is well constructed, the code is modified to extract new features whenever the object to be inspected changes. There is also an advantage that can solve the practical hassle of developing a new algorithm.

In addition, since the learning process itself uses an optimization that reduces the total learning error by using a back propagation method, there is no need for an engineer to make a selection rule by using features like a conventional technique. Through this, there is an advantage of naturally obtaining a multi-dimensional decision boundary.

Specifically, the learning of the CNN ends when the cost (total error) defined in the following Equation 4 does not change beyond a preset criterion or reaches a predetermined number of repetitions.

[Equation 4]

When the ROI patch of the bad candidate is injected into the learned CNN (Convolution Neural Network), the output value appears as a probability map for the defect, that is, the ROI patch of the bad candidate is learned by the learned Convolution Neural Network (CNN). Calculation, if the probability of the defect is greater than a predetermined value, it is determined as a defective product, and if it is less than the value, it is determined as a good product.

According to an embodiment, the CNN (Convolution Neural Network) applied to the present invention may be configured and learned only by a Convolutional Fully-connected Net (CFN). In this case, there is an advantage that the productivity can be further improved by maximizing the inspection speed.

According to another embodiment, the Convolution Neural Network (CNN) may be composed of a Convolutional Auto-Encoder (CAE) and a Convolutional Fully-connected Net (CFN).

Referring to FIG. 2, CNN, a deep neural net for feature extraction and classification, is composed of two parts, Convolutional Auto-Encoder (CAE) and Convolutional Fully-Connected Net (CFN), and is simultaneously learned. The activation map 30 is a W x H x m dimension and m is the channel depth.

)인지 불량 픽셀(

)인지 아래 식 2에 의해 정해질 수 있다.Any pixel (i, j) of the activation map 30 is a good pixel (

) Cognitive bad pixels (

) May be determined by Equation 2 below.

[Equation 2]

는 (i,j) 픽셀 위치에서의 활성화 맵(activation map)의 활성화 값(activation value)을 나타내며

는 알고 있는 R개의 양품 영상 패치(patch)를 CNN에 투입했을 때 CAE(Convolutional Auto-Encoder)로부터 얻은 활성화 맵(activation map)에서의 활성화 값(activation value)들의 전체 평균값이며, H는 양품 영상 패치의 높이, W는 양품 영상 패치의 폭, m은 채널 깊이임).(Equation 2 above,

Denotes the activation value of the activation map at the (i, j) pixel position

Is a total average value of activation values in an activation map obtained from a convolutional auto-encoder (CAE) when a known R quality patch is supplied to CNN, and H is a quality video patch. Is the height, W is the width of the quality video patch, and m is the channel depth).

Another aspect of the present invention relates to a defect inspection system.

According to one embodiment, the appearance defect inspection system detects a defect candidate with a learned dA (Denoising Auto-Encoder) of a camera photographing the appearance of an inspected object, and an image captured by the camera, and learned CNN (Convolution Neural) Network) includes a control unit for determining a defective product, and when the control unit determines that the defective product, a control unit for outputting a signal according to the signal of the control unit, the control unit to determine the defect by the inspection method according to an aspect of the present invention You can.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and may be manufactured in various different forms, and those skilled in the art to which the present invention pertains have technical spirit of the present invention. However, it will be understood that other specific forms may be practiced without changing essential features. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and not restrictive.

Claims (7)

Obtaining an inspection image;
Restoring the inspection image to a learned dA (Denoising Auto-Encoder), and extracting a difference between the restored image and the inspection image into a restoration error map;
Converting the restored error map into a binary error map;
Calculating a size of a defective area by morphology image processing and blob processing on the binary error map, and detecting a defective candidate whose size is larger than a preset size;
Extracting the defective candidate region of interest (ROI) patch and scaling to a preset size; And
Calculating a defective probability by using the learned convolutional neural network (CNN) of the scaled defective candidate ROI patch, and determining that the defective probability is a defective product when the defective probability is greater than a predetermined value, and determining that the defective product is a good product;
Appearance defect inspection method comprising a.
According to claim 1,
The dA (Denoising Auto-Encoder) is a defect inspection method characterized in that the learning only with images of good products.
According to claim 1,
The step of converting the binary error map to Equation 1 below

Defect inspection method using as a threshold:
[Equation 1]

,
(In the above formula 1,

Is the brightness of the pixel corresponding to the (i, j) position of the input image,

Is the pixel brightness of the restored image,

Is the width of the input image,

Is the height of the input image,

Is the total number of images used in learning).
According to claim 1,
The step of scaling the ROI (region of interest) patch of the defect candidate is a defect inspection method of scaling to a size of 128x128.
According to claim 1,
The CNN (Convolution Neural Network) is a visual defect inspection method consisting of a Convolutional Auto-Encoder (CAE) and a Convolutional Fully-connected Net (CFN).
The method of claim 5,
It is a visual defect inspection method that detects the location of a defect by using an activation map from the CAE,
Quality of any pixel (i, j) of the activation map (

) And defective products (

) Determination of the appearance defect inspection method according to the following equation 2:
[Equation 2]

(Equation 2 above,

Denotes the activation value of the activation map at the (i, j) pixel position

Is a total average value of activation values in an activation map obtained from a convolutional auto-encoder (CAE) when a known R good quality image patch is applied to CNN, and H is a good quality image patch. Is the height, W is the width of the quality video patch, and m is the channel depth).
A camera photographing the appearance of the inspected object;
A control unit for detecting a defect candidate with a learned denoising auto-encoder (dA), and determining a defective product by using the learned convolution neural network (CNN); And
When the control unit is determined to be defective, including a notification unit for outputting a signal according to the signal of the control unit,
The control unit is a defect inspection system for determining a defect by the inspection method of any one of claims 1 to 6.

Images