WO2022214853A1

WO2022214853A1 - Method and apparatus for detecting surface defects

Info

Publication number: WO2022214853A1
Application number: PCT/IB2021/052945
Authority: WO
Inventors: Chandana Darshanapriya GAMAGE; Chathura Ranjan DE SILVA; Gardiye Hewawasam Suresh Indika DHARMARATNA; Pallemullage Sajith Harshana PALLEMULLA; Ranathunga Mudiyanselage Shashikala RANATHUNGA; Ranhotige Aruna Sri Kumara JAYASENA; Ratnayake Mudiyanselage Kalana Vidurath RATNAYAKE; Sachin Mibashara KAHAWALA; Sulochana Jayashamalee SOORIYAARACHCHI
Original assignee: University Of Moratuwa
Priority date: 2021-04-08
Filing date: 2021-04-09
Publication date: 2022-10-13

Abstract

A method and apparatus for detecting defects on surfaces using a fixed reference colour palette to distinguish between a change in material colour and a change in the ambient light and the use of UV illumination to convert defects caused by UV-active stains into visible spectral range. The apparatus captures images of a moving surface using imaging devices in the presence of visible, and Ultra violet lighting from front and rear of the surface with continuous calibration of the images according to the changes of the reference images under different lighting.

Description

Method and Apparatus for Detecting Surface Defects

TECHNICAL FIELD

The invention is a method and an apparatus to detect defects on surfaces TECHNICAL FEATURES

The invention consists of a method and an apparatus having a plurality of imaging devices, physical colour reference palettes, multispectral light sources illuminating the material from front and back, and a processing unit comprising multiple parallel matrix processing modules. This method and apparatus perform continuous calibration of the captured images during material inspection, based on the changes in the images of the physical reference colour palettes due to the changes in the images of physical reference colour palettes caused by brightness variations in the lighting system and/or ambient light variations. The steps of the method in this invention compensate for these variations and isolate the image variations caused by material defects. Thereby the defects on the surface of material such as fabric, tiles, paper and leather are accurately identified and reported.

BACKGROUND OF THE INVENTION

Quality control of input material is crucial for product manufacturing industries because a product that does not align to the quality standards of the customer may have to be discarded at a loss, provided at a significant price reduction or if passed undetected would cause reputational damage to the manufacturer. Hence, inspection of input material at pre-production as well as finished product at post-production are vital for manufacturers. The pre-production inspection is critical for many manufacturing industries due to the extensive value addition that occurs resulting in steep losses for products that were created using defective input material such as in the garment manufacturing industry.

In general, it is observed in material inspection systems that image characteristics of the same defective area vary under front, back and multispectral lighting conditions even though the reflections are captured under the visible spectrum by industrial-grade inspection cameras. Furthermore, the backlight illumination of a translucent material needs to be uniform throughout the camera frame to avoid misclassification of non-uniformly illuminated areas as defects.

In existing calibration systems that do not use a physical colour reference, the observed colour is a subjective measure and it is with reference to the initial colour of the material being inspected itself. Furthermore, when backlight illumination is not calibrated for uniform illumination continuously, then it leads to poor detection of defects on the inspection surface.

PRIOR ART

In reference to existing patents, JPH0815032A and CN111402341A propose methods in which material colour is compared with a reference colour value set by the operator or by a calibrating object moving with the material. However, these inventions do not address the complications that arise due to a change in ambient lighting. Many inventions with automated visual detection systems explained in patents CN210180905U, CN205426750U, US20070211460A1 and JP2017167047A, to name a few, have not factored in ambient light change in spite of its effects on the captured image, thereby interfering with the detection process. Colour calibration is also common in the fields of digital scanning and printing. However, in these fields, there is only an initial calibration and no successive or continuous calibration. In existing inventions, there is no continuous comparison with a physical reference throughout the inspection. Hence, changes in the ambient light occurring in industrial inspection environments that affect the material inspection process are not correctly dealt with by existing solutions.

A majority of the patents, for example, CN102221559A, CN102175692A, CN108986065A, KR100234593B1, US2019268522, US20140036061 and EP742431B1 to name a few, utilize only RGB lighting to illuminate the material. This limits the visibility of certain types of defects such as stains which are not prominent under RGB lighting, but fluoresce under UV lighting. The inventions proposed in JP2008268228A, US5572319A and CN210180905U utilize UV lighting to illuminate the material and capture the image by using an RGB camera, but are neither automated nor capable of continuous inspection. The invention proposed in CN110779899A is automated but is only suited for silk fabrics, while that in CN205426750U is only capable of detecting missing needle defects even though both these inventions use UV lighting for illumination.

The lighting used in during inspection should uniformly illuminate the region of the material being captured by the camera. This would facilitate the detection of smaller defects that cause only a slight change in the image brightness. In most inventions in the field of material inspection that involve backlighting, the brightness of the backlight has been assumed to be uniform. CN211199838U is one of the few inventions which proposes a backlight brightness adjustment technique where inputs from a brightness sensor and fabric thickness sensor are used to ensure uniform brightness. The optimal brightness setting for the backlight is obtained from a lookup table based on the fabric thickness, and the brightness is changed by adjusting the power input to the backlight. The main limitation in this invention is the difficulty of achieving linear and persistent brightness control through power regulation for the light-emitting diode based backlighting systems used in the industry.

A majority of the inventions related to brightness adjustment are related to organic light-emitting diode (OLED) displays and liquid crystal displays (LCD). CN108428436B attempts to minimize the adverse effects of luminance degradation of the OLEDs, which is not as critical a problem in LEDs. Other inventions which provide solutions for maintaining uniform luminosity in displays, such as CN101661124A, US2016071485A1, JP2007173108A and CN205982698U suggest altering the position, orientation and material of the backlight. The requirement of additional material and mechanisms for moving the lighting system to control brightness further negatively impacts the maintenance of a consistent surface lighting condition.

TECHNICAL PROBLEM

A major technical problem in the current solutions is performing only an initial calibration of image characteristics.

1. Setting an initial reference colour to be compared with the material colour will suffice in the detection of a colour variation in the material only if the ambient light is assumed to be constant. However, in practice, the ambient light varies on a factory floor causing a colour variation in the captured image of the same inspection surface. This colour variation will be incorrectly detected as a surface defect. Furthermore, sensitivity of image sensors show temporal variations due to numerous external factors, which may also result in a variation in image brightness and contrast.

Therefore, continuous calibration is essential in order to distinguish between a change in material colour and a change in the ambient light and to allow compensation against changes in the image sensor.

Another limitation in the current solutions is the utilisation of only visible lighting for material inspection. Solutions that use UV lighting to enhance detection exist but are not capable of continuous inspection.

2. Some defects, like those caused by organic material stains, might not appear prominently in the captured image when imaged only under visible lighting. Hence, these types of defects might be missed in the detection process.

Therefore, multispectral lighting is required to ensure the detection of certain defects.

A third technical problem is the non-uniformity of illumination in the field of view of the camera.

3. An initial setting of the lighting condition will suffice if the illumination is assumed to be constant both temporally and spatially. However, this environment is seldom available under practical conditions. Current solutions have not provided a method to continuously compensate for any non-uniformity in illumination.

Therefore, a continuous correction to the image brightness is required.

TECHNICAL SOLUTION

This invention proposes solutions to the technical problems above. They are as follows:

1. A fixed reference colour source such as a standard colour palette and an associated computational module to distinguish between a change in material colour and a change in the ambient light.

2. UV illumination to convert defects caused by UV-active stains into visible spectral range to ensure the detection of defects which otherwise could be undetectable under visible light.

3. A computational module to provide continuous compensation for any variations in illumination field under both front and rear illumination.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 - Isometric view of the apparatus

Figure 2 - A flow chart describing the method for detecting surface defects

Figure 3 - Camera, lighting, colour palette arrangement of the Apparatus

Figure 4 - Reference colour palette

The surface inspection machine [100] consists of imaging devices [102], front lighting [106], ultraviolet lighting [118], back lighting [108], an inspection bed [110], colour reference palettes [104], fabric/material roll [302], forward rolling mechanism [114], backward rolling mechanism [116] and a processing and controlling unit [112] The surface inspection method [200] has two phases. The initial phase [232] consists of capturing two initial images under two lighting conditions [202], namely, front lighting and back lighting. The initial stage should be run once a new material/fabric is loaded for surface inspection. In the initial phase, the brightness correction matrices [204] for front lighting and back lighting are calculated. Subsequently, three images [206] are captured under all three lighting conditions, namely, front lighting, back lighting and ultraviolet lighting (UV Lighting). Out of these three images, the images captured with front lighting and back lighting are recalibrated by applying the previously calculated brightness correction weight matrices [208,210] and the outputs are stored as reference images for front lighting and back lighting respectively for use during the second phase. Each of these two images are split into two parts, one with the colour palette reference area [402] and the other with the surface material area [404] The image captured under UV lighting in the previous stage is kept as the reference for UV lighting [212] to be used in second phase. The second phase [234] of the method is carried out repetitively throughout the inspection till the surface inspection material/fabric is changed. In the second phase, an image is captured under front lighting [214] Then the brightness correction weight matrix of front lighting is applied to the captured image [216] Thereafter, the brightness adjusted image is split into two parts as the colour palette reference area [402] and surface material area [404] Then these are compared with the reference images prepared in the initial phase and annotated if any deviation is detected. In order to remove any deviation caused by ambient lighting changes, the colour palette area of the reference image is compared with that of the current image and their difference is calculated. This difference is applied to the material surface area of the current image and the adjusted material surface area image is obtained. Subsequently, this current adjusted material surface area image is compared with the reference material area image [218] Any deviation in the former from the latter is detected as a defect [230] The same process is carried out for the back light image. The UV light image is directly compared with the reference UV image [228] and any dissimilarity is detected as a defect [230]

ADVANTAGES

This invention differentiates colour variations caused strictly due to defects in the inspected material from similar variations caused by changes in the ambient lighting and temporal and spatial variations in the camera sensors. Moreover, any non-uniformity in the illumination sources, including dynamic variations, are also compensated. The latter requires no additional physical components such as power regulation circuits, light diffusers or sliders thereby lowering the negative impact on maintaining a consistent lighting condition. The cause of image colour changes can be accounted to the material defects because the captured images are continuously compensated for changes in ambient lighting and illumination variations. Furthermore, this makes the inspection process robust for operation even under fluctuating lighting conditions. As a multispectral approach is used in which the image is captured by using both UV lighting and visible lighting, defects are distinctly differentiated.

DETAILED DESCRIPTION AND MODE OF INVENTION

This invention provides a method and an apparatus having a plurality of imaging devices, physical colour reference palettes, multispectral light sources illuminating the material from front and back, and a processing unit comprising multiple parallel matrix processing modules. This method and apparatus perform continuous calibration of the captured images during material inspection based on the changes in the images of the physical reference colour palettes due to the changes in the images of physical reference colour palettes caused by brightness variations in the lighting system and/or ambient light variations. The steps of the method in this invention compensate for these variations and isolate the image variations caused by material defects. Thereby the defects on the surface of material such as tiles, fabric, paper and leather are identified and reported. Automated inspection of surfaces such as tiles, fabric, paper and leather for defects, anomalies, and other imperfections as part of quality assurance requires images that highlight the defects. Image acquisition depends on the material being inspected, the light sources that illuminate the material and the camera parameters. It is required to identify deviations in images caused by defects and isolate them from the deviations caused by light variations. Further, the illumination conditions may vary while the cameras are capturing images during material inspection. Therefore, it is necessary to do system calibration continuously for each image captured by the camera system.

The invention uses multiple high speed high resolution industrial grade cameras that operate in parallel to capture multiple images across the width of the moving inspection surface. Therefore, a high-speed communication interface is implemented for real time image capturing and for transmitting the images to the processing unit in the apparatus. Each camera captures images in three modes as follows to highlight a variety of material defects.

In the first mode, the inspection surface is illuminated by lighting from the front and the image is captured to identify defects that cause significant changes in the reflection of incident light. In the second mode, the inspection surface is illuminated by lighting from behind the surface (back) being inspected and the image is captured for material that is translucent. This highlights defects that cause significant changes to the light that diffuses through the material. In the third mode, the inspection surface is illuminated with ultraviolet (UV) light from the camera side and the image is captured to highlight defects such as biological stains that cause emission of visible light under UV illumination (UV-active stains). The processing unit of the apparatus processes the images captured above using multiple parallel matrix processing modules such as Graphical Processing Units (GPUs) for real time operation.

The inspection surface is moved by means of an electromechanical system comprising forward and backward rolling and associated sensing and control. A controlling unit reads sensor data and issues commands to control this electromechanical system. These control commands are generated by the processing unit and transmitted using a high data rate communication protocol such as CAN-bus.

The first object of the invention is to eliminate the effect of ambient light changes on image acquisition in material inspection systems. In the proposed solution, every image captured by the camera is calibrated continuously by referring to the image details in the fixed reference colour palette. This method significantly improves the accuracy of defect detection over the current state- of-the-art mechanisms.

The second object of the invention is to enhance the detectability of a wider range of defects in the captured image. In the proposed solution, multispectral lighting (visible and UV lighting) is used, in different perspectives (front lighting, back lighting) which further improves the defect detection ability. For example in fabric defect detection, UV-active stains such as oil stains show higher contrast under UV illumination while defects such as holes, dislocated and missing yarns show higher contrast under backlighting. This method significantly improves the accuracy of error detection over the current state-of-the-art mechanisms by eliminating detection limitations imposed by only using visible spectrum of light to illuminate the surface. The third object of the invention is to eliminate any lighting-induced artifacts caused by unevenness in illumination that may interfere with the defect detection process to improve the accuracy of the output.

In summary, this invention discloses a method and an apparatus to calibrate the imaging system during every image frame capture, to mitigate the effect of changes in backlight and/or front light illumination and to illuminate the material using both visible and UV light. The calibration method differentiates image variations caused by defects on the inspection surface from image changes caused by illumination light variations and other similar effects.

Claims

1. A Method for detecting surface defects in material inspection, the said method comprising the steps of: providing a fixed reference colour palette on the surface under inspection; providing plurality of imaging devices to capture images of the surface using front lighting, back lighting, and ultraviolet illumination; causing the rectification of the brightness values of the image received under front lighting and back lighting; causing the rectification of ambient light variation of the image received under front lighting; determining the image variations of the captured surface under back lighting and/or front lighting and under ultraviolet illumination; identifying the defect by continuous calibration of the test images with the reference images.

2. The method for detecting surface defects claimed in 1 wherein, the method further includes capturing an image comprising the material and the fixed reference colour palette is captured under front lighting at the start of inspection and is referred to as the initial front light reference image.

3. The method for detecting surface defects claimed in 1 wherein, the method further includes capturing an image comprising the material under back lighting at the start of inspection and is referred to as the initial back light reference image.

4. The method for detecting surface defects claimed in 1 wherein, the method further includes capturing an image comprising the material under ultraviolet illumination at the start of inspection and is referred to as the ultraviolet light reference image.

5. The method for detecting surface defects according to claim 1 wherein, the method further includes determining a brightness correction matrix for the image captured under front lighting at the start of inspection and is referred to as the brightness correction matrix for front lighting.

6. The method for detecting surface defects according to claim 1 wherein, the method further includes determining a brightness correction matrix for the image captured under back lighting at the start of inspection and is referred to as the brightness correction matrix for back lighting.

7. The method for detecting surface defects according to claim 1 wherein, the method further includes rectifying the brightness values of the initial front light reference image in claim 2 using the brightness correction weight matrix of front lighting in claim 5 at the start of inspection and the resulting image is referred to as the front light reference image.

8. The method for detecting surface defects according to claim 1 wherein, the method further includes rectifying the brightness values of the initial back light reference image in claim 3 using the brightness correction weight matrix of back lighting in claim 6 at the start of inspection and the resulting image is referred to as the back light reference image.

9. The method for detecting surface defects claimed in 1 wherein, the method further includes capturing an image comprising the material and the fixed reference colour palette under front lighting continuously during inspection and is referred as to as the front light test image.

10. The method for detecting surface defects according to claim 1 wherein, the method further includes rectifying the brightness values of the front light test image in claim 9 continuously during inspection using the brightness correction weight matrix for front lighting in claim 5 and is referred to as the brightness-corrected front light test image.

11. The method for detecting surface defects according to claim 1 wherein, the method further includes rectifying the ambient light variations continuously during inspection by comparing the fixed reference colour palette area of the front light reference image in claim 7 and that of the brightness-corrected front light test image in claim 10, and the resulting image is referred to as the rectified front light image.

12. The method for detecting surface defects according to claim 1 wherein, the method further includes identifying the defected images continuously during inspection by comparing the rectified front light image in claim 11 and the front light reference image in claim 7.

13. The method for detecting surface defects claimed in 1 wherein, the method further includes capturing an image comprising the material continuously during inspection under back lighting and is referred to as the back light test image.

14. The method for detecting surface defects according to claim 1 wherein, the method further includes rectifying the brightness values of the back light test image in claim 13 continuously during inspection using the brightness correction weight matrix for back lighting in claim 6 and is referred to as the rectified back light test image.

15. The method for detecting surface defects according to claim 1 wherein, the method further includes identifying the defected images continuously during inspection by comparing the rectified back light test image in claim 14 and the back light reference image in claim 8.

16. The method for detecting surface defects claimed in 1 wherein, the method further includes capturing an image comprising the material continuously during inspection under ultraviolet illumination and is referred to as the ultraviolet test image.

17. The method for detecting surface defects according to claim 1 wherein, the method further includes identifying the defected images continuously during inspection by comparing the ultraviolet test image in claim 16 and the ultraviolet reference image in claim 4.

18. The method for detecting surface defects according to claim 1 wherein, the method further includes subjecting the defected images to a series of processing steps that evaluate image features to determine the type and nature of the defects.

19. An apparatus for inspecting and detecting an existence of a defect on the surface , said apparatus comprising; an inspection bed for laying the material; imaging devices for capturing images of the material on the inspection bed; a front light source for illuminating the material surface on the inspection bed from the front; a back light source for illuminating the material on the inspection bed from the back; an ultraviolet light source for illuminating the material surface on the inspection bed from the front; colour reference palettes mounted on the inspection bed above the material to be visible throughout the inspection process; a rolling mechanism for moving the material forward and/ or backward on the inspection bed; sensors for measuring speed and material alignment in unrolling/ rolling; a controller unit for acquiring sensor readings from the rolling mechanism and to generate control commands such as stop, start, speed-up, speed down of the rolling mechanism; a processing unit comprising multiple parallel matrix processing modules for the processing of images and to identify the defected images; an interface to connect imaging devices to the processing unit; an interface for the processing unit to communicate with the controller unit of the rolling mechanism; a display unit for reporting the defect statistics of the inspection.

20. The apparatus for inspecting and detecting an existence of a defect on the surface in claim 19 wherein, the ultraviolet light source is projected on to the material surface to capture the visible light emitted by the UV-active defects such as oil spots and biological stains.

21. The apparatus for inspecting and detecting an existence of a defect on the surface in claim 19 wherein, a single imaging device is a high speed high resolution industrial inspection camera.

22. The apparatus for inspecting and detecting an existence of a defect on the surface in claim 19 wherein, a single imaging device is mounted and focused on to a part of the inspection bed of the size of the field of view of the imaging device at the mounted height and captures images in the visible spectrum.

23. The apparatus for inspecting and detecting an existence of a defect on the surface in claim 19 wherein, the imaging devices are arranged in an array to capture the whole area of the inspection bed.

24. The apparatus for inspecting and detecting an existence of a defect on the surface in claim 19 wherein, a physical colour reference palette is mounted in the field of view of each imaging device.

25. The apparatus for inspecting and detecting an existence of a defect on the surface in claim 19 wherein, the physical colour reference palette is an object that appears as identical images under ideal and uniform lighting for each imaging device and any differences in captured images of the physical colour reference palettes indicate changes in the lighting condition.

26. The apparatus for inspecting and detecting an existence of a defect on the surface in claim 19 wherein, the interface between the processing unit and the controller of the rolling mechanism is a high data rate communication protocol for real-time roller control.

27. The apparatus for inspecting and detecting an existence of a defect on the surface in claim 19 wherein, the processing unit includes multiple parallel matrix processing modules to process images and to detect defects.

28. The apparatus for inspecting and detecting an existence of a defect on the surface in claim 19 wherein, the interface between the imaging devices and the processing unit is a high data rate communication interface that operate parallelly for each imaging device.

29. The apparatus for inspecting and detecting an existence of a defect on the surface in claim 19 wherein, the interface between the processing unit and the control unit is used to control the rolling mechanism to mark identified defects and to control image capture.