JP7557962B2 - Image processing device and control method thereof - Google Patents

Description

The present invention relates to quality inspection of printed matter using image processing.

In printed matter output by a printing device, stains may occur due to color materials such as ink or toner adhering to unintended locations. Alternatively, color materials may adhere to locations where an image should be formed, resulting in insufficient color material adhering to those locations, causing color loss and making those locations lighter than they should be. These types of printing defects, such as stains and color loss, reduce the quality of printed matter. For this reason, it is necessary to inspect printed matter for defects in order to guarantee the quality of printed matter.

Visual inspection, in which an inspector visually checks for defects, requires a lot of time and costs money, so in recent years, inspection systems have been proposed that perform inspections automatically without relying on visual inspection. In such inspection systems, image data obtained by scanning a printout without defects is registered in advance as a reference image for inspection, and the presence or absence of defects is calculated based on the difference between the image data obtained by scanning the printout to be inspected (image to be inspected) and the reference image for inspection.

However, the positional relationship between the medium and the image printed on the medium generally fluctuates slightly for each print due to misalignment of the medium transport and printing position of the printing device. Therefore, to properly inspect for defects based on the above-mentioned differences, it is necessary to accurately align the images in the inspection reference image and the inspection target image. A commonly used method for alignment is to use template matching to compare the brightness of partial images commonly observed in the inspection reference image and the inspection target image, and to determine the amount of displacement that maximizes the degree of match (correlation) between the two.

To perform template matching, it is necessary to appropriately set a search range for one partial image, which is an area that may be included in the other image. For this reason, there is a method of detecting the position or orientation of the print in the image to be inspected in advance and performing alignment based on this. Patent Document 1 discloses a method of detecting the edges at both ends of each line that makes up the image to be inspected and calculating the inclination of the print by referencing these edges for multiple lines.

JP 2009-111472 A

Inspection systems often use line sensors, such as contact image sensors (CIS), to scan printed matter. If the width of the line sensor is shorter than one side of the printed matter, some areas of the printed matter will extend beyond the image obtained by scanning.

In the technology disclosed in Patent Document 1, if each line constituting an image is shorter than the width of the document, the edges at both ends that are assumed to be the subject of observation cannot be observed, and the position and inclination cannot be calculated. As a result, the patterns of the inspection reference image and the inspection target image cannot be accurately aligned, and the presence or absence of defects cannot be properly inspected.

The present invention was made in consideration of these problems, and aims to provide a technology that enables defects in printed matter to be detected more effectively.

In order to solve the above-mentioned problems, the image processing device according to the present invention has the following arrangement.
a first acquisition means for acquiring a reference image obtained by reading a first medium having a given image formed on a surface thereof by a printing device;
a second acquisition means for acquiring a target image obtained by reading a second medium having the given image formed on a surface thereof by the printing device;
a determining means for determining coordinates of vertices of the first medium in the reference image and coordinates of vertices of the second medium in the target image;
a first alignment means for aligning the reference image and the target image based on the vertices in the reference image and the target image determined by the determination means;
a second alignment means for aligning the reference image and the target image based on a result of template matching between the reference image and the target image aligned by the first alignment means;
an inspection means for inspecting the target image by comparing the reference image aligned by the second alignment means with the target image;
having
The determining means determines the coordinates of at least one vertex by estimation when at least one of the reference image and the target image does not include the entire area of the medium ;
The second alignment means determines a search range in the target image for performing the template matching based on the attributes of the vertices in the target image determined by the determination means .
Alternatively, the image processing device includes a first acquisition unit that acquires a reference image obtained by reading a first medium having a given image formed on a surface thereof by a printing device;
a second acquisition means for acquiring a target image obtained by reading a second medium having the given image formed on a surface thereof by the printing device;
a determining means for determining coordinates of vertices of the first medium in the reference image and coordinates of vertices of the second medium in the target image;
a first alignment means for aligning the reference image and the target image based on the vertices in the reference image and the target image determined by the determination means;
a second alignment means for aligning the reference image and the target image based on a result of template matching between the reference image and the target image aligned by the first alignment means;
an inspection means for inspecting the target image by comparing the reference image aligned by the second alignment means with the target image;
having
The determining means determines the coordinates of at least one vertex by estimation when at least one of the reference image and the target image does not include the entire area of the medium;
The second alignment means determines a search range in the target image for performing the template matching based on a maximum value of a variation amount of an image forming position in the printing device.
Alternatively, the image processing device includes a first acquisition unit that acquires a reference image obtained by reading a first medium having a given image formed on a surface thereof by a printing device;
a second acquisition means for acquiring a target image obtained by reading a second medium having the given image formed on a surface thereof by the printing device;
a determining means for determining coordinates of vertices of the first medium in the reference image and coordinates of vertices of the second medium in the target image;
a first alignment means for aligning the reference image and the target image based on the vertices in the reference image and the target image determined by the determination means;
a second alignment means for aligning the reference image and the target image based on a result of template matching between the reference image and the target image aligned by the first alignment means;
an inspection means for inspecting the target image by comparing the reference image aligned by the second alignment means with the target image;
having
The determining means determines the coordinates of at least one vertex by estimation when at least one of the reference image and the target image does not include the entire area of the medium;
The second alignment means determines a search range in the target image for performing the template matching based on a maximum value of size variation that occurs when cutting the second medium in a manufacturing process.

The present invention provides a technology that enables more efficient detection of defects in printed matter.

1 is a diagram showing an overall configuration of a printing system including an image processing apparatus. FIG. 1 is a block diagram showing a configuration of an image processing device. 11 is a flowchart illustrating a process of the image processing device. FIG. 13 is a diagram showing an example of an inspection reference image. 11A and 11B are diagrams illustrating examples of protrusion of an inspection target image from a print medium. FIG. 11 is a diagram for explaining a method for calculating the number of observable vertices. 11A and 11B are diagrams illustrating a process of estimating a vertex of a print medium. 13 is a diagram illustrating a process when the number of observable vertices is 0. FIG. FIG. 11 is a diagram illustrating a position alignment process. FIG. 13 is a diagram illustrating a method for determining a coefficient R.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

First Embodiment
As a first embodiment of an image processing device according to the present invention, an image processing device that inspects an inspection target image obtained by scanning a printed matter for the presence or absence of defects will be described below. In particular, an example will be described in which the image processing device detects whether the printout is protruding from the inspection target image and performs position adjustment for inspection.

<Device Configuration>
1 is a diagram showing the overall configuration of a printing system including an image processing device 100 according to a first embodiment. The printing system includes the image processing device 100, a printing server 180, and a printing device 190. The printing server 180 generates a print job for an original to be printed, and submits the print job to the printing device 190.

The printing device 190 forms an image on a print medium based on a print job submitted from the printing server 180. The printing device 190 has a paper feed unit 191, and the user supplies print paper to the paper feed unit in advance. When a print job is submitted, the printing device 190 transports the print medium supplied to the paper feed unit 191 along a transport path 192, forms an image on the surface (one side or both sides), and sends it to the image processing device 100.

The image processing device 100 performs an inspection process to check for defects on the print medium transported from the printing device 190 through the transport path 192. The image processing device 100 includes a CPU 101, a RAM 102, a ROM 103, and an image reading device 105. It also includes a printing device interface (I/F) 106, a general-purpose interface (I/F) 107, a user interface (UI) panel 108, and a main bus 109.

Furthermore, the printer 190 is provided with a print medium transport path 110 connected to the transport path 192 of the printing device 190, an output tray 111 for outputting print media that have been determined to have passed the inspection, and an output tray 112 for outputting print media that have been determined to have failed the inspection. The CPU 101 selects whether the print media is to be transported to the output tray 111 or the output tray 112 depending on the inspection results.

The CPU 101 is a processor that provides overall control of each unit within the image processing device 100. Each process described below is executed by the CPU 101, and is realized by controlling each module within the image processing device 100 in a timely manner. The RAM 102 temporarily stores applications executed by the CPU 101, data used in image processing, etc. The ROM 103 stores a group of programs executed by the CPU 101.

The image reading device 105 reads one or both sides of the print medium sent from the printing device on the conveying path 110 and obtains it as image data. The conveying path 110 is configured in a color (e.g., black) that is easily distinguishable from the print medium in the image, since it is the background when the image reading device 105 reads the image of the print medium. The printing device I/F 106 is connected to the printing device 190, appropriately synchronizes the printing device 190 and the image processing device 100, and notifies each other of their operating status. The UI panel 108 is realized by a display device such as a liquid crystal display, and functions as a user interface for the image processing device 100, notifying the user of the current status and settings. In addition, by providing an input device such as a touch panel or buttons, instructions from the user are accepted. The main bus 109 is a transmission path that connects each module of the image processing device 100.

As the print medium output from the printing device 190 is transported along the transport path 110, the image processing device 100 performs an inspection process to check for defects based on image data acquired by the image reading device 105. If the inspection process results in the print medium passing the inspection, the print medium is transported to the output tray 111. If the inspection process results in the print medium failing the inspection, the print medium is transported to the output tray 112. This ensures that only print media without defects is present on the output tray 111.

Figure 2 is a block diagram showing the configuration of the image processing device 100. The rectangular frames represent functional modules that perform each process, and the arrows represent the flow of the process. However, the configuration in Figure 2 is one example and is not limited to the one shown.

The image acquisition unit 201 acquires the print medium on the transport path 110 as image data using the image reading device 105. Here, the image acquisition unit 201 reads the print medium at least twice at different points in time to acquire image data. In the following description, the image acquired by the image acquisition unit 201 at the first point in time is called the inspection reference image 301, and the image acquired at the second point in time or later is called the inspection target image 302. However, it is sufficient to use the inspection reference image 301 as the image that serves as the reference for inspection (an image without defects), and the method of acquiring the inspection reference image 301 and the inspection target image 302 is not limited to the above order.

The overhang detection unit 202 detects whether the entire print medium is included in the inspection target image 302 acquired by the image acquisition unit 201. If even a part of the print medium is not included in the image data, the overhang detection unit 202 determines that overhang has been detected. The medium edge estimation unit 203 detects the edge positions of the print medium based on the detection results of the overhang detection unit 202.

The alignment unit 204 calculates the amount of displacement between the images of the pattern area 305 by referring to the estimation result by the edge estimation unit 203 based on the luminance of the pattern area 305 observed in the inspection reference image 301 and the inspection target image 302. The amount of displacement includes that due to translation, rotation, and enlargement/reduction.

The inspection unit 205 calculates the difference between the inspection reference image 301 and the inspection target image 302 after alignment by the alignment processing unit 204, and performs the inspection. The inspection unit 205 determines that a portion where the difference between the inspection reference image 301 and the inspection target image 302 is large is defective.

<Device Operation>
3A and 3B are flow charts for explaining the processing of the image processing device 100. Fig. 3A is a flow chart showing the flow of processing of the entire device. Fig. 3B is a flow chart showing details of the medium edge estimation processing (S1040). Fig. 3C is a flow chart showing details of the registration processing (S1050).

In S1010, the image processing unit 201 controls the image reading device 105 to read the print medium on the transport path 110 and obtain image data. The image data read in S1010 is stored in the RAM 103 as the inspection reference image 301 and is used in subsequent processing.

Figure 4 is a diagram showing an example of an inspection reference image 301. The inspection reference image 301 is composed of a background area 303 (area shown in black) and a print medium area 304 (area other than black). The print medium area 304 includes a picture area 305, which is an area in which a given image is formed. Vertices 308a to 308d are vertices of the print medium. The background area 303 is an area in which the transport path 110, which forms the background of the print medium 304 when scanned, is reflected.

In S1020, the image processing unit 201 controls the image reading device 105 to read the print medium on the transport path 110 and obtain image data. The image data read in S1020 is stored in the RAM 103 as the inspection target image 302, and is used in subsequent processing. The inspection target image 302 is composed of a background area 303 and a print medium area 304, similar to the inspection reference image 301.

In S1030, the overflow detection unit 202 detects whether the entire area of the print medium is included in each of the inspection reference image 301 and the inspection target image 302. In other words, it determines whether there is overflow of the print medium in the image. This overflow detection process is performed as follows. The overflow detection unit 202 first binarizes the image and then tracks the pixels that are the boundary between white and black. If the overflow detection unit 202 comes into contact with any of the top, bottom, left, or right edges of the image during this pixel tracking process, it determines that there is overflow of the print medium in the image. If the overflow detection unit 202 finishes tracking without coming into contact with any of the top, bottom, left, or right edges of the image as a result of pixel tracking, it determines that there is no overflow.

In S1040, the medium edge estimation unit 203 estimates the position of the print medium edge based on the overflow detection result in S1030. The details of the medium edge estimation process are described with reference to the flowchart in FIG. 3(b).

In S1510, the medium edge estimation unit 203 refers to the overflow detection result in S1030, and if overflow is detected, proceeds to S1530. If overflow is not detected, proceeds to S1520.

In S1520, the medium edge estimation unit 203 detects the four vertices 308a to 308d that make up the print medium area 304 from the inspection target image 302. This detection process is performed as follows. The medium edge estimation unit 203 first binarizes the inspection target image 302, then tracks the pixels that form the boundaries between black and white, and linearly approximates each side. This estimates four straight lines that form the outline of the print medium area 304. Next, the medium edge determination unit 202 calculates the intersections of the four straight lines. The four intersections calculated at this time are called vertices 308a to 308d.

In S1530, the medium edge estimation unit 203 calculates the number of clearly observable vertices out of the vertices 308. Figure 5 is a diagram showing an example of print medium overhang in the inspection target image. Figure 5(a) shows an example where there are no clearly observable vertices 308 due to overhang. Figure 5(b) shows an example where there is one clearly observable vertex 308, Figure 5(c) shows two, and Figure 5(d) shows an example where there are three.

Figure 6 is a diagram illustrating an example of a method for calculating the number of observable vertices (S1530). The medium edge detection unit 203 first fills in rectangular areas 313 within a few pixels from both the left and right ends of the inspection target image 302 with the background color, generating an image 314 (Figure 6 (b)).

Next, the medium end detection unit 203 binarizes the image 303 and tracks the pixels that form the boundary between black and white. The medium end detection unit 203 detects the tentative vertices 310 from this trajectory. Since image 314 is an image in which both ends are filled with the background color, the tracking process is always successful and at least four tentative vertices are detected. Then, if five or more vertices are detected, four that form approximately right angles are selected as valid tentative vertices.

In the example of FIG. 6(b), five tentative vertices 310a to 310e are detected, but because the angle formed by tentative vertex 310c is not a substantially right angle, media end detection 203 excludes it and determines the remaining four as valid tentative vertices.

The medium end detection unit 203 then refers to the coordinates of the valid tentative vertices, and counts those that are not in contact with the rectangular area 313 as clearly observable vertices. In the example of FIG. 6(b), of the four tentative vertices excluding tentative vertex 310c, only tentative vertex 310b is not in contact with the rectangular area 313. Therefore, the number of observable vertices is calculated to be "1".

In S1540, the medium end calculation unit 203 calculates the coordinates of the corresponding vertex 308 for the tentative vertex 310 that was clearly observable in S1530, using a method similar to that described in S1520.

In S1550, the medium end calculation unit 203 estimates the coordinates of the remaining vertices 308 that have not been calculated, based on the coordinates of the vertices 308 calculated in S1540. Details of the estimation process of the vertices 308 in S1550 are described below with reference to FIG. 7.

Figure 7 is a diagram illustrating an example of the print medium vertex estimation process (S1550). First, the medium edge estimation unit 203 estimates straight lines 311a and 311b that form the upper and lower sides of the print medium area 304 from the inspection target image 302.

The medium edge estimation unit 203 checks whether the coordinates of the vertex 308 detected in S1540 are on either line 311a or 311b. Then, it estimates the coordinates on either line 311a or 311b that are the paper width W away from the already detected vertex 308 as the next vertex 308. Note that the paper width W is assumed to be known by notification from the printing device 190 or user input via the UI panel 108. The medium edge estimation unit 203 performs this process for all vertices 308 detected in S1540.

If the sum of the number of vertices 308 estimated at this stage and the number of vertices 308 detected in S1540 is less than "4", the medium end estimation unit 203 performs the following process. The medium end estimation unit 203 estimates straight lines 312a, 312b that pass through the vertices 308 estimated in the process up to this point in S1550 and are perpendicular to the straight lines 311a, 311b. The medium end estimation unit 203 estimates the intersection of the straight lines 311a, 311b and the straight lines 312a, 312b that has not been detected or estimated so far as the next vertex 308.

In the example of Figure 7, the vertex of vertex 308b exists on line 311a. Based on this information, the medium end estimation unit 203 estimates the coordinate on line 311a that is the paper width W away from vertex 308b of the print medium as vertex 308a of the print medium. At this stage, since the sum of the estimated vertex 308 and the vertex 308 detected in S1540 is less than "4", the medium end estimation unit 203 estimates lines 312a and 312b. The medium end estimation unit 203 determines the intersections of lines 312a and 312b with line 311b as vertices 308d and 308c of the print medium, respectively.

In S1530, if the number of observable vertices is "0", the medium edge estimation unit 203 performs the following process. Details of this process are described below with reference to FIG. 8.

Figure 8 is a diagram illustrating an example of processing when the number of observable vertices is "0". The medium edge estimation unit 203 estimates straight lines 311a and 311b that form the upper and lower sides of the print medium area 304 from the inspection target image 302.

Next, the medium edge estimation unit 203 estimates the first vertex 308a on the straight line 311a from the length L of the image reading unit 105 and the paper width W. The length L of the image reading unit 105 is known and is stored in, for example, the ROM 103. More specifically, the coordinates on the straight line 311a that are a distance (W-L)/2 away from the left end of the inspection target image 302 are estimated as the vertex 308a. Similarly, for the vertex 308b, the coordinates on the straight line 311a that are a distance (W-L)/2 away from the right end of the inspection target image 302 are set. The medium edge estimation unit 203 estimates straight lines 312a and 312b that pass through the vertices 308a and 308b and are perpendicular to the straight line 311a, and sets the intersections of the straight lines 311b and 312a and 312b as the vertices 308d and 308c, respectively.

In S1050, the alignment unit 204 aligns the inspection reference image 301 and the inspection target image 302 based on the image area 305. Usually, the positional relationship between the medium and the image printed on the medium varies slightly for each print due to misalignment of the medium transport or printing position of the printing device. In this step, alignment is first performed at the four corners of the medium, and then alignment is performed using the image based on the alignment reference points. Details of the alignment process will be described with reference to the flowchart in FIG. 3(c). FIG. 9 is an exemplary diagram for explaining an overview of the alignment process (S1050).

Figure 9 (a-1) and (b-1) respectively show an example of an inspection reference image 301 and an inspection target image 302. In particular, the pattern area 305, alignment reference points 307a to 307c, and vertices 308a to 308d in each image are shown.

The alignment reference points 307 are points that indicate the same part of the picture area 305 in both the inspection reference image 301 and the inspection target image 302. Here, three alignment reference points 307 are set. For example, the alignment reference point 307a in the inspection reference image 301 and the alignment reference point 307a in the inspection target image 302 have different coordinates in each image, but represent the same part of the picture. The same is true for the alignment reference points 307b and 307c.

Here, it is assumed that the alignment reference points 307a to 307c are determined in advance for the image to be printed by the printing device. The alignment reference points 307a to 307c may be set manually by the user while looking at the image, or may be set automatically using image features such as SIFT and SURF. At this time, the alignment unit 204 sets templates 313a to 313c consisting of partial images of a specified size centered on each of the alignment reference points 307a to 307c for the alignment reference points 307a to 307c in the inspection reference image 301. Here, a region of "32 x 32 pixels" is set for the size of the templates 313a to 313c. (b-1) in Figure 9 shows the relationship between the templates 313a to 313c and other components.

In S1810, the alignment unit 204 aligns the print medium area 304 with the inspection reference image 301 and the inspection target image 302. The alignment processing unit 204 obtains the displacement amount X of the print medium area 304 from the inspection reference image 301 to the inspection target image 302 from the vertices 308a to 308d of the inspection reference image 301 and the inspection target image 302 detected in S1030. The displacement amount X is expressed, for example, as in formula (1). Note that here, the coordinates of the vertex 308a in the inspection reference image 301 are expressed as (x1a, y1a), and the coordinates in the inspection target image 302 are expressed as (x2a, y2a). Similarly, the coordinates of the vertex 308b in the inspection reference image 301 are expressed as (x1b, y1b), and the coordinates in the inspection target image 302 are expressed as (x2b, y2b).

X in formula (1) is a transformation matrix for affine transformation, which associates the coordinates of the inspection reference image 301 with the coordinates of the inspection target image 302, with respect to the vertex 308. The transformation matrix X represents a displacement amount resulting from a combination of translation, rotation, enlargement/reduction, and shear. The alignment unit 204 calculates the transformation matrix X by multiplying the inverse matrix of matrix A in formula (1) from the right of matrix B. Note that matrices A and B in formula (1) are not square matrices, so the alignment unit 204 finds the Moore-Penrose pseudo-inverse matrix for matrix A to calculate the transformation matrix X.

In S1820, the alignment unit 204 sets a search range for the inspection target image 302. (b-2) in FIG. 9 shows the relationship between the inspection target image 302 and the search range 314. The search range 314 is used in the template matching performed later in S1830.

The alignment unit 204 needs to set search ranges 314a to 314c that include the corresponding templates 313a to 313c in the inspection target image 302. Therefore, the alignment unit 204 sets the search ranges 314a to 314c to an area that is sufficiently larger than the size of the templates 313a to 313c, such as "64 x 64 pixels."

The alignment unit 204 also calculates the positions of the search ranges 314a to 314c using formula (2). Note that here, the coordinates of the alignment reference point 307a in the inspection reference image 301 are (u1a, v1a), the coordinates of the alignment reference point 307b are (u1b, v1b), and the coordinates of the alignment reference point 307c are (u1c, v1c).

Regarding matrix P' in formula (2), the alignment unit 204 sets the center coordinates of search area 314a to (u2a0, v2a0). Similarly, it sets the center coordinates of search area 314b to (u2b0, v2b0) and the center coordinates of search area 314c to (u2c0, v2c0). Furthermore, X is the transformation matrix calculated by formula (1). In other words, matrix P' is obtained by performing coordinate transformation on matrix P using transformation matrix X.

In S1830, the alignment unit 204 performs template matching to detect alignment reference points 307a to 307c in the inspection target image 302. Template matching is a method of calculating the similarity of the template 313 for each pixel in the search area 314 and determining the position with the highest similarity, but since this is a well-known technique, a detailed description will be omitted.

As a result, the alignment processing unit 204 obtains the coordinates of alignment reference points 307a to 307c for the inspection reference image 301 and the inspection target image 302, respectively. Here, the coordinates of alignment reference point 307a in the inspection reference image 301 are expressed as (x1a, y1a), and the coordinates in the inspection target image 302 are expressed as (x2a, y2a). Similarly, the coordinates of alignment reference point 307b in the inspection reference image 301 are expressed as (x1b, y1b), and the coordinates in the inspection target image 302 are expressed as (x2b, y2b). Similarly, the coordinates of alignment reference point 307c in the inspection reference image 301 are expressed as (x1c, y1c), and the coordinates in the inspection target image 302 are expressed as (x2c, y2c).

In S1840, the alignment unit 204 obtains the displacement X' of the pattern area 305 from the inspection reference image 301 to the inspection target image 302, which is expressed by formula (3). X' in formula (3) is the transformation matrix of the affine transformation. In other words, X' represents the displacement amount resulting from a combination of translation, rotation, enlargement/reduction, and shear from the inspection reference image 301 to the inspection target image 302, when the alignment reference point 307 is used as a reference point.

The alignment unit 204 then applies the calculated transformation matrix X' to each pixel constituting the print medium area 304 in the inspection reference image 301 to generate a transformed image 309. (c) of FIG. 9 shows how the transformed image 309, which is the result of transforming the inspection reference image 301 with X', is aligned with the inspection target image 302 through the alignment process. In other words, the positions of the alignment reference points 307a to 307c of the transformed image 309 and the inspection target image 302 match.

In S1060, the inspection unit 205 calculates the difference between the converted image 309 after the alignment process in S1050 and the inspection target image 302 for each pixel of the print medium area 304. The inspection unit 205 determines that any pixel in the calculated difference that is greater than a threshold value is a defective pixel.

If a defective pixel is detected in S1060, the inspection unit 205 determines that the inspection has failed. At this time, the CPU 101 performs control to output the print medium to the output tray 112. If no defective pixel is detected, the inspection unit 205 determines that the inspection has passed. At this time, the CPU 101 performs control to output the print medium to the output tray 111.

In S1070, if more print media with the same image formed on the surface have been printed out, the image processing unit 201 returns to S1020 and continues processing. If not, the processing ends.

As described above, according to the first embodiment, even if the print extends beyond the edge of the image to be inspected, it is possible to preferably align the inspection reference image with the image to be inspected. Specifically, if the print extends beyond the edge of the image to be inspected, the remaining vertices are estimated based on the vertices of the print that are directly observed, and then alignment is performed. This makes it possible to preferably inspect the image to be inspected for defects. In other words, even if the image width (length L) is shorter than one side of the print (paper width W) and part of the print extends beyond the edge of the image to be inspected, it is possible to preferably inspect for defects.

(Modification)
In the above description, in S1010, the image acquired by the image reading device 105 at the first time point is used as the inspection reference image 301. However, the inspection reference image 301 may be an image without defects, and the method of acquiring the inspection reference image 301 is not limited to this. For example, the present invention is also applicable to a case where the inspection reference image 301 acquired by another reading device is used. For example, the inspection reference image 301 may be read by a reading device (not shown) different from the image reading device 105, and the inspection reference image 301 may be stored in an auxiliary storage device (not shown). In this case, the image acquisition unit 201 in S1010 acquires the inspection reference image 301 from the auxiliary storage device 107 instead of the image reading device 105.

In the above description, the image acquired by the image reading device 105 after the second time point is set as the inspection target image 302 in S1020. However, the method of acquiring the inspection target image 302 is not limited to this. For example, it is also applicable to the case where the inspection target image 302 acquired by another reading device is used. For example, the inspection target image 302 may be read by a reading device (not shown) different from the image reading device 105, and the inspection target image 302 may be stored in an auxiliary storage device (not shown). In this case, the image acquisition unit 201 in S1020 acquires the inspection target image 302 from the auxiliary storage device 107 instead of the image reading device 105.

Furthermore, in the above description, in S1040, the medium edge estimation unit 203 detects or estimates the vertices of the print medium, but this is not limited to the above. The medium edge estimation unit 203 may also estimate the sides of the print medium.

Furthermore, in the above description, in S1840, the inspection reference image 301 is transformed by the transformation matrix X' to generate the transformed image 309. Alternatively, the inspection target image 302 may be transformed by the transformation matrix Y to generate the transformed image 311. Here, the transformation matrix Y is expressed by the formula (4). The matrices A and B in the formula (4) are the same as those in the formula (3). In this case, in S1070, the inspection processing unit 205 calculates the difference between the inspection reference image 301 and the transformed image 311.

The processes in steps S1030 to S1040 may also be applied to the inspection reference image 301 instead of the inspection target image 302.

Furthermore, the image acquisition unit may be any type that can read the print medium on the transport path 110 as an image and input it to the image processing device. Furthermore, the image to be read may be any type of image, such as an RGB color image, a grayscale image, or a black and white image.

Second Embodiment
In the second embodiment, a mode in which the search range is adaptively set in the alignment process (S1050) according to the vertex estimation method and the characteristics of the printing device will be described. Note that the configuration of the image processing device 100 according to the second embodiment is the same as that of the first embodiment (FIGS. 1 and 2), and therefore a description thereof will be omitted. The operation of the image processing device 100 is also substantially the same as that of the first embodiment (FIG. 3) shown in FIG. 3. In the following, the alignment process (S1050) will be described with a focus on the parts (S1820) that are different from the first embodiment, and a description of the similar parts will be omitted.

In S1820, the alignment unit 204 sets a search range for the inspection target image 302. However, while in the first embodiment the size of the search ranges 314a to 314c was set to a given size (e.g., "64 x 64 pixels"), in the second embodiment the size of the search ranges 314a to 314c is adaptively determined.

Specifically, the alignment unit 204 determines the size of the search ranges 314a to 314c based on the printing position error of the printing device 190 and the method for estimating the vertices 308.

The printing position error of the printing device 190 represents the maximum amount of variation in the image formation position where the printing device 190 prints an image on a printing medium. The value of the printing position error is Le, where Le is the number of pixels at the resolution of the inspection target image 302.

The estimation method of the vertices 308 is a parameter indicating the attributes of the four vertices (whether they are detected vertices or estimated vertices) determined in the medium edge estimation process (S1040). In S1540, the vertices that are directly observed in the inspection target image 302 are detected, so the accuracy of the coordinates of the vertices 308 is high. On the other hand, in S1550, the positions of vertices that are not clearly observed in the inspection target image 302 are estimated, so the accuracy of the coordinates of the vertices 308 is relatively low. For this reason, the alignment unit 204 controls the size of the search range 314 according to the attributes of the four vertices.

The alignment unit 204 determines the size S of the search range 314 using formula (5). T represents the size of the template 313. R is a coefficient determined according to the estimation method of the vertices 308.

Figure 10 is a diagram illustrating an example of a method for determining the coefficient R for the search range 314a. The alignment unit 204 determines the distances da to dd between the alignment reference point 307a corresponding to the target search range 314a and the four vertices 308. In Figure 10, the vertices 308 detected in S1540 are indicated by black circles, and the vertices 308 estimated in S1550 are indicated by white circles. That is, in the example of Figure 10, only the vertex 308b is detected in S1540, and the remaining three points are estimated in S1550.

The alignment unit 204 determines the coefficient R according to formula (6). In formula (6), R1 is the coefficient (here, "1.0") at the vertex 308 detected in S1540. Also, R2 is the coefficient (here, "1.5") at the vertex 308 estimated in S1550.

In other words, the alignment unit 204 changes the size S of the search range 314 according to the distance between each of the four vertices 308 and the alignment reference point 307 based on formulas (5) and (6).

As described above, the vertex clearly detected in S1540 has a high degree of certainty, so the size of the search range 314 is reduced by multiplying the alignment reference point 307 close to it by a small R1. On the other hand, the vertex estimated in S1550 has a relatively low degree of certainty, so the search range 314 is increased by multiplying the alignment reference point 307 close to it by a large R1.

In addition, in S1820, the alignment unit 204 may add the medium cutting error Ce to the calculation of S. Here, the medium cutting error Ce is the maximum value of the size variation that occurs when cutting in the manufacturing process of the print medium. In this case, formula (7) is used instead of formula (5).

As described above, according to the second embodiment, it is possible to appropriately align the inspection reference image and the inspection target image even when the printout extends beyond the edge of the inspection target image. In particular, it is possible to more appropriately set the search range in the alignment process.

Other Examples
Among the above-mentioned processing units, the overhang detection unit 202, the medium edge estimation unit 203, the alignment unit 204, the inspection unit 205, etc. may be processed using a trained model that has been machine-learned instead. In that case, for example, a plurality of combinations of input data and output data to the processing unit are prepared as training data, knowledge is acquired from them by machine learning, and a trained model that outputs output data for the input data as a result based on the acquired knowledge is generated. The trained model can be configured, for example, as a neural network model. Then, the trained model operates in cooperation with a CPU or a GPU as a program for performing the same processing as the processing unit, thereby performing the processing of the processing unit. The trained model may be updated after a certain amount of processing as necessary.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

100 Image processing device; 201 Image acquisition unit; 202 Overhang detection unit; 203 Medium edge estimation unit; 204 Alignment unit; 205 Inspection unit

Claims (12)

a first acquisition means for acquiring a reference image obtained by reading a first medium having a given image formed on a surface thereof by a printing device;
a second acquisition means for acquiring a target image obtained by reading a second medium having the given image formed on a surface thereof by the printing device;
a determining means for determining coordinates of vertices of the first medium in the reference image and coordinates of vertices of the second medium in the target image;
a first alignment means for aligning the reference image and the target image based on the vertices in the reference image and the target image determined by the determination means;
a second alignment means for aligning the reference image and the target image based on a result of template matching between the reference image and the target image aligned by the first alignment means;
an inspection means for inspecting the target image by comparing the reference image aligned by the second alignment means with the target image;
having
The determining means determines the coordinates of at least one vertex by estimation when at least one of the reference image and the target image does not include the entire area of the medium ;
The second alignment means determines a search range in the target image for performing the template matching based on the attributes of the vertices in the target image determined by the determination means.
13. An image processing device comprising: The determining means determines coordinates of four vertices of the first medium in the reference image and coordinates of at least four vertices of the second medium in the target image.
2. The image processing device according to claim 1, The determining means is
a first determination means for determining whether the reference image includes an entire area of the first medium;
a first determination means for determining, by estimation, coordinates of at least one vertex located in the area of the first medium that is not included in the reference image when the first determination means determines that the reference image does not include the entire area of the first medium;
a second determination means for determining whether the target image includes the entire area of the second medium;
a second determination means for determining, by estimation, coordinates of at least one vertex located in the area of the second medium that is not included in the target image when the second determination means determines that the entire area of the second medium is not included in the target image;
3. The image processing apparatus according to claim 2, further comprising: the first determination means determines coordinates of vertices included in the reference image among four vertices of the first medium in the reference image by detection from the reference image;
4. The image processing apparatus according to claim 3, wherein the second determination means determines coordinates of vertices included in the target image, among four vertices of the second medium in the target image, by detection from the target image. the first determination means, when it is determined by the first judging means that the reference image does not include the entire area of the first medium and the reference image does not include the four vertices of the first medium, determines by estimation the coordinates of the four vertices of the first medium based on the width of the first medium and the length of a reading unit that reads the reference image;
The image processing device described in claim 3, characterized in that when the second judgment means determines that the target image does not include the entire area of the second medium and the target image does not include the four vertices of the second medium, the second determination means estimates the coordinates of the four vertices of the second medium based on the width of the second medium and the length of the reading unit that read the target image. the first alignment means calculates a transformation matrix that associates coordinates of the reference image with coordinates of the target image based on four vertices in the reference image and four vertices in the target image, and performs coordinate transformation of the reference image and the target image based on the transformation matrix;
The image processing device according to any one of claims 2 to 5, characterized in that the second alignment means performs template matching between a partial image included in the coordinate-transformed reference image and a partial image included in the target image, and calculates a displacement amount for the alignment based on a result of the template matching. a first acquisition means for acquiring a reference image obtained by reading a first medium having a given image formed on a surface thereof by a printing device;
a second acquisition means for acquiring a target image obtained by reading a second medium having the given image formed on a surface thereof by the printing device;
a determining means for determining coordinates of vertices of the first medium in the reference image and coordinates of vertices of the second medium in the target image;
a first alignment means for aligning the reference image and the target image based on the vertices in the reference image and the target image determined by the determination means;
a second alignment means for aligning the reference image and the target image based on a result of template matching between the reference image and the target image aligned by the first alignment means;
an inspection means for inspecting the target image by comparing the reference image aligned by the second alignment means with the target image;
having
The determining means determines the coordinates of at least one vertex by estimation when at least one of the reference image and the target image does not include the entire area of the medium;
The second position matching means determines a search range in the target image for performing the template matching based on a maximum value of a fluctuation amount of an image forming position in the printing device.
13. An image processing device comprising: a first acquisition means for acquiring a reference image obtained by reading a first medium having a given image formed on a surface thereof by a printing device;
a second acquisition means for acquiring a target image obtained by reading a second medium having the given image formed on a surface thereof by the printing device;
a determining means for determining coordinates of vertices of the first medium in the reference image and coordinates of vertices of the second medium in the target image;
a first alignment means for aligning the reference image and the target image based on the vertices in the reference image and the target image determined by the determination means;
a second alignment means for aligning the reference image and the target image based on a result of template matching between the reference image and the target image aligned by the first alignment means;
an inspection means for inspecting the target image by comparing the reference image aligned by the second alignment means with the target image;
having
The determining means determines the coordinates of at least one vertex by estimation when at least one of the reference image and the target image does not include the entire area of the medium;
The second alignment means determines a search range in the target image for performing the template matching based on a maximum value of size variation that occurs when cutting the second medium in a manufacturing process.
13. An image processing device comprising: A method for controlling an image processing device that inspects an image formed on a medium, comprising the steps of:
A first acquisition step of acquiring a reference image obtained by reading a first medium having a given image formed on a surface thereof by a printing device;
a second acquisition step of acquiring a target image obtained by reading a second medium having the given image formed on a surface thereof by the printing device;
determining coordinates of vertices of the first medium in the reference image and coordinates of vertices of the second medium in the target image;
a first alignment step of aligning the reference image and the target image based on the vertices in the reference image and the target image determined in the determination step;
a second alignment step of aligning the reference image and the target image based on a result of template matching between the reference image and the target image aligned in the first alignment step;
an inspection step of inspecting the target image by comparing the reference image and the target image aligned in the second alignment step ;
Including,
In the determining step, when at least one of the reference image and the target image does not include the entire area of the medium, the coordinates of at least one vertex are determined by estimation ;
In the second alignment step, a search range in the target image for performing the template matching is determined based on the attributes of the vertices in the target image determined in the determination step.
A control method comprising: A method for controlling an image processing device that inspects an image formed on a medium, comprising the steps of:
A first acquisition step of acquiring a reference image obtained by reading a first medium having a given image formed on a surface thereof by a printing device;
a second acquisition step of acquiring a target image obtained by reading a second medium having the given image formed on a surface thereof by the printing device;
determining coordinates of vertices of the first medium in the reference image and coordinates of vertices of the second medium in the target image;
a first alignment step of aligning the reference image and the target image based on the vertices in the reference image and the target image determined in the determination step;
a second alignment step of aligning the reference image and the target image based on a result of template matching between the reference image and the target image aligned in the first alignment step;
an inspection step of inspecting the target image by comparing the reference image and the target image aligned in the second alignment step;
Including,
In the determining step, when the entire area of the medium is not included in at least one of the reference image and the target image, the coordinates of at least one vertex are determined by estimation;
In the second alignment step, a search range in the target image for performing the template matching is determined based on a maximum value of a variation amount of an image forming position in the printing device.
A control method comprising: A method for controlling an image processing device that inspects an image formed on a medium, comprising the steps of:
A first acquisition step of acquiring a reference image obtained by reading a first medium having a given image formed on a surface thereof by a printing device;
a second acquisition step of acquiring a target image obtained by reading a second medium having the given image formed on a surface thereof by the printing device;
determining coordinates of vertices of the first medium in the reference image and coordinates of vertices of the second medium in the target image;
a first alignment step of aligning the reference image and the target image based on the vertices in the reference image and the target image determined in the determination step;
a second alignment step of aligning the reference image and the target image based on a result of template matching between the reference image and the target image aligned in the first alignment step;
an inspection step of inspecting the target image by comparing the reference image and the target image aligned in the second alignment step;
Including,
In the determining step, when the entire area of the medium is not included in at least one of the reference image and the target image, the coordinates of at least one vertex are determined by estimation;
In the second alignment step, a search range in the target image for performing the template matching is determined based on a maximum value of size variation that occurs when cutting in a manufacturing process of the second medium.
A control method comprising: A program for causing a computer to execute the control method according to any one of claims 9 to 11 .

Images