JP7335545B2 - Training image data generation method, prediction model - Google Patents

Description

The present specification relates to predictive models for processing image data.

Various types of image processing have been conventionally performed. For example, there has been proposed a technique of determining the color information of the background color of image data acquired by an image reading device such as a scanner and removing the background according to the determination result (Patent Document 1). In this technique, furthermore, by performing a correction to lower the saturation of an area where the saturation is equal to or less than a predetermined value, coloring of the image due to color deviation during reading is suppressed.

JP 2018-152670 A

In recent years, techniques for processing image data using predictive models such as neural networks and support vector machines have been developed. For proper processing, a prediction model is trained using training image data. However, it was not easy to prepare learning image data for proper training.

This specification discloses techniques for generating training image data for training a predictive model that processes image data obtained by optical reading, as appropriate.

The technology disclosed in this specification can be implemented as the following application examples.

[Application Example 1] A method of generating learning image data for training a prediction model, comprising a type 1 background image which is at least a part of a sheet image obtained by optically reading a printing sheet; a first generation step of generating type 1 learning image data of a type 1 learning image that is an image in which the object is arranged on the type 1 background image, using an object image that is an image of a first storing step of storing the generated type 1 learning image data in a storage device, wherein the first generating step stores the color value of a first object pixel representing the object on the type 1 learning image; wherein the color value of the first object pixel is the color value of the pixel associated with the position of the first object pixel on the object image with a first type object coefficient greater than zero. and a value obtained by multiplying the color value of the pixel associated with the position of the first object pixel on the first background image by a first background coefficient greater than zero. and a step of determining a color value of a first background pixel indicating a portion of the type 1 learning image where the object is not arranged, wherein the color value of the first background pixel is , determining using a value obtained by multiplying a color value of a pixel associated with the position of the first background pixel on the first background image by a coefficient larger than the first background coefficient; generation method, including

According to this configuration, the type 1 learning image can appropriately represent an image obtained by optically reading the sheet on which the object is printed. Training image data for training a predictive model that processes the image data obtained can suitably be generated.

[Application Example 2] The generation method according to Application Example 1, wherein the representative color value identification identifies a representative color value representing a plurality of color values associated with a plurality of positions on the type 1 background image. a type 2 background image represented by the representative color value; a second generating step of generating type 2 learning image data; and a second storing step of storing the generated type 2 learning image data in the storage device. determining a color value of a second object pixel representing the object on the seed learning image, wherein the color value of the second object pixel is associated with the position of the second object pixel on the object image; a value obtained by multiplying the color value of the pixel obtained by multiplying the color value of the second type object pixel by a second type object coefficient larger than zero, and the color value of the pixel associated with the position of the second object pixel on the second type background image and a value obtained by multiplying a large second type background coefficient, and determining the color value of the second background pixel indicating the part where the object is not arranged on the second type learning image. determining the color value of the second background pixel relative to the color value of the pixel associated with the position of the second background pixel on the background image of the second type relative to the second background coefficient; and determining using a value obtained by multiplying by a large factor.

According to this configuration, the type 2 learning image can appropriately represent the image obtained by reducing the noise of the type 1 learning image. Training image data for training using seed training images can be suitably generated.

[Application Example 3] In the generation method according to Application Example 2, the representative color value is an average value of the plurality of color values associated with the plurality of positions on the type 1 background image. , how to generate.

According to this configuration, the type 2 background image can appropriately represent a background image with reduced noise compared to the type 1 background image, so that suitable type 2 learning image data can be generated.

[Application Example 4] In the generation method according to any one of Application Examples 1 to 3, the object is M lines (M is an integer equal to or greater than 2) parallel to the first direction, A method of generation comprising M lines aligned in a second direction perpendicular to the one direction.

According to this configuration, it is possible to appropriately generate learning image data for training a prediction model that processes an image of an object including M parallel lines.

[Application Example 5] In the generation method according to Application Example 4, the first generation step is performed Q times (where Q is an integer equal to or greater than 2) to generate Q first learning images of Q first type learning images. generating type 1 learning image data, wherein each of the Q type 1 learning images has a brightness different from the reference brightness associated with the type 1 learning image and the type 1 learning image; and the order of the line in the second direction is different from the reference position in the second direction and the reference width is different from the reference width associated with the type 1 learning image. and one or more non-reference lines that are lines having one or more of three parameter values of width and width.

According to this configuration, it is possible to appropriately generate learning image data for training a prediction model that processes images of objects that include one or more non-reference lines.

[Application Example 6] In the generation method according to Application Example 5, the Q type 1 learning images include one or more non-reference lines having the luminance different from the reference luminance. a type 1 learning image, at least one type 1 learning image including at least one non-reference line arranged at the position in the second direction different from the reference position in the second direction, and the reference width; and one or more Type 1 training images including one or more non-reference lines having different said widths.

According to this configuration, it is possible to appropriately generate learning image data for training a prediction model that processes an image of an object when each of the brightness, the position in the second direction, and the width can change.

[Application Example 7] In the generation method according to Application Example 5 or 6, the Q type 1 learning images include a plurality of different total numbers of the non-reference lines included in the type 1 learning images. A generation method, including a type 1 training image.

According to this configuration, it is possible to appropriately generate learning image data for training a predictive model that processes an image of an object when the total number of non-reference lines can change.

[Application Example 8] The generation method according to any one of Application Examples 1 to 7, wherein the object includes a character.

According to this configuration, it is possible to appropriately generate learning image data for training a prediction model that processes images of objects including characters.

[Application Example 9] Based on image data of a print sheet image, which is an image obtained by optically reading a sheet on which an object is printed, related information that is information related to the object printed on the sheet is generated. A predictive model for causing a computer to function so as to perform a first type background image that is at least a part of a sheet image obtained by optically reading a sheet for printing, and an object image that is an image of an object , which is trained using the first type learning image data of a first type learning image which is an image in which the object is arranged on the first type background image, The first type learning image data is a step of determining a color value of a first object pixel representing the object on the first type learning image, wherein the color value of the first object pixel is determined on the object image. a value obtained by multiplying the color value of the pixel associated with the position of the first object pixel in by a first-type object coefficient larger than zero, and the position of the first object pixel on the first-type background image and a value obtained by multiplying the color value of the pixel associated with by a type 1 background coefficient greater than zero; determining a color value of a first background pixel indicating a portion not to be colored, wherein the color value of the first background pixel is a pixel associated with the position of the first background pixel on the first background image; and determining using a value obtained by multiplying the color value of by a coefficient larger than the first type background coefficient.

It should be noted that the technology disclosed in the present specification can be implemented in various aspects. and processing device, determination method and determination device using image data, computer programs for realizing the functions of those methods or devices, recording media recording the computer programs (for example, non-temporary recording media), etc. can be realized in the form

It is an explanatory view showing a data processor of an example. (A) and (B) are explanatory diagrams showing examples of nozzle patterns NP. FIG. 4 is an explanatory diagram of a learning model 235; 6 is a flowchart illustrating an example of learning data generation processing; 4A is a flowchart showing an example of reference base pattern image generation processing; (B) is an explanatory diagram showing an example of a reference base pattern image; 8A is a flowchart showing an example of non-reference base pattern image generation processing; (B) is an explanatory diagram showing an example of a non-reference base pattern image; (A) is a flowchart showing an example of background image generation processing. (B) is an explanatory diagram showing an example of a type 1 background image. (A)-(D) are explanatory diagrams of noise-containing pattern images. (A) to (D) are explanatory diagrams of noise reduction pattern images. (E) is a graph showing an example of contrast enhancement processing. (A)-(C) are explanatory diagrams of the training process. (A)-(D) are explanatory diagrams of the position of the line PL. 4 is a flowchart showing an example of image processing; FIG. 10 is an explanatory diagram of another example of a learning model; 9 is a flow chart showing another example of learning data generation processing. (A) to (D) are explanatory diagrams of learning data. FIG. 10 is an explanatory diagram of another example of a learning model;

A. First example:
A1. Device configuration:
FIG. 1 is an explanatory diagram showing the data processing apparatus of the embodiment. In this embodiment, the MFP 200 is used for data processing. The MFP 200 has a control section 205 , a printing section 260 and a reading section 300 . The control unit 205 has a processor 210 , a storage device 215 , a display unit 240 , an operation unit 250 and a communication interface 270 . These elements are connected to each other via buses. Storage 215 includes volatile storage 220 and nonvolatile storage 230 .

The processor 210 is a device that performs data processing, such as a CPU. The volatile memory device 220 is, for example, a DRAM, and the non-volatile memory device 230 is, for example, a flash memory.

The nonvolatile storage device 230 stores a first program 231, a second program 232, a third program 233, a learning model 235, and filter data FLD. In this embodiment, the learning model 235 is a prediction model of an artificial neural network, and is a machine learning model trained by a training process to be described later. In this example, learning model 235 is a program module. Details of the learning model 235 will be described later.

The processor 210 stores various intermediate data used for executing the first program 231, the second program 232, the third program 233, and the learning model 235 in the storage device 215 (for example, the volatile storage device 220 , non-volatile storage device 230).

The display unit 240 is a device that displays images, such as a liquid crystal display, an organic EL display, and an LED display. The operation unit 250 is a device that receives user operations, such as buttons, levers, and a touch panel overlaid on the display unit 240 . The communication interface 270 is an interface for communicating with other devices (eg, USB interface, wired LAN interface, IEEE802.11 wireless interface).

The printing unit 260 is a device that prints an image on paper (an example of a print medium) using a predetermined method (for example, a laser method or an inkjet method).

The reading unit 300 is a reading device that optically reads an object such as a document using photoelectric conversion elements such as CCD and CMOS. The reading unit 300 generates read data representing a read image (referred to as “read image”). In this embodiment, the read data is grayscale bitmap data representing a read image. In this embodiment, the luminance value of bitmap data is represented by 256 gradations from 0 to 255. FIG.

The figure shows a printing apparatus 100 different from the multifunction machine 200 . The printing device 100 is an inkjet printing device that prints an image on paper. In this embodiment, the printing device 100 utilizes four types of ink: cyan, magenta, yellow, and black. The printing device 100 includes a print head 110 having multiple nozzles for ejecting ink droplets. The printhead 110 can have various nozzle defects. In order to identify the defect, the printing apparatus 100 prints the nozzle pattern NP on the sheet SH2. In this embodiment, the sheet SH2 is a sheet of paper. However, other types of sheets such as cloth, film, etc. may be used.

FIGS. 2A and 2B are explanatory diagrams showing examples of the nozzle pattern NP. FIG. 2A shows a case where the print head 110 does not have a problem, and FIG. 2B shows a case where the print head 110 has a problem. Each figure shows a nozzle pattern NP printed on the sheet SH2, a print head 110, a plurality of nozzles Nz provided in the print head 110, and mutually perpendicular directions D1 and D2. . The drawing shows M nozzles Nz (M is an integer equal to or greater than 2) for ejecting one type of ink. The nozzle pattern NP is printed using M nozzles Nz of one type of ink to be processed.

The first direction D1 is the scanning direction of the print head 110 . The printing device 100 includes a moving device (not shown) that moves the print head 110 in a direction parallel to the first direction D1. The print head 110 forms ink dots on the sheet by ejecting ink droplets from the nozzles Nz while moving with respect to the sheet in a direction parallel to the first direction D1. The second direction D2 is the sheet conveying direction. The printing apparatus 100 includes a conveying device (not shown) that moves the sheet in the second direction D2. The printing apparatus 100 prints an image on a sheet by repeating a scanning process of forming ink dots while moving the print head 110 and a conveying process of conveying the sheet.

The positions in the second direction D2 are different among the M nozzles Nz. The positions of the M nozzles Nz in the second direction D2 are arranged at equal intervals in the second direction D2. The nozzle pattern NP is composed of M lines PL parallel to the first direction D1. The M lines PL are arranged in a second direction D2 perpendicular to the first direction D1. One line PL is printed in one scanning process by one nozzle Nz. The drawing shows the central axis AxP of each line PL. Each line PL has a symmetrical shape with respect to the central axis AxP.

When M nozzles have no defects (FIG. 2A), the width of the lines PL in the second direction D2 and the color of the lines PL are the same between the M lines PL. Also, the M lines PL are arranged at regular intervals in the second direction D2. Hereinafter, the line PL printed by the non-defective nozzles Nz will also be referred to as a reference line.

In the example of FIG. 2B, three nozzles Nza, Nzb, and Nzc have defects. The first line PLa printed by the first nozzles Nza is printed at a position shifted in the direction opposite to the second direction D2. Such misalignment of ink dots can occur due to various causes such as defects in the processing of the nozzles Nz. The width of the second line PLb printed by the second nozzle Nzb is narrower than the width of the normal line PL (that is, the ink dot size is smaller). Such a deviation in ink dot size can be caused by various causes such as malfunction of a nozzle driving device (piezo element, heater, etc.) for ejecting ink droplets. The color of the third line PLc printed by the third nozzle Nzc is lighter than the color of the normal line PL. Such color shifts can result from a variety of causes, such as mixing of the two types of ink due to ink penetration within the printhead 110 . Thus, if the configuration (position, width, density, etc.) of the line PL is different from the configuration of the reference line, the nozzle Nz that printed the line PL has a defect. A line PL having a configuration different from that of the reference line is hereinafter also referred to as a non-reference line.

The sheet SH2 on which the nozzle pattern NP is printed is read by the reading unit 300 (FIG. 1) in order to specify the configuration (position, width, density, etc.) of each of the plurality of lines PL. For example, when manufacturing the printing apparatus 100, an operator can identify the configuration of each of the plurality of lines PL by observing the read image of the nozzle pattern NP, and can detect defects in the nozzles Nz. Here, if the nozzle pattern NP is printed using thin ink, the ink cost for printing the nozzle pattern NP can be reduced. In this case as well, the visibility of the nozzle pattern NP can be improved by performing brightness correction (for example, contrast enhancement processing) on the read image. However, simple brightness correction (eg, brightness correction that adjusts the tone curve) can emphasize noise in addition to the nozzle pattern NP. For example, the pattern of fibers in the sheet of paper SH2 and noise caused by digital image processing can be enhanced. The learning model 235 (FIG. 1) is a model for generating noise-reduced image data of the nozzle pattern NP from the read data of the nozzle pattern NP.

FIG. 3 is an explanatory diagram of the learning model 235. As shown in FIG. Input data 950 representing an input image 950i of the nozzle pattern NP is input to the learning model 235 . The input data 950 is data obtained by executing crop processing for cutting out a portion indicating the nozzle pattern NP from the read data of the sheet SH2 (FIG. 1) (the cropped portion is predetermined). As will be described later, learning model 235 uses input data 950 to generate output data 970 representing output image 970i of the same nozzle pattern NP.

The input image 950i and the output image 970i are rectangular images having two sides parallel to the horizontal direction Dx and two sides parallel to the vertical direction Dy perpendicular to the horizontal direction Dx. These images 950i and 970i are represented by respective color values (luminance values in this embodiment) of a plurality of pixels arranged in a matrix along the horizontal direction Dx and the vertical direction Dy. The size of the input image 950i (the number of pixels IW in the horizontal direction Dx and the number of pixels IH in the vertical direction Dy) is determined in advance. The size of the output image 970i is the same as the size of the input image 950i. Directions D1 and D2 in the drawing indicate directions D1 and D2 of the printing apparatus 100 with respect to the nozzle pattern NP. Hereinafter, it is assumed that the input data 950 is generated such that the first direction D1 is approximately the same as the vertical direction Dy, and the second direction D2 is approximately the same as the horizontal direction Dx.

In this example, learning model 235 consists of one convolutional layer 310 . Input data 950 is input to the convolutional layer 310 . The convolution layer 310 performs a convolution process using one convolution filter FL (hereinafter the convolution filter FL is also simply referred to as "filter FL"). The convolution process by the convolution layer 310 is a process of calculating a correlation value indicating the correlation between the input data and the filter FL. The horizontal size Fx in the drawing indicates the size (the number of pixels) of the filter FL in the horizontal direction Dx, and the vertical size Fy indicates the size (the number of pixels) of the filter FL in the vertical direction Dy. In this embodiment, Fx=5 and Fy=5. The filter FL includes Fx×Fy weights W. The number (i, j) in parentheses attached to each weight W in the figure indicates the position i in the horizontal direction Dx and the position j in the vertical direction Dy within the filter FL.

The convolutional layer 310 obtains from the input data 950 a list of “Fx×Fy” luminance values of “Fx×Fy” pixels in the portion corresponding to the position of the filter FL. The convolution layer 310 calculates the inner product of the obtained list and the list of “Fx×Fy” weights of the filter FL. In this embodiment, the bias addition process is omitted. Convolutional layer 310 inputs the inner product to the activation function. The calculated value of the activation function indicates the correlation value. In this example, the activation function is the identity function. The calculation of the correlation value is performed at each of a plurality of positions of the filter FL while sliding the filter FL. The convolutional layer 310 produces bitmap data indicating correlation values at each of the multiple positions of the filter FL. The generated bitmap data is output data 970 . The position of the pixel in the output image 970i indicates the position of the filter FL. In this embodiment, convolution processing is performed so that the size of the output image 970i (that is, the number of pixels in the horizontal direction Dx and the number of pixels in the vertical direction Dy) is the same as the size of the input image 950i. Therefore, the stride (the amount by which the filter is slid) is one. Also, pixels are supplemented by zero padding around the input image 950i.

A2. Training image data generation process:
FIG. 4 is a flowchart showing an example of processing for generating learning data used for training the learning model 235 . In this embodiment, noise-containing data, which is image data of a nozzle pattern NP containing noise, and noise-reduced data, which is image data of the same nozzle pattern NP with reduced noise, are generated. In training, which will be described later, the learning model 235 generates output data by inputting noise-containing data to the learning model 235 . The weights of the filter FL are adjusted so that the difference between this output data and the noise reduction data is small. The trained learning model 235 can use the noisy nozzle pattern NP image data to generate noise-reduced image data for the same nozzle pattern NP. In this embodiment, processor 210 ( FIG. 1 ) generates training image data according to first program 231 .

In S105, the processor 210 (FIG. 1) acquires read data of the sheet SH1 by causing the reading unit 300 to read the sheet SH1. As the sheet SH1, a blank sheet on which the nozzle patterns NP are not printed, which is a sheet that can be used for printing the nozzle patterns NP, is used. Note that the nozzle pattern NP can be printed on various types of sheets such as plain paper and recycled paper. Also, the properties may differ between multiple sheets of the same type. For example, different sheets of plain paper may have different brightness. In S105, a plurality of pieces of read data are obtained by reading a plurality of sheets having different characteristics (hereinafter, a read image of the read data is also referred to as a sheet image). The read data is used as image data for a background image, which will be described later. Processor 210 stores the acquired read data in nonvolatile storage device 230 .

In S110, processor 210 selects the type of nozzle pattern NP from standard and non-standard. Reference indicates a nozzle pattern NP composed only of reference lines, and non-reference indicates a nozzle pattern NP including one or more non-reference lines. Processor 210 selects an incomplete type. At S120, the processor 210 identifies the selected type. If a reference is selected, processor 210 performs a reference base pattern image generation process at S130 and proceeds to S150. If non-reference is selected, processor 210 performs non-reference base pattern image generation processing at S140 and proceeds to S150.

FIG. 5A is a flow chart showing an example of the reference base pattern image generation process of S130 of FIG. FIG. 5B is an explanatory diagram showing an example of the reference base pattern image. The reference base pattern image 810 is an image of the nozzle pattern NP composed only of reference lines and has the same size as the input image 950i (FIG. 3). In the example of FIG. 5B, the reference base pattern image 810 shows M lines PL0-PLk. The numbers following "PL" in the line code are numbers that identify individual lines PL, and are assigned in order from zero so as to line up in ascending order in the horizontal direction Dx (where k=M− 1). The central axes Ax0-Axk are central axes of the lines PL0-PLk, respectively. The central axis Ax0-Axk is parallel to the vertical direction Dy. The line PL0-PLk has a rectangular shape centered on the central axis Ax0-Axk, and is symmetrical about the central axis Ax0-Axk.

In S210 (FIG. 5A), the processor 210 initializes the luminance values of all pixels of the reference base pattern image 810 to white (255) and initializes the target number n to zero.

At S215, the processor 210 randomly (for example, according to a random number) determines a reference line width nW from within a predetermined reference width range. The reference width range is the allowable range of the width of the reference line, and is the range from the lower limit width to the upper limit width, which is greater than zero. The reference width range may be set, for example, to the distribution range of the widths of the reference lines on the actual input images 950i (FIG. 3).

At S220, processor 210 randomly determines a reference line color nD from within a predetermined reference color range. The reference color range is the allowable range of the color (here, luminance value) of the reference line, and is the range of zero or more and the upper limit luminance value or less. The reference color range may be set, for example, to a distribution range of colors of a plurality of reference lines on a plurality of real input images 950i.

In S225, the processor 210 determines the reference position nS of the left edge line in the horizontal direction Dx. The left edge line is the edge line PL0 on the left side (that is, the side opposite to the horizontal direction Dx) in the reference base pattern image 810 (FIG. 5B). In this embodiment, the reference position nS is randomly selected from within a predetermined reference position range. The reference position range is the allowable range of the position of the reference line, and is the range from the first position to the second position. The reference position range may be set, for example, to a distribution range of the positions of the left edge lines on the actual input images 950i.

In S260, processor 210 calculates left end position nL and right end position nR of attention line PLn specified by attention number n. FIG. 5B shows the left end position nL and the right end position nR of the line PL0. The left end position nL is the end position on the side opposite to the horizontal direction Dx, and the right end position nR is the end position in the horizontal direction Dx. In this embodiment, M lines PL having the reference line width nW are arranged side by side at regular intervals in the horizontal direction Dx. A positional difference nP in the lateral direction Dx between two adjacent lines PL is predetermined. From the above, nL=nS+nP.times.n and nR=nS+nP.times.n+nW.

In S265, the processor 210 paints the inside of the line of interest PLn on the reference base pattern image 810 with the reference line color nD. Specifically, the luminance values of a plurality of pixels included in the range from the left edge position nL to the right edge position nR are set as the reference line color nD.

In S290, the processor 210 adds 1 to the attention number n. At S295, the processor 210 determines whether or not all M lines PL have been processed. If unprocessed lines PL remain (S295: No), the processor 210 proceeds to S260. If the processing of M lines PL has been completed (S295: Yes), the processor 210 stores the reference base pattern image data representing the reference base pattern image in the storage device 215 (eg, the non-volatile storage device 230) in S297. Store. Processor 210 then ends the process of FIG. 5A, that is, the process of S130 in FIG.

FIG. 6A is a flow chart showing an example of the non-reference base pattern image generation process of S140 of FIG. FIG. 6B is an explanatory diagram showing an example of a non-reference base pattern image. The non-reference base pattern image 820 is an image of the nozzle pattern NP including non-reference lines, and has the same size as the input image 950i (FIG. 3). In FIG. 6(B), like in FIG. 5(B), a plurality of lines are labeled PL0-PLk. In the example of FIG. 6B, three lines PL1, PL2, and PL3 are non-reference lines (details will be described later).

S210, S215, S220, and S225 in FIG. 6A are the same as S210, S215, S220, and S225 in FIG. 5A, respectively. At S240, the processor 210 determines the line width pW of each of the M lines PL. In this example, the processor 210 randomly selects the first non-reference line from the M lines PL. The line width pW of the lines not selected as the first non-reference line is set to the reference line width nW. The line width of the first non-reference line is randomly selected from within a predetermined non-reference width range. The non-reference width range is a range that is greater than or equal to zero and less than or equal to the non-reference upper limit width. A width of zero indicates no line. The non-reference upper limit width may be the same as the upper limit width of the reference width range described in S215 of FIG. 5A. In FIG. 6B, the width pW of the first non-reference line PL2 is different from the reference line width nW.

At S245, the processor 210 determines the color pD (here, the luminance value) of each of the M lines. In this example, processor 210 randomly selects the second non-reference line from the M lines. The color pD of lines not selected as second non-reference lines is set to the reference line color nD. The second non-reference line color pD is randomly selected from within a predetermined non-reference color range. The non-reference color range is a range between zero and the non-reference upper limit brightness value. The non-reference upper limit luminance value may be the same as the upper limit luminance value of the reference color range described in S220 of FIG. 5A, or may be greater than the upper limit luminance value. In FIG. 6B, the color pD of the second non-reference line PL3 is different from the reference line color nD.

At S250, the processor 210 determines the amount of misalignment pZ in the lateral direction Dx for each of the M lines. In this example, processor 210 randomly selects a third non-reference line from the M lines. The displacement amount pZ of the lines not selected as the third non-reference line is set to zero. The positional displacement amount pZ of the third non-reference line is randomly selected from within a predetermined displacement amount range. The deviation range is from -XB to +XB. The value XB is determined experimentally beforehand. In FIG. 6B, the positional deviation amount pZ of the third non-reference line PL1 is not zero.

In S260a, processor 210 calculates left end position nL and right end position nR of attention line PLn specified by attention number n. The difference from S260 in FIG. 5A is that the line width pW(n) of the target number n and the positional deviation amount pZ(n) are considered. Specifically, nL=nS+nP*n+pZ(n) and nR=nS+nP*n+pW(n)+pZ(n). In the example of FIG. 6B, the position of the third non-reference line PL1 in the horizontal direction Dx is displaced according to the displacement amount pZ. Also, the width of the first non-reference line PL2 is set to a width pW different from the reference line width nW.

In S265a, the processor 210 paints the inside of the line of interest PLn on the non-reference base pattern image 820 with the color pD(n) of the line of interest PLn. Specifically, the luminance values of a plurality of pixels included in the range from the left end position nL to the right end position nR are set as the color pD(n) of the line of interest PLn.

In addition, in S240, S245, and S250, the selection of the non-reference line is performed independently. Therefore, for one non-reference line, two or more of the three parameters of width pW, color pD, and displacement amount pZ can differ from the corresponding parameter values of the reference line.

S290 and S295 that follow are the same as S290 and S295 in FIG. 5A, respectively. If unprocessed lines PL remain (S295: No), the processor 210 proceeds to S260a. If the processing of M lines PL has been completed (S295: Yes), the processor 210 stores the non-reference base pattern image data representing the non-reference base pattern image in the storage device 215 (eg, the non-volatile storage device 230) in S297a. ). Processor 210 then ends the process of FIG. 6A, that is, the process of S140 in FIG.

At S150 (FIG. 4), processor 210 performs a smoothing process on the base pattern image data generated at S130 or S140 to generate smoothed base pattern image data. Smoothing processing is also called blurring processing. In the actual read image of the nozzle pattern NP, the outline of the nozzle pattern NP is blurred. S150 is a process for bringing the base pattern image closer to the actual read image. In this embodiment, smoothing processing using a mean value filter is performed. The smoothing process may be various other processes (eg, other smoothing filters such as median filters, Gaussian filters, etc. may be used).

At S155, the processor 210 executes background image generation processing. FIG. 7A is a flowchart showing an example of background image generation processing. In S310, the processor 210 randomly selects one target read image from a plurality of read images of a plurality of sheets (FIG. 4: S105). At S315, processor 210 extracts a background image, which is a portion of the same size as input image 950i (FIG. 3), from the target read image. The position of the background image within the target read image is randomly determined. Hereinafter, the background image extracted from the read image is also referred to as the first type background image, and the image data of the first type background image is also referred to as the first type background image data.

FIG. 7B is an explanatory diagram showing an example of the first type background image. The drawing shows two read images SHR1 and SHR2. Background images BG1 to BG4 are extracted from these read images SHR1 and SHR2. Note that one background image is extracted in one process of FIG.

At S160 (FIG. 4), the processor 210 generates noisy pattern image data for the noisy pattern image. A noise-containing pattern image is generated using the base pattern image generated in S130 or S140 and the first type background image. FIGS. 8A to 8D are explanatory diagrams of noise-containing pattern images. FIG. 8A shows a noise-containing reference pattern image 850a generated from the reference base pattern image 810a and the first type background image 830a. FIGS. 8B and 8C show noise-containing non-reference pattern images 850b and 850c generated from non-reference base pattern images 820b and 820c and first type background images 830b and 830c, respectively. . A luminance value Pt (x, y) in the drawing is the luminance value of the pixel (x, y) of the base pattern images 810a to 810c (referred to as base pattern luminance value Pt). The value x is the pixel position in the horizontal direction Dx in the image, and the value y is the pixel position in the vertical direction Dy in the image. For the luminance values Pu and Pn of other images, (x, y) also indicates the pixel position within the image. The brightness value Pu is the brightness value of the first type background images 830a to 830c (referred to as the first type background brightness value Pu). The brightness value Pn is the brightness value of the noise-containing reference pattern images 850a-850c (referred to as the first type target brightness value Pn).

FIG. 8D shows a formula for calculating the type 1 target luminance value Pn. When the base pattern luminance value Pt(x, y) is 255, that is, when the pixel (x, y) indicates the background instead of the line PL, the first type target luminance value Pn(x, y) is the background luminance It is set to the same value as the value Pu(x,y). That is, the weight of the background luminance value Pu is 1, and the weight of the base pattern luminance value Pt is zero.

If the base pattern intensity value Pt(x,y) is less than 255, i.e. if the pixel (x,y) represents the line PL rather than the background, then Pn(x,y) is k1*Pu(x,y )+k2×Pt(x,y). Weights k1 and k2 are both greater than zero. In this embodiment, k1+k2=1. For example, k1=0.75 and k2=0.25.

The processor 210 determines the luminance value of each pixel of the noise-containing pattern image according to the calculation formula of FIG. 8(D). In addition to the base pattern luminance value Pt, the background luminance value Pu is used to calculate the type 1 target luminance value Pn of the pixel indicating the line PL. Therefore, the noise-containing pattern image can appropriately reproduce the read image of the sheet on which the base pattern image is printed.

At S165 (FIG. 4), the processor 210 stores the generated noise-containing pattern image data in the storage device 215 (non-volatile storage device 230 in this embodiment).

At S170, the processor 210 identifies representative color values of the type 1 background image generated at S155. In this embodiment, the processor 210 calculates the average luminance value of all pixels included in the type 1 background image as the representative value.

At S175, the processor 210 generates noise reduction pattern image data for the noise reduction pattern image. A noise reduction pattern image is generated using the base pattern image generated in S130 or S140 and the second type background image represented by the representative color values identified in S170. 9A to 9D are explanatory diagrams of noise reduction pattern images. FIG. 9A shows a noise reduction pattern image 851a generated from the reference base pattern image 810a and the second type background image 830am. FIGS. 9B and 9C show noise-reduced non-reference pattern images 851b and 851c generated from non-reference base pattern images 820b and 820c and second type background images 830bm and 830cm, respectively. . The color values of all the pixels of the second type background image 830am-830cm are set to the representative color values. The second type background luminance value Pum(x, y) is the luminance value of the pixel (x, y) of the second type background image 830am-830cm, and is the same as the representative color value. The luminance value Pc(x, y) is the luminance value of the pixel (x, y) of the noise reduction reference pattern images 851a to 851c (referred to as the second type target luminance value Pc).

FIG. 9D shows a formula for calculating the second type target luminance value Pc. When the base pattern luminance value Pt(x, y) is 255 (the pixel (x, y) indicates the background), the type 2 target luminance value Pc(x, y) is the type 2 background luminance value Pum ( x, y), that is, the representative color value (the average value of the background luminance values Pu in this embodiment). Thus, the weight of the second type background luminance value Pum is 1, and the weight of the base pattern luminance value Pt is zero.

If the base pattern luminance value Pt(x,y) is less than 255 (pixel (x,y) indicates line PL), Pc(x,y) is k1×Pum(x,y)+k2×Pt( x, y). The weight k1 of the type 2 background luminance value Pum is the same as the weight k1 of the type 1 background luminance value Pu in FIG. 8(D). The weight k2 of the base pattern luminance value Pt is the same as the weight k2 of the base pattern luminance value Pt in FIG. 8(D).

The processor 210 determines the luminance value of each pixel of the noise reduction pattern image according to the calculation formula of FIG. 9(D). In addition to the base pattern luminance value Pt, the second type background luminance value Pum (that is, the representative color value) is used to calculate the second type target luminance value Pc of the pixels representing the line PL. Therefore, the noise reduction pattern image can appropriately represent an image obtained by reducing noise (for example, the pattern of the fibers of the sheet) from the read image of the sheet on which the base pattern image is printed.

At S180 (FIG. 4), the processor 210 performs contrast enhancement processing of the noise reduction pattern image data. FIG. 9E is a graph showing an example of contrast enhancement processing. The horizontal axis indicates the type 2 target luminance value Pc before adjustment, and the vertical axis indicates the type 2 target luminance value Pcc after adjustment. This correspondence relationship is configured such that when the unadjusted luminance value Pc changes from the first threshold value Va to the second threshold value Vb, the adjusted luminance value Pcc changes from zero to 255 (zero< Va<Vb<255). If Pc<Va, the adjusted luminance value Pcc is set to zero. The adjusted brightness value Pcc is set to 255 if Vb<Pc. It should be noted that various processes for emphasizing the contrast may be executed without being limited to the process using the correspondence shown in FIG. 9(E).

Contrast-enhanced noise-reduced pattern images 860a-860c are shown in FIGS. 9A-9C. The distinction between the line PL and the background is easy because the contrast is enhanced.

At S185 (FIG. 4), processor 210 associates the contrast-adjusted adjusted noise-reduced pattern image data with the noise-containing pattern image data in storage device 215 (in this example, non-volatile storage device 230). store in

At S190, the processor 210 determines whether or not the generation of the image data is completed. In this embodiment, the conditions for completion are that the number of times the image data of the reference nozzle pattern NP is generated is equal to or greater than a predetermined reference number of times P (P is an integer equal to or greater than 2), and that the non-reference nozzle pattern NP is equal to or greater than a predetermined non-reference number of times Q (Q is an integer equal to or greater than 2). Note that the completion condition may be other various conditions.

If it is determined that image data generation has not been completed (S190: No), the processor 210 proceeds to S110. If it is determined that generation of image data has been completed (S190: Yes), processor 210 terminates the processing of FIG. Multiple sets of noise-containing pattern image data and adjusted noise-reduced pattern image data are stored in non-volatile storage 230 .

As described above, in the process of generating learning image data according to this embodiment, noise-containing pattern image data is generated in S155 and S160 of FIG. As described with reference to FIGS. 8A to 8C, the noise-containing pattern images 850a to 850c consist of the nozzle patterns NP (plurality of lines PL) arranged on the first type background images 830a to 830c. It is an image. As described in S160, the noise-containing pattern image is generated using the type 1 background image and the base pattern image. As described in S105 and S155, the first type background image is at least part of the read image (that is, the sheet image) obtained by optically reading the printing sheet SH1. As described in S130 and S140, the base pattern image is the image of the nozzle pattern NP. At S<b>165 , the noise-containing pattern image data is stored in the storage device 215 .

As described with reference to FIG. 8D, the process of determining the color values (here, luminance values) of the pixels of the noise-containing pattern image includes two processes. The first processing is processing when the target pixel indicates the nozzle pattern NP (lower stage of FIG. 8(D)). In the first process, the luminance value Pn(x, y) of the target pixel is set to the weighted sum of the type 1 background luminance value Pu(x, y) and the base pattern luminance value Pt(x, y). be done. The weight k1 by which the type 1 background luminance value Pu is multiplied is greater than zero, and the weight k2 by which the base pattern luminance value Pt is multiplied is greater than zero. The second processing is processing when the target pixel indicates a portion where the nozzle pattern NP is not arranged (upper part of FIG. 8(D)). In the second process, the luminance value Pn(x, y) of the target pixel is added to the type 1 background luminance value Pu(x, y) more than the weight k1 of the type 1 background luminance value Pu in the first process. It is set to a value obtained by multiplying by a large weight (1 in this embodiment). As described above, the noise-containing pattern image can appropriately express the image obtained by optically reading the sheet on which the nozzle pattern NP is printed. The learning model 235 is a model for processing image data obtained by actually optically reading a sheet on which the nozzle pattern NP is printed. In the example of FIG. 4, learning image data for training such learning model 235 can be generated appropriately.

Also, in S170, a representative color value representing a plurality of luminance values associated with a plurality of positions on the background image generated in S155 is specified. At S175, noise reduction pattern image data is generated, and at S180, the contrast of the noise reduction pattern image data is adjusted. As described with reference to FIGS. 9A to 9C, the noise reduction pattern images 851a to 851c are images in which the nozzle patterns NP (plurality of PL) are arranged on the second type background images 830am to 830cm. is. As described in S175, the noise reduction pattern image is generated using the second type background image and the base pattern image. As described in S170 and S175, the second type background image is an image represented by representative color values. At S<b>185 , the noise reduction pattern image data adjusted at S<b>180 is stored in the storage device 215 .

As described with reference to FIG. 9D, the process of determining the color value (here, luminance value) of the pixel of the noise reduction pattern image includes two processes. The first processing is processing when the target pixel indicates the nozzle pattern NP (lower stage of FIG. 9D). In the first process, the luminance value Pc(x, y) of the target pixel is set to the weighted total value of the second type background luminance value Pum(x, y) and the base pattern luminance value Pt(x, y). be done. The weight k1 by which the second type background luminance value Pum is multiplied is greater than zero, and the weight k2 by which the base pattern luminance value Pt is multiplied is greater than zero. The second processing is processing when the target pixel indicates a portion where the nozzle pattern NP is not arranged (upper part of FIG. 9(D)). In the second process, the luminance value Pc(x, y) of the target pixel is added to the second type background luminance value Pum(x, y) more than the weight k1 of the second type background luminance value Pum in the first process. It is set to a value obtained by multiplying by a large weight (1 in this embodiment). As described above, the noise-reduced pattern image can appropriately represent an image obtained by reducing the noise of the noise-containing pattern image. In the example of FIG. 4, training image data for training with noisy images and noise-reduced images can be suitably generated.

Also, as described in S170, in this embodiment, the representative color value is the average value of multiple color values of multiple pixels on the type 1 background image. Therefore, since the second type background image can appropriately represent a background image with noise reduced compared to the first type background image, appropriate noise reduction pattern image data can be generated. Note that the representative color value is not limited to the average value, and may be various values specified using a plurality of color values such as the median value and mode value. Also, the representative color value may be specified using some of the plurality of pixels evenly selected from the first type background image, instead of all the pixels of the first type background image.

As described with reference to FIG. 5B and the like, the nozzle pattern NP is N lines PL (N is an integer equal to or greater than 2) parallel to the first direction D1 and perpendicular to the first direction D1. It includes N lines PL aligned in the second direction D2. In this embodiment, such learning image data for the nozzle pattern NP can be generated appropriately.

Also, as described in S190, the generation of the noise-containing pattern image and the noise reduction pattern image of the non-reference nozzle pattern NP is performed Q times (Q is an integer of 2 or more). As described with reference to FIG. 6A, the reference line included in the noise-containing pattern image is associated with the reference line width nW, the reference line color nD, the reference position nS, and the zero displacement amount pZ. . The reference line width nW, the reference line color nD, and the reference position nS are associated with the noise-containing pattern image and determined for each noise-containing pattern image. The position of the reference line in the horizontal direction Dx is determined based on the reference position nS and the target number n (that is, the number of the line PL in the horizontal direction Dx), as shown in S260a of FIG. 6A. As described in FIGS. 6A, 6B, etc., each of the Q noise-containing pattern images includes one or more non-reference lines. A non-baseline has one or more of the following three parameter values.
(1) Color pD (here, luminance) different from reference line color nD
(2) Position different from reference position in horizontal direction Dx associated with reference position nS and target number n (3) Width pW different from reference line width nW
Therefore, this embodiment can appropriately generate learning image data for training the learning model 235 that processes an image of a nozzle pattern having defects as shown in FIG. 2(A).

Further, by the processing of FIG. 6A, the Q noise-containing non-reference pattern images include the following three types of noise-containing pattern images.
(1) A noise-containing pattern image including one or more non-reference lines having a color pD (here, luminance) different from the reference line color nD (2) One arranged at a position different from the reference position in the horizontal direction Dx Noise-containing pattern image including the above non-reference lines (3) Noise-containing pattern image including one or more non-reference lines having a width pW different from the reference line width nW Therefore, the luminance and the position and width in the horizontal direction Dx Learning image data for training can be generated appropriately if each is variable. For example, the noise-containing non-reference pattern image 850b in FIG. 8B has one or more non-reference lines having a color pD different from the reference line color nD and is arranged at a position different from the reference position in the horizontal direction Dx. and one or more non-reference lines. Then, the noise-containing non-reference pattern image 850c of FIG. 8C can include one or more non-reference lines having widths pW different from the reference line width nW. Thus, a single noisy pattern image may contain multiple types of non-reference lines.

In addition, in the processing of FIG. 6A, non-reference lines are randomly selected for each noise-containing pattern image. Therefore, the total number of non-reference lines can be different between the multiple noise-containing non-reference pattern images. Therefore, learning image data for training can be appropriately generated when the total number of non-reference lines included in the nozzle pattern NP may change. For example, the total number of non-reference lines may differ between the noise-containing non-reference pattern image 850b of FIG. 8B and the noise-containing non-reference pattern image 850c of FIG. 8C.

A3. Training process:
10A to 10C are explanatory diagrams of the training process of the learning model 235. FIG. FIG. 10A is a flowchart showing an example of training processing for the learning model 235. FIG. In this embodiment, the learning model 235 is trained such that when image data of a nozzle pattern NP is input, image data of the same nozzle pattern NP with reduced noise is generated. The training adjusts the “Fx×Fy” weights of the filter FL (FIG. 3). In the following description, it is assumed that the MFP 200 (FIG. 1) executes the training. Processor 210 performs training according to second program 232 . It should be noted that the training may be performed by other data processing devices with high computing power.

In S310, the processor 210 initializes a plurality of calculation parameters of the learning model 235 (here, “Fx×Fy” weights of the filter FL). For example, each calculation parameter is set to a random value.

In S315, the processor 210 selects V (V is an integer equal to or greater than 1) noise-containing pattern image data from a plurality of noise-containing pattern image data, and inputs the V noise-containing pattern image data to the learning model 235. , V output image data. As the V pieces of noise-containing pattern image data, unused noise-containing pattern image data from among the plurality of noise-containing pattern image data may be selected. Alternatively, V pieces of noise-containing pattern image data may be randomly selected from a plurality of noise-containing pattern image data.

In S320, the processor 210 calculates an error value, which is an evaluation value of the difference between the adjusted noise-reduced pattern image data and the output image data, for each of the V noise-containing pattern image data input to the learning model 235. calculate. An error value is calculated based on a predetermined loss function. FIG. 10B is an explanatory diagram showing an example of a loss function. In this embodiment, the loss function LF is the sum of the first term Limg and the second term Lfilter.

The first term Limg indicates the difference between the adjusted noise reduction pattern image and the output image of the output image data. In this example, the first term Limg is the mean square root of the squared error normalized by 255. The number of pixels IW of the first term Limg in the figure is the number of pixels in the horizontal direction Dx of each image, and the number of pixels IH is the number of pixels in the vertical direction Dy of each image (see FIG. 3). The pixel value Po(x, y) is the pixel value of the pixel (x, y) of the output image. The value x is the pixel position in the horizontal direction Dx in the image, and the value y is the pixel position in the vertical direction Dy in the image. The pixel value Pcc(x, y) is the pixel value of the pixel (x, y) of the adjusted noise reduction pattern image (see FIGS. 9A-9C). The first term Limg becomes smaller as the difference between the adjusted noise reduction pattern image and the output image becomes smaller. Note that the first term Limg may be represented by various functions that calculate a smaller value as the difference between the two images is smaller.

The second term Lfilter is an evaluation value indicating the degree of symmetry of the weight W of the filter FL. FIG. 10C is an explanatory diagram of the second term Lfilter. The figure shows a filter FL and a filter axis of symmetry AxF. The filter symmetry axis AxF is parallel to the symmetry axis (for example, symmetry axis AxP, Ax0-Axk) of each line PL of the nozzle pattern NP (FIGS. 2A, 5B, etc.). In this embodiment, the filter axis of symmetry AxF is parallel to the first direction D1 (that is, the longitudinal direction Dy). Also, the filter symmetry axis AxF passes through the center of the filter FL. In this embodiment, since the horizontal size Fx of the filter FL is 5, the filter axis of symmetry AxF overlaps the weights W(3,1)-W(3,5) in the third column.

The second term Lfilter is the average value of the absolute values of the differences between two weights W arranged at symmetrical positions with respect to the filter symmetry axis AxF. A symmetrical pair PW1 and PW2 in FIG. 10(C) respectively shows an example of two weights W arranged at line-symmetrical positions. The first pair PW1 is composed of weight W(1,2) and weight W(5,2). The second pair PW2 consists of weights W(2,4) and weights W(4,5).

In the formula for calculating the second term Lfilter in FIG. 10B, the lower and upper limits of indices i and j of the summation symbol are set so as to select all of the plurality of weights on the left side of the filter symmetry axis AxF. It is In this example, ten weights W(1,1)-W(1,5) and W(2,1)-W(2,5) are selected. The variable added by the summation symbol is the absolute value of the difference of the symmetrical weight pair (W(i,j) and W(Fx+1-i,j)) identified by the indices i,j.

The second term Lfilter becomes smaller as the symmetry of the plurality of weights W with respect to the filter symmetry axis AxF is higher. The reason why the loss function LF includes such a second term Lfilter is to suppress deformation (for example, displacement) of the line PL due to image processing using the filter FL.

11(A) to 11(D) are explanatory diagrams of the position of the line PL. FIG. 11A is a graph showing luminance values on the image of the nozzle pattern NP. The horizontal axis indicates the position Px in the horizontal direction Dx, and the vertical axis indicates the luminance value V. FIG. This graph shows the distribution of luminance values V on one pixel line extending in the horizontal direction Dx. The direction of the vertical axis is downward. Therefore, the luminance value V is small in the upper part of the graph, and the luminance value V is large in the lower part of the graph. A peak in the graph indicates a line PL, and a valley in the graph indicates the background portion BG. Variables on the vertical axis and variables on the horizontal axis are common to each graph of FIGS. 11(A) to 11(D).

FIG. 11B is a graph of luminance values showing one line PL. A dotted line graph indicates the brightness value Po of the output image generated by the learning model 235, and a solid line graph indicates the brightness value Pc of the noise reduction pattern image. FIG. 11B shows the brightness values Po output by the learning model 235 before training is completed. The luminance value Po indicates the luminance value at each pixel position on the output image, like the pixel value Po(x, y) of the output image described in FIG. 10B. The luminance value Pc indicates the luminance value at each pixel position on the noise reduction pattern image, like the luminance value Pc(x, y) described with reference to FIG. 9A and the like. As shown, there is a difference between these luminance values Po and Pc. In the training of the embodiment, the difference between the luminance values Po and Pcc is smaller than the difference between the luminance values Po and Pc (the luminance value Pcc is the pixel value Pcc (x, y) described in FIG. 10B and the like). Similarly, the luminance value at each pixel location on the adjusted noise reduction pattern image is shown). However, here, in order to simplify the explanation, it is assumed that training reduces the difference between the luminance values Po and Pc.

FIG. 11C is an explanatory diagram of a reference example. This graph shows luminance values Po and Pc in a hypothetical case where the loss function LF is composed only of the first term Limg. The brightness value Po is the brightness value output by the trained learning model 235 . The first peak position PxC in the drawing is the peak position of the line PL indicated by the luminance value Pc of the noise reduction pattern image. This peak position PxC is the same as the position of the axis of symmetry of the line PL, and is the same as the peak position of the line PL on the noise-containing pattern image input to the learning model 235 . Hereinafter, the first peak position PxC is also called the axis of symmetry AxC.

A second peak position PxO in the drawing is the position of the peak of the line PL indicated by the luminance value Po of the output image. The second peak position PxO is shifted from the first peak position PxC. The reason for this is as follows. In training using the first term Limg, the filter FL is adjusted so that the error value calculated according to the first term Limg is small without distinguishing between the line PL and the background portion BG. As a result, in order to reduce the large difference in the background portion GB (FIG. 11B), the adjusted filter FL reduces the difference between the luminance values Po and Pc of the portion indicating the line PL (FIG. 11C). , can be adjusted asymmetrically with respect to the axis of symmetry AxC of the line PL. As a result, the line PL on the output image may be deformed. For example, the second peak position PxO of the line PL on the output image may deviate from the original peak position PxC. When such an output image is used, the specific error of the position of the line PL on the output image becomes large.

FIG. 11D is an explanatory diagram of this embodiment. The brightness value Po is the brightness value output by the trained learning model 235 . As illustrated, the peak position PxO of the line PL indicated by the luminance value Po of the output image is approximately the same as the original peak position PxC. The reason for this is as follows. The loss function LF (FIG. 10B) includes a second term Lfilter in addition to the first term Limg. The value of the second term Lfilter increases as the symmetry of the weight W of the filter FL decreases. In the operation by the convolutional layer 310 (FIG. 3), the filter FL is applied to a plurality of evenly spaced positions on the input image (in this example, all pixel positions of the input image). That is, the filter FL is applied to a plurality of positions overlapping one line PL and symmetrical with respect to the symmetry axis AxC of the line PL. Therefore, if the filter FL adjusts the difference between the luminance values Po and Pc asymmetrically with respect to the axis of symmetry AxC of the line PL as shown in FIG. The symmetry of the weight W for (C)) is low. As a result, although the value of the first term Limg can be reduced, the value of the second term Lfilter increases, so the error value based on the loss function LF increases.

When the weight W of the filter FL is adjusted so that the error value based on the loss function LF (FIG. 10B) is reduced, the symmetry of the weight W with respect to the filter symmetry axis AxF (FIG. 10C) increases. . Accordingly, the adjusted filter FL adjusts the difference between the luminance values Po, Pc symmetrically with respect to the axis of symmetry AxC of the line PL. As a result, the deviation between the second peak position PxO of the line PL on the output image and the original peak position PxC is suppressed.

In S325 (FIG. 10A), the processor 210 uses V error values based on such a loss function LF to calculate a plurality of calculation parameters of the learning model 235 (in this embodiment, the weight W of the filter FL ). Specifically, processor 210 adjusts a plurality of calculation parameters according to a predetermined algorithm so that the error value becomes small. As an algorithm, for example, an algorithm using backpropagation and gradient descent is used.

At S330, processor 210 determines whether the training is complete. The training completion condition may be, for example, that all error values calculated in S320 are smaller than a predetermined error threshold. Alternatively, the training completion condition may be that the number of times the processes of S315-S325 have been executed is greater than or equal to a predetermined number threshold.

If it is determined that training has not been completed (S330: No), the processor 210 moves to S315. If it is determined that the training has been completed (S330: Yes), in S335 the processor 210 extracts the data of the filter FL (that is, the data of Fx×Fy weights W) from the learning model 235. FIG. In S340, the processor 210 stores the filter data FLD, which is the data of the filter FL, in the storage device 215 (nonvolatile storage device 230 in this embodiment). Then, the processing of FIG. 10A ends.

A4. Image processing:
FIG. 12 is a flow chart showing an example of image processing using the filter data FLD (FIG. 1). In this image processing, a filtering process using a filter FL is performed on the read data of the sheet SH2 on which the nozzle pattern NP is printed. This image processing is performed to check for defects in the nozzles Nz of the print head 110 . For example, the process of FIG. 12 is performed when the printing apparatus 100 is manufactured. In this embodiment, the processor 210 of the MFP 200 executes image processing according to the third program 233. FIG.

In S410, the printing apparatus 100 prints the nozzle pattern NP on the sheet SH2. For example, an operator causes the printer 100 to print the nozzle pattern NP by operating the printer 100 . Alternatively, the printing device 100 may be connected to the MFP 200 and the processor 210 may supply the printing instructions to the printing device 100 .

In S420, the sheet SH2 on which the nozzle pattern NP is printed is read by the reading unit 300 to generate read data. In S430, the processor 210 acquires the read data from the reading unit 300, and acquires the target data by executing crop processing for cutting out a portion indicating the nozzle pattern NP from the read data. A portion to be cut out from the read image is determined in advance. Note that the cropping process may be omitted and the read data may be used as it is as the target data.

At S440, the processor 210 generates processed image data by performing filtering using the filter FL on the target data. The filtering process is the same as the convolution process by convolution layer 310 (FIG. 3). The processed image of the processed image data shows the same non-deformed nozzle pattern NP as the nozzle pattern NP of the target data. The processed image then has reduced noise and enhanced contrast.

At S450, processor 210 stores the processed image data in storage device 215 (non-volatile storage device 230 in this example). In S460, the processor 210 determines whether or not the nozzle pattern NP of the processed image has defects by analyzing the processed image data. For example, processor 210 detects multiple lines PL in the processed image according to known line detection methods (eg, edge extraction and Hough transform). Processor 210 then identifies the position, width, and brightness value of each line PL in the horizontal direction Dx by analyzing the plurality of detected lines PL. If at least one parameter value of the position, width, and luminance value of a line of interest is outside the allowable range corresponding to the parameter value, the processor 210 determines that the line of interest has a defect. do. If the position, width, and luminance values of the line of interest are all within the corresponding tolerances, processor 210 determines that the line of interest is flawless. In S470, the processor 210 stores result data indicating the determination result in the storage device 215 (eg, the non-volatile storage device 230). Then, the process of FIG. 12 ends.

As described above, in the process of FIG. 10A, the convolution filter FL for image processing is generated. At S<b>315 , the processor 210 generates output image data by inputting the noisy pattern image data to the learning model 235 . As described with reference to FIG. 3, the learning model 235 is a neural network model composed of one convolutional layer 310 . The convolution filter FL indicates the weight W of each of a plurality of pixels within the filter area, which is the area corresponding to the convolution filter FL on the input image. A convolution layer 310 performs a convolution operation using a convolution filter FL. In S320 (FIG. 10A), the processor 210 uses the output image data and the noise reduction pattern image data to calculate an error value based on the loss function LF. At S325, the processor 210 adjusts the multiple weights W of the convolution filter FL so that the error value is small. In S335, the processor 210 extracts the filter data FLD of the convolution filter FL indicating the adjusted weights W as the data of the image processing filter. Here, as shown in FIGS. 8(A) to 8(C), the noise-containing pattern images 850a to 850c are images of the nozzle pattern NP including the line PL and containing noise. As described with reference to FIG. 5B, the lines PL0-PLk have shapes that are symmetrical with respect to the respective axes of symmetry Ax0-Axk. As described in FIGS. 9A-9C, the noise-reduced pattern images 860a-860c are images of the nozzle pattern NP with reduced noise compared to the noise-containing pattern images 850a-850c. It is an image. Also, the noise reduction pattern images 860a-860c respectively show the same nozzle patterns NP as the nozzle patterns NP of the noise-containing pattern images 850a-850c. Then, as shown in FIG. 10B, the loss function LF includes a first term Limg and a second term Lfilter. The first term Limg indicates the magnitude of the difference between the output image data and the noise reduction pattern image data. The second term Lfilter decreases as the symmetry of the plurality of weights W with respect to the filter symmetry axis AxF in the convolution filter FL (FIG. 10(C)) increases. The filter axis of symmetry AxF is parallel to the axis of symmetry Ax0-Axk of the line PL0-PLk. Therefore, when image processing using the filter FL is performed on the image of the nozzle pattern NP, noise can be reduced, and deformation of the nozzle pattern NP due to image processing can be suppressed. Specifically, as described with reference to FIGS. 11A to 11D, changes in the position of the line PL are suppressed.

Both the image data of the reference nozzle pattern NP and the image data of the non-reference nozzle pattern NP are used for training. Therefore, the learning model 235 (and thus the filter FL) can generate an output image of the same undeformed nozzle pattern NP regardless of whether the nozzle pattern NP of the read image includes non-reference lines.

Further, as described in S180 and S185 of FIG. 4 and FIGS. 9A to 9C, the noise reduction pattern images 860a to 860c used for error value calculation are the noise containing pattern images 850a to 850c ( 8(A) to 8(C)) is an image in which the contrast is enhanced. Therefore, contrast can be enhanced by image processing using the filter FL.

Further, as described with reference to FIGS. 10B and 10C, the second term Lfilter is defined between two weights arranged in line-symmetrical positions with respect to the filter symmetry axis AxF in the filter FL. The smaller the difference between , the smaller. Therefore, it is possible to appropriately suppress deformation (for example, displacement) of the line PL due to image processing using the filter FL.

Further, in S410 of FIG. 12, the printing apparatus 100 (FIG. 1) prints the nozzle pattern NP on the sheet SH2. The printing device 100 has a print head 110 . The print head 110 (FIGS. 2A and 2B) has a plurality of nozzles Nz configured to eject ink. The nozzle pattern NP includes a plurality of parallel lines PL. Each of the plurality of lines PL has a line-symmetrical shape with respect to the central axis AxP. One line PL is printed by one nozzle Nz. In S420 (FIG. 12), the reading unit 300 optically reads the nozzle pattern NP to generate read data of the image of the nozzle pattern NP. In S430, processor 210 acquires target data, which is image data of nozzle pattern NP, using the read data. In S440, processor 210 generates processed image data by performing image processing using filter FL on the target data. At S<b>450 , processor 210 stores the processed image data in storage device 215 . Noise is reduced on the image of the processed image data, and deformation of the lines PL of the nozzle pattern NP is suppressed. Therefore, by using the processed image data, the states of the multiple nozzles Nz can be appropriately identified. Thus, the image processing using the filter FL can improve the image quality of the read data generated by the reading unit 300 , particularly the image quality of the read image of the nozzle pattern NP printed by the printing apparatus 100 . As described with reference to FIGS. 2A and 2B, the nozzle pattern NP can be used to confirm the quality of the plurality of nozzles Nz (FIG. 2A). For example, when the printing apparatus 100 is manufactured, printing of the nozzle pattern NP and visual confirmation may be performed. Here, as a method for reducing the manufacturing cost, the nozzle pattern NP is printed using low-density and inexpensive ink, the printed nozzle pattern NP is read by the reading unit 300, and the nozzle pattern NP is obtained by image processing of the read image. A method of determining quality (that is, quality of nozzle Nz) can be employed. However, when the nozzle pattern NP is printed with low-density ink, the color of the nozzle pattern NP is light, so it is not easy to distinguish between the nozzle pattern NP and the background on the read image. By correcting the brightness of the read image, the nozzle pattern NP can be emphasized. However, sheet patterns (for example, paper fiber patterns) and digital noise are also emphasized, making it difficult to determine the quality of the nozzles Nz by image processing. Since the filter FL of the present embodiment can reduce sheet patterns (for example, paper fiber patterns) and digital noise, it is possible to improve the accuracy of determining whether the nozzles Nz are good or bad by image processing. As described above, low-concentration, low-cost ink can be used, so that the manufacturing cost can be greatly reduced. Note that S460 and S470 in FIG. 12 may be omitted. The processed image data stored in the storage device in S450 can be used in various other processes instead of the processes in S460 and S470. For example, processor 210 may display the processed image of the processed image data on display 240 . By observing the processed image, the operator can easily identify defects in the lines PL (and thus defects in the nozzles Nz).

Also, in this embodiment, image processing is performed using a filter FL instead of the learning model 235 . Therefore, the computational complexity of image processing can be greatly reduced.

B. Second embodiment:
FIG. 13 is an explanatory diagram of another example of the learning model. The learning model 235d of this embodiment is a model of an artificial neural network, also called an autoencoder. In this embodiment, the learning model 235d uses the input image data to generate noise-reduced image data. Learning model 235d can be used in place of learning model 235 above.

The learning model 235d includes a first encoder 411, a second encoder 412, a third encoder 413, a third decoder 415, a second decoder 416, and a first decoder 417, for example. These processing units 411-413 and 415-417 are connected in this order. Encoders 411-413 extract image features to generate latent variables 414 indicative of the extracted features. Each encoder 411-413 is composed, for example, of a convolutional layer and a pooling layer that processes data from the convolutional layer. Decoders 415-417 recover the features indicated by latent variables 414 to generate image data. Each decoder 415-417 is composed, for example, of a convolutional layer and an upsampling layer that processes data from the convolutional layer.

The noise-containing and noise-reduced pattern images generated in the example of FIG. 4 are used to train learning model 235d. Training is performed similarly to the example of FIG. 10(A). In the following description, it is assumed that the MFP 200 (FIG. 1) executes the training. It should be noted that the training may be performed by other data processing devices with high computing power.

At S352 (FIG. 13), the processor 210 inputs the noisy pattern image data 850d to the learning model 235d. In S354, processor 210 generates output image data 870d by performing calculations on learning model 235d. In S356, processor 210 calculates error value 890d from noise reduction pattern image data 860d and output image data 870d according to loss function LF (FIG. 10B). The noise reduction pattern image data 860d is data associated with the noise containing pattern image data 850d. In S358, processor 210 adjusts the calculation parameters of learning model 235d according to a predetermined algorithm so that error value 890d is reduced. The operational parameters that are adjusted include the weights and biases of the convolution filters of processing units 411-413, 415-417. As an algorithm, for example, an algorithm using backpropagation and gradient descent is used. The processor 210 repeats the above S352-S358. Then, in response to the training completion condition being satisfied, the processor 210 stores the learned learning model 235d in the non-volatile storage device 230 in S360.

The learned learning model 235d is used for noise reduction processing of image data. The processor 210 inputs the read data 750d (FIG. 13) of the nozzle pattern NP to the learning model 235d. Processor 210 can then generate noise-reduced image output data 770d by performing operations on learning model 235d.

C. Third embodiment:
FIG. 14 is a flow chart showing another embodiment of the learning data generation process. The difference from the embodiment of FIG. 4 is that an image of characters is generated instead of an image of the nozzle pattern NP. S105 is the same as S105 in FIG. In S110a, processor 210 selects a target character, which is a character to be processed. In this embodiment, the target character is identified by a combination of character and typeface. Characters are selected from a predetermined character set (eg, a set of alphabets and numbers). The typeface is selected from a plurality of predetermined typefaces (eg, normal, bold, italic). In S135a, processor 210 generates character image data of a character image that is an image of the target character. The luminance value of pixels representing text is set to zero, and the luminance value of pixels representing background is set to 255. In S150a, the processor 210 executes the smoothing process of the character image data, similarly to S150 (FIG. 4). 15A to 15D are explanatory diagrams of learning data. Character images 810e and 810f are shown in FIGS. 15A and 15B. In this embodiment, the character images 810e and 810f are images of one character. The sizes of the character images 810e and 810f are predetermined.

At S155a (FIG. 14), the processor 210 generates the first type background image data as at S155 (FIG. 4). The size of the first type background image is the same as the size of the character image. 15(A) and 15(B) show type 1 background images 830e and 830f. In S160a, processor 210 generates noise-containing character image data, as in S160 (FIG. 4). Noise-containing character images 850e and 850f are shown in FIGS. 15A and 15B. The luminance values Pn(x, y) of the noise-containing character images 850e, 850f are the luminance values Pt(x, y) of the character images 810e, 810f and the luminance values Pu(x, y) of the first type background images 830e, 830f. is calculated according to the relational expression of FIG. 8(D). In S165a, the processor 210 stores the generated noise-containing character image data in the storage device 215 (nonvolatile storage device 230 in this embodiment).

At S170a, the processor 210 identifies representative color values of the first type background image, as at S170 (FIG. 4). In S175a, processor 210 generates noise-reduced character image data, as in S175 (FIG. 4). 15C and 15D show character images 810e and 810f, second-type background images 830em and 830fm represented by representative color values, and noise-reduced character images 851e and 851f. ing. The luminance values Pc(x, y) of the noise-reduced character images 851e, 851f are the luminance values Pt(x, y) of the character images 810e, 810f and the luminance values Pum(x, y) of the second type background images 830em, 830fm. is calculated according to the relational expression of FIG. 9(D).

In S180a, processor 210 executes contrast enhancement processing of the noise-reduced character image data, as in S180 (FIG. 4). FIGS. 15(C) and 15(D) show noise-reduced character images 860e-860f with enhanced contrast. In S185a, the processor 210 stores the adjusted noise-reduced character image data whose contrast has been adjusted in the storage device 215 (non-volatile storage device 230 in this embodiment) in association with the noise-containing character image data.

At S190a, processor 210 determines whether or not generation of image data is completed. In this embodiment, the completion condition is that image data of all combinations of a plurality of characters and a plurality of typefaces have been generated. At S110a, an unprocessed combination is selected. Note that the completion condition may be other various conditions.

The noise-containing text image data and the adjusted noise-reduced text image data generated in this embodiment can be used to train various predictive models, such as learning models 235, 235d (FIGS. 3, 13). Here, the character image data may represent characters that are not line-symmetrical (eg, the letter "F" is not line-symmetrical). Therefore, the second term Lfilter is omitted from the loss function LF (FIG. 10(B)).

D. Fourth embodiment:
FIG. 16 is an explanatory diagram of another example of the learning model. The learning model 235g of this embodiment is a model of an artificial neural network, also called a convolutional neural network. In this example, learning model 235g classifies images into multiple categories. Specifically, the learning model 235g classifies the image of the nozzle pattern NP into a reference nozzle pattern NP and a non-reference nozzle pattern NP.

The learning model 235g includes, for example, a first convolutional layer 421, a first pooling layer 422, a second convolutional layer 423, a second pooling layer 424, a first fully connected layer 425, and a second fully connected layer. 426 and a third fully connected layer 427 . These processing units 421-427 are connected in this order. This learning model 235g is a model of a typical convolutional neural network. The learning model 235g uses the input image data to generate output data OD indicating the probability PA of the reference nozzle pattern NP and the probability PB of the non-reference nozzle pattern NP.

The noisy pattern images generated in the example of FIG. 4 are used to train the learning model 235g. As described above, the multiple noise-containing pattern images include an image representing the reference nozzle pattern NP and an image representing the non-reference nozzle pattern NP. Label data indicating "reference" or "non-reference" is associated with each noise-containing pattern image. This label data is used as teacher data, which is the target value of the output data. In this embodiment, adjusted noise reduction pattern image data is not utilized. Therefore, the generation of adjusted noise reduction pattern image data may be omitted. In the following description, it is assumed that the MFP 200 (FIG. 1) executes the training. It should be noted that the training may be performed by other data processing devices with high computing power.

At S372 (FIG. 16), the processor 210 inputs the noisy pattern image data 850g to the learning model 235g. In S374, the processor 210 generates the output data OD by operating the learning model 235g. In S376, processor 210 calculates error value 890g according to a predetermined loss function using output data OD and teacher data 880g. The loss function can be various functions suitable for classification, for example the cross-entropy error. The teacher data 880g is label data associated with the noise-containing pattern image data 850g. In S378, processor 210 adjusts the calculation parameters of learning model 235g according to a predetermined algorithm so that error value 890g is reduced. The operational parameters that are adjusted include the weights and biases of the convolutional filters of the convolutional layers 421 and 423 and the weights and biases of the fully connected layers 425 , 426 and 427 . As an algorithm, for example, an algorithm using backpropagation and gradient descent is used. Processor 210 repeats the above S372-S378. Then, in response to the training completion condition being satisfied, the processor 210 stores the learned learning model 235g in the non-volatile storage device 230 in S380.

The learned learning model 235g is used for classification of the nozzle pattern NP. The processor 210 inputs the read data 750g (FIG. 16) of the nozzle pattern NP to the learning model 235g. The processor 210 then generates the output data OD by performing calculations on the learning model 235g. The processor 210 refers to the probabilities PA, PB of the output data OD and classifies the nozzle pattern NP of the input image into the highest probability category.

E. Variant:
(1) The process of generating learning image data may be various other processes instead of the above process. For example, in S105 of FIGS. 4 and 14, one read data of one sheet may be obtained. Then, in S155 and S155a, the first type background image data may be generated from one piece of read data. Also, one piece of type 1 background image data may be used in common for a plurality of pieces of learning image data. Also, the weight k1 in FIG. 9(D) may be different from the weight k1 in FIG. 8(D). Also, the weight k2 in FIG. 9(D) may be different from the weight k2 in FIG. 8(D). In any case, the weights k1 and k2 in FIG. 9(D) are each preferably greater than zero. Also, in the processing of FIG. 6A, among the three parameters of width, luminance value, and position, only one or two parameters may be changed from the reference values. That is, one or two of the three parameters may be fixed at the reference value.

Further, in the processing of FIGS. 4 and 14, S150 and S150a for smoothing may be omitted. Moreover, S180 and S180a for adjusting the contrast may be omitted.

(2) The second term Lfilter of the loss function LF (FIG. 10B) may be represented by various functions that calculate a larger value as the difference between two weights at line-symmetrical positions increases. For example, the second term Lfilter may be the average squared difference of the two weights.

(3) The size of the filter FL (FIG. 10(C)) may be of various sizes. For example, Fx=7 and Fy=7. Moreover, the horizontal size Fx may be different from the vertical size Fy. Also, in order to improve the symmetry of the weight W with respect to the filter axis of symmetry AxF, the size in the direction perpendicular to the filter axis of symmetry AxF (here, horizontal size Fx) is preferably an odd number.

(4) The test pattern printed by the printing apparatus 100 may be various other patterns instead of the nozzle pattern NP including a plurality of lines PL (FIG. 2A). For example, the test pattern may be a grid pattern including a plurality of vertical lines and a plurality of horizontal lines. The test patterns may include patterns of various shapes such as circles, ellipses, triangles, rectangles, and the like. A test pattern may include specific portions having shapes that are symmetrical about an axis of symmetry, such as lines, circles, ellipses, triangles, and rectangles. When the loss function LF includes a second term that improves the symmetry of the weight W of the filter FL, it is possible to suppress deformation (for example, displacement) of a specific portion due to image processing using the filter FL. In addition, the size of the filter FL in the direction perpendicular to the filter axis of symmetry AxF (the horizontal size Fx in the example of FIG. 10C) is the size of the specific portion in the direction perpendicular to the axis of symmetry (for example, the width of the line PL ) or smaller. The test pattern may include both the specific portion (for example, the nozzle pattern NP) and characters. The test pattern may contain both graphics and characters. Note that the above specific portion may be omitted from the test pattern. In this case, the second term may be omitted from the loss function LF.

(5) Image data may be represented by multiple color components (eg, RGB, CMYK, YCbCr, etc.). The read data, input data, output data, learning image data (noise-containing pattern image data, noise reduction pattern image data, etc.), and target data may each be RGB bitmap data. When the color value of each pixel of the image data is represented by u gradation values of u (u is an integer equal to or greater than 1) color components, the convolution filter FL in FIGS. It contains Fx×Fy×u weights. This single convolution filter FL can be decomposed into u sub-filters corresponding to u color components. One sub-filter is composed of Fx×Fy weights and is applied to the gradation value of the color component associated with that sub-filter. Here, the second term Lfilter of the loss function LF (FIG. 10(B)) is preferably a comprehensive evaluation value indicating the degree of symmetry of each of the u sub-filters. For example, the formula for calculating the second term Lfilter shown in FIG. 10B can be used as the formula for calculating the evaluation value of one sub-filter. As a comprehensive evaluation value of u sub-filters, various values specified using u evaluation values of u sub-filters (for example, representative values such as average value, median value, total value ) may be used.

(6) The configuration of the prediction model may be various other configurations instead of the configurations of the learning models 235, 235d and 235g of FIGS. 3, 13 and 16. FIG. The prediction model is not limited to a model that performs noise reduction of image data or a model that classifies image data, and may be a model that performs other various processes. Predictive models may include, for example, one or more models arbitrarily selected from the group of artificial neural networks, hidden Markov models, and inference engines.

(7) The device used for optical reading may be various reading devices such as a digital camera instead of the reading unit 300 .

(8) The data processing of generation of learning image data, training of prediction models, and image processing using filters may be performed by different data processing devices. The data processing device may be a device of a different type from the MFP 200 (eg, personal computer, digital camera, scanner, smart phone). In addition, a plurality of devices (for example, computers) that can communicate with each other via a network may share the function of data processing by the data processing device, and provide the function of data processing as a whole ( A system comprising these devices corresponds to a data processing device).

In each of the above embodiments, part of the configuration implemented by hardware may be replaced with software, or conversely, part or all of the configuration implemented by software may be replaced with hardware. good too. For example, the learning models 235, 235d, and 235g (FIGS. 3, 13, and 16) may be realized by hardware circuits such as ASICs (Application Specific Integrated Circuits) instead of program modules.

In addition, when part or all of the functions of the present invention are realized by a computer program, the program is provided in a form stored in a computer-readable recording medium (for example, a non-temporary recording medium). be able to. The program can be used while being stored in the same or different recording medium (computer-readable recording medium) as when it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but also internal storage devices such as various ROMs in computers, and hard disk drives that are connected to computers. An external storage device may also be included.

Although the present invention has been described above based on examples and modifications, the above-described embodiments of the present invention are intended to facilitate understanding of the present invention, and are not intended to limit the present invention. The present invention may be modified and improved without departing from its spirit, and the present invention includes equivalents thereof.

DESCRIPTION OF SYMBOLS 100...Printing apparatus 110...Print head 200...MFP 205...Control part 210...Processor 215...Storage device 220...Volatile storage device 230...Nonvolatile storage device 231...First program 232 2nd program 233 3rd program 235, 235d, 235g learning model 240 display unit 250 operation unit 260 printing unit 270 communication interface 300 reading unit 310 convolution layer , 411... first encoder, 412... second encoder, 413... third encoder, 415... third decoder, 416... second decoder, 417... first decoder, 421... first convolution layer, 422... first pooling Layers 423... Second convolution layer 424... Second pooling layer 425... First fully connected layer 426... Second fully connected layer 427... Third fully connected layer D1... First direction D2... Third Two directions, FL... Convolution filter, Dx... Horizontal direction, Dy... Vertical direction, Nz... Nozzle

Claims (9)

A method of generating training image data for training a predictive model, comprising:
Using a first type background image that is at least part of a sheet image obtained by optically reading a printing sheet and an object image that is an image of an object, the object is displayed on the first type background image. a first generation step of generating type 1 learning image data of a type 1 learning image that is an image in which is arranged;
a first storing step of storing the generated type 1 learning image data in a storage device;
with
The first generation step includes
determining a color value of a first object pixel representing the object on the first type learning image, wherein the color value of the first object pixel is mapped to the position of the first object pixel on the object image; A value obtained by multiplying the color value of the associated pixel by a first-type object coefficient greater than zero, and the color value of the pixel associated with the position of the first object pixel on the first-type background image. a value obtained by multiplying by a type 1 background factor greater than zero; and
A step of determining a color value of a first background pixel indicating a portion of the first type learning image in which the object is not arranged, wherein the color value of the first background pixel is set to the determining using a value obtained by multiplying the color value of the pixel associated with the position of the first background pixel by a coefficient larger than the first type background coefficient;
generation method. The method of claim 1, comprising:
a representative color value specifying step of specifying a representative color value representing a plurality of color values associated with a plurality of positions on the first type background image;
A second type of a type 2 learning image, which is an image in which the object is arranged on the type 2 background image, using the type 2 background image represented by the representative color value and the object image. a second generation step of generating learning image data;
a second storing step of storing the generated type 2 learning image data in the storage device;
with
The second generation step includes
determining a color value of a second object pixel representing the object on the second type learning image, wherein the color value of the second object pixel is assigned to the position of the second object pixel on the object image; A value obtained by multiplying the color value of the associated pixel by a second type object coefficient greater than zero, and the color value of the pixel associated with the position of the second object pixel on the second type background image. a value obtained by multiplying by a type 2 background factor greater than zero; and
A step of determining a color value of a second background pixel indicating a portion of the second type learning image where the object is not arranged, wherein the color value of the second background pixel is set to the color value of the second background pixel on the second type background image. determining using a value obtained by multiplying the color value of the pixel associated with the position of the second background pixel by a coefficient larger than the second type background coefficient;
generation method. The production method of claim 2, wherein
The representative color value is an average value of the plurality of color values associated with the plurality of positions on the first type background image,
generation method. A production method according to any one of claims 1 to 3,
The object includes M lines (where M is an integer equal to or greater than 2) parallel to a first direction and M lines aligned in a second direction perpendicular to the first direction,
generation method. 5. The method of claim 4, comprising:
A step of generating Q pieces of type 1 learning image data of Q pieces of type 1 learning images by performing the first generation step Q times (Q is an integer of 2 or more),
Each of the Q type 1 learning images has a brightness different from the reference brightness associated with the type 1 learning image, and an order of the type 1 learning image and the line in the second direction. Of the three parameter values of the position in the second direction different from the associated reference position in the second direction and the width different from the reference width associated with the type 1 learning image including one or more non-reference lines that are lines with one or more parameter values;
generation method. 6. The production method of claim 5, comprising:
The Q type 1 learning images are
one or more type 1 learning images including one or more non-reference lines having the luminance different from the reference luminance;
one or more type 1 learning images including one or more non-reference lines arranged at the position in the second direction different from the reference position in the second direction;
one or more type 1 learning images including one or more non-reference lines having the width different from the reference width;
generation method, including A production method according to claim 5 or 6,
The Q type 1 learning images include a plurality of type 1 learning images with different total numbers of the non-reference lines included in the type 1 learning images,
generation method. A production method according to any one of claims 1 to 7,
the object comprises a character,
generation method. In response to input of image data of a print sheet image, which is an image obtained by optically reading a sheet on which an object is printed, related information that is information related to the object printed on the sheet is generated. A predictive model for operating a computer, comprising:
Image data generated using a first type background image that is at least a part of a sheet image obtained by optically reading a printing sheet and an object image that is an image of an object, trained using the first type learning image data of the first type learning image, which is an image in which the object is arranged on the first type background image,
The first type learning image data is
determining a color value of a first object pixel representing the object on the first type learning image, wherein the color value of the first object pixel is mapped to the position of the first object pixel on the object image; A value obtained by multiplying the color value of the associated pixel by a first-type object coefficient greater than zero, and the color value of the pixel associated with the position of the first object pixel on the first-type background image. a value obtained by multiplying by a type 1 background factor greater than zero; and
A step of determining a color value of a first background pixel indicating a portion of the first type learning image in which the object is not arranged, wherein the color value of the first background pixel is set to the determining using a value obtained by multiplying the color value of the pixel associated with the position of the first background pixel by a coefficient larger than the first type background coefficient;
is generated by a generation method that includes
The first type learning image data includes data of a reference image, which is an image in which an object having no defect is arranged, and data of a non-reference image, which is an image in which an object having a defect is arranged,
The predictive model is
generating pertinent information related to the non-defective object in response to input of the reference image data;
generating relevant information related to the defective object in response to the input of the non-reference image data, wherein the relevant information is different from the relevant information generated in response to the input of the reference image data;
are trained to
The prediction model performs calculation of the prediction model on the input of the image data of the print sheet image,
if the object printed on the sheet has no defects, generating relevant information related to the non-defective object printed on the sheet;
if the object printed on the sheet has a defect, related information related to the object having the defect printed on the sheet, and if the object printed on the sheet does not have the defect generate related information that is different from the related information generated in
make your computer work like
predictive model.

Images