JP2022174516A - Image processing method, computer program, image processing apparatus, and training method - Google Patents

Abstract

To provide an image processing method, a computer program, an image processing apparatus, and a training method for appropriately forming a generation model for generating an image representing trouble in an object of interest.SOLUTION: An image processing method inputs input image data of interest representing an optically-read object of interest to a first generation model 510 to generate first generated image data of interest corresponding to the input image data of interest. The first generation model 510 is a generation model trained so that an image of image data generated by inputting image data representing an image of an object to the first generation model comes closer to a computer graphics image of the object. The first generated image data of interest is input to a second generation model 600 to generate second generated image data of interest representing an image of the object of interest with no trouble. Difference image data of a difference image is generated which indicates the difference between the image of the first generated image data of interest and the image of the second generated image data of interest.SELECTED DRAWING: Figure 1

Description

The present specification relates to image processing using generative models.

Traditionally, machine learning models, also called neural networks, are used for image processing. For example, Patent Literature 1 proposes a technique for classifying microscope images using a neural network.

Japanese Patent No. 6767966

A generative model that generates image data can be used as the machine learning model. The generative model, for example, uses input image data to generate image data corresponding to the input image data. There was room for improvement in constructing the generative model so as to generate appropriate image data.

This specification discloses techniques for appropriately configuring a generative model.

The technology disclosed in this specification can be implemented as the following application examples.

[Application Example 1] In an image processing method for processing image data, inputting attention input image data representing an optically read attention target object to a first generation model allows correspondence to the attention input image data. a first generating step of generating first generated image data of interest, wherein the first generative model inputs image data representing an optically read image of an object into the first generative model inputting the first generated image data of interest to a second generation model, wherein the image of the generated image data is a generative model trained so as to approximate the computer graphics image of the object; Thus, in the second generating step of generating the second generated image data of interest representing the image of the object of interest without defects, the second generation model generates the image data representing the image of the object in the second generation step. The second generating step, which is a generative model trained to generate image data representing an image of a non-defective object by inputting to the model; a third generating step of generating difference image data of a difference image indicating a difference between the generated image data of interest and the image of generated image data of interest, wherein the first generative model is a generative adversarial network having a generative network and an identification network; wherein the identification network is an image of image data generated by inputting input image data for training into the generation network, a first partial region in a first image a first partial image data representing a first partial image showing at least part of an image of and a second part showing at least part of an image of a second partial area in a second image which is a computer graphics image for training and second partial image data representing an image into said identification network.

According to this configuration, the variety of image data input to the identification network is increased, and the training speed of the identification network is suppressed, so that the training of the generation network and the training of the identification network can be balanced. . Therefore, the first generative model is designed so that the image of the image data generated by inputting the image data representing the optically read image of the object to the first generative model approaches the computer graphics image of the object. properly trained to Since the first generated image data of interest generated by the first generative model is input to the second generative model, the second generative model can generate the second generated image data of interest representing an image of the object of interest without defects. can be generated properly. Therefore, the difference image data of the difference image indicating the difference between the image of the first generated image data of interest and the image of the second generated image data of interest is the input image data of interest regardless of the conditions for reading the object of interest. , can be adequately represented.

[Application Example 2] A training method for an adversarial generative network having a generative network and a discriminating network, wherein input image data for training is input to the generative network to generate first image data of a first image. randomly determining a first partial area in said first image; and a first partial image representing at least a part of an image of said first partial area in said first image. acquiring data using said first image data and said first partial area; acquiring first identification data by inputting said first partial image data into an identification network; randomly determining a second partial region in a second image of second image data associated with input image data; obtaining second partial image data representing a two partial image using said second image data and said second partial area; and inputting said second partial image data into said identification network, thereby obtaining two identification data; and adjusting parameters of the identification network by using the first identification data and the second identification data.

According to this configuration, the variety of images input to the identification network for training of the adversarial generative network is improved, so the identification network, and thus the adversarial generative network having the identification network and the generative network, can be appropriately can train.

It should be noted that the technology disclosed in this specification can be implemented in various aspects, for example, image processing method and image processing device, adversarial generative network training method and training device, methods and devices thereof It can be realized in the form of a computer program for realizing functions, a recording medium (for example, a non-temporary recording medium) recording the computer program, or the like.

1 is an explanatory diagram showing an image processing apparatus as one embodiment; FIG. 5 is a block diagram showing an example of a first model 500; FIG. (A) is a block diagram showing an example of a second generative model 600; (B) is a block diagram showing an outline of training of the second generative model 600. FIG. 5 is a flow chart showing an example of a first training process for the first model 500; 5 is a flow chart showing an example of a first training process for the first model 500; 6 is a flowchart showing an example of a second training process for the second generative model 600; 6 is a flowchart showing an example of inspection processing; 1 is a schematic diagram of an inspection process; FIG. (A) and (B) are schematic diagrams showing examples of images displayed on the display unit 240. FIG. 1 is a schematic diagram of an inspection process; FIG.

A. First example:
A1. Device configuration:
FIG. 1 is an explanatory diagram showing an image processing apparatus as one embodiment. In this embodiment, the image processing apparatus 200 is, for example, a personal computer. The image processing apparatus 200 processes the image data of the photographed image of the object (in this embodiment, the label sheet LB pasted on the printer DV). The image processing device 200 includes a processor 210 , a storage device 215 , a display section 240 , an operation section 250 and a communication interface 270 . These elements are connected to each other via buses. Storage 215 includes volatile storage 220 and nonvolatile storage 230 .

The display unit 240 is a device configured to display an image, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device such as a button, a lever, a touch panel overlaid on the display unit 240, and configured to receive an operation by a user. A user can input various instructions to the image processing apparatus 200 by operating the operation unit 250 . The communication interface 270 is an interface for communicating with other devices (eg, USB interface, wired LAN interface, IEEE802.11 wireless interface). A digital camera 100 is connected to the communication interface 270 . The digital camera 100 optically reads the label sheet LB pasted on the printer DV to generate image data representing the read label sheet LB. In this embodiment, the label sheet LB is a rectangular sheet representing various objects including character strings. Appropriate character strings are associated in advance with printer DV models. It should be noted that manufacturing errors can cause defects in the character string on the label sheet LB. For example, label sheet LB may represent an incorrect string. Also, part or all of the appropriate character string may be missing. In this embodiment, the image processing apparatus 200 inspects label sheets for defects.

The processor 210 is a device configured to perform data processing, such as a CPU. The volatile memory device 220 is, for example, a DRAM, and the non-volatile memory device 230 is, for example, a flash memory. The nonvolatile storage device 230 stores programs 231 , 232 , 233 , the first model 500 , the first type training data 590 , the second generation model 600 and the second type training data 690 . Models 500, 600 are each predictive models with artificial neural networks. First model 500 includes first generative model 510 and first discriminative model 520 .

The first generative model 510 of the first model 500 uses the image data representing the label sheet LB optically read by the digital camera 100 to generate a computer graphics image (details will be described later) of the same label sheet LB. generates image data such as The first generative model 510 is trained to mitigate the effects of noise in the image and reading conditions. First discriminative model 520 is used for training first generative model 510 . In this example, the first model 500 is a program module. The first type training data 590 is image data for training the first model 500 .

The second generative model 600 uses the image data generated by the first generative model 510 to generate image data representing defect-free label sheets. The second generative model 600 is trained so that even if the label sheet represented by the image data input to the second generative model 600 has defects, image data representing a label sheet without defects is generated. be done. In this embodiment, the second generative model 600 is a program module. The second type training data 690 is image data for training the second generative model 600 .

A first program 231 is a program for training the first model 500 .
A second program 232 is a program for training the second generative model 600 . The third program 233 is a program for label sheet inspection processing. Processor 210 temporarily stores various intermediate data used to execute programs 231, 232, and 233 in storage device 215 (eg, either volatile storage device 220 or nonvolatile storage device 230). . Details of the programs 231, 232, 233 and the models 500, 600 will be described later.

A2. Configuration of the first model 500:
FIG. 2 is a block diagram showing an example of the first model 500. As shown in FIG. The first model 500 is constructed using so-called Generative Adversarial Networks (GANs). The first model 500 has a first generative model 510 (ie, generative network) and a first discriminative model 520 (ie, discriminative network).

The image data I5id of the image I5i representing the label sheet LBi read by the digital camera 100 is input to the first generation model 510 of the first model 500 . The first generation model 510 generates image data I51d representing an image I51 of the same label sheet LBi as the label sheet LBi represented by the input image data I5id. The position and size of the label sheet LBi in the generated image I51 are the same as the position and size of the label sheet LBi in the input image I5i. First generative model 510 is trained so that image I51 of image data I51d generated by first generative model 510 approaches a computer graphics image. A computer graphics image is an image generated by a computer. Computer graphics images are generated, for example, by rendering (also called rasterizing) vector data containing drawing commands for drawing objects. An image representing an object read by a reading device such as the digital camera 100 (also referred to as a read image) may contain various noises generated by the reading device. Moreover, the read image can vary in various ways depending on the reading conditions such as the brightness of the light source during reading, the color of the light source, and the positional relationship between the object and the reading device. The color of the object in the read image can vary depending on the brightness of the light source and the color of the light source. The shape of the object in the read image can be geometrically distorted depending on the positional relationship between the object and the reader (e.g. a rectangular object looks like a trapezoid or parallelogram in the read image). ). Computer graphics images are produced insensitive to reading conditions and noise introduced by the reading device. The trained first generative model 510 mitigates the effects of reading conditions and noise contained in the image of the label sheet.

Hereinafter, the image data I5id input to the first model 500 is also called input image data I5id, and the image I5i of the input image data I5id is also called input image I5i. The image data I51d generated by the first model 500 is also called first generated image data I51d, and the image I51 of the first generated image data I51d is also called first generated image I51. In this embodiment, the images I5i and I51 are each rectangular images (256 pixels high and 512 pixels wide). The shapes of the images I5i and I51 are the same as the shape of the label sheet. Image data I5id and I51d are color bitmap data indicating the color of each pixel in three channels of R (red), G (green) and B (blue). The number of elements of the input image data I5id (here, height*width*number of channels) is 256*512*3. Each color value of RGB is represented by 256 gradations from 0 to 255.

Note that the label sheet LBi of the input image I5i may have defects such as missing character strings and writing errors. As will be described later, if the label sheet LBi of the input image I5i has a defect, the first generated image I51 represents the label sheet LBi having the same defect.

The first generative model 510 is configured using a so-called autoencoder. The first generative model 510 has a first encoder 512 and a first decoder 517 . The first encoder 512 performs dimension reduction processing on the input image data to generate first feature data 515 indicating features of the input image. In this embodiment, the configuration of the first encoder 512 is as follows. The first encoder 512 has five convolutional layers arranged in order from the upstream side. Each convolutional layer uses a stride of 2 to reduce the number of elements corresponding to the height and the number of elements corresponding to the width (the number of elements corresponding to the number of channels increases). Batch normalization is applied to each of the second to fifth convolutional layers. The activation function for each layer is LeakyReLU. The five convolution layers change the number of data elements from "256*512*3" to "128*256*64", "64*128*128", "32*64*256", "16* 32*512” and then “8*16*512”. The number of elements of the first feature data 515 is 8*16*512.

The first decoder 517 executes dimension restoration processing on the first feature data 515 to generate first generated image data I51d. In this embodiment, the configuration of the first decoder 517 is as follows. The first decoder 517 has five transposed convolution layers arranged in order from the upstream side. Each transposed convolutional layer uses a stride of 2 or more (2 in this example) to increase the number of elements corresponding to the height and the number of elements corresponding to the width (the number of elements corresponding to the number of channels is reduced). Batch normalization is applied to the first to fourth transposed convolutional layers. Dropout is also applied to the first two transposed convolutional layers during training. The activation function for each layer is ReLU. As the activation function of the final transposed convolutional layer, a function suitable for generating the image data I51d (for example, Tanh function or sigmoid function) is used. The number of elements of the first generated image data I51d is the same as the number of elements of the input image data I5id.

Also, in this embodiment, the first generative model 510 has an architecture similar to the so-called U-Net. A plurality of connections 519 are provided between the first encoder 512 and the first decoder 517 . Specifically, the output data from the activation function of the first convolutional layer is concatenated with the output data from the activation function of the fourth transposed convolutional layer. Similarly, the pair of the second convolutional layer and the third transposed convolutional layer, the pair of the third convolutional layer and the second transposed convolutional layer, the pair of the fourth convolutional layer and the first For each transposed convolutional layer pair, the output data from the activation function of the convolutional layer is concatenated with the output data from the activation function of the transposed convolutional layer. Concatenation of the two output data is performed such that the concatenated output data represents the respective channels of the two output data. For example, when output data having 64*128*256 elements and output data having 64*128*128 elements are concatenated, data having 64*128*384 elements are generated. The number of data elements is reduced from "8*16*512" to "16*32*512 (transposed convolution)" and "16*32*1024 (concatenated)", "32*64*256 (transposed convolution)", "32*64*512 (concatenated)", "64*128*256 (transposed convolution)", "64 *128*384 (concatenated)”, “128*256*128 (transposed convolution)”, “128*256*192 (concatenated)”, and “256*512*3 (transposed convolution)”. A plurality of connections 519 convey information about position from the first encoder 512 to the first decoder 517 . Thereby, the first decoder 517 can generate the image data I51d of the image I51 representing the detailed shape. Such a connection 519 is also called a skip connection.

A first discriminative model 520 is trained to discriminate between real image data and fake image data. Specifically, the first identification model 520 identifies the first generated image data I51d generated by the first generation model 510 as fake data, and the image data Iod of the computer graphics image Io representing the label sheet as real data. trained to identify On the other hand, first generative model 510 is trained such that trained first discriminant model 520 incorrectly discriminates first generated image data I51d from real data. Through such training, the first generative model 510 can generate the first generated image data I51d of the first generated image I51 close to the computer graphics image Io.

The first identification model 520 performs dimension reduction processing on the input image data to generate identification data 529 representing one value. In this embodiment, the configuration of the first identification model 520 is as follows. The first discriminant model 520 has a convolutional layer group 522, a global average pooling layer 525, and a fully connected layer group 527 arranged in order from the upstream side. As will be described later, partial image data PA1d and PA2d representing partial images PA1 and PA2 that are parts of the images I51 and Io are input to the first identification model 520 . The number of elements of the partial images PA1 and PA2 is 64*64*3.

The convolutional layer group 522 has three convolutional layers arranged in order from the upstream side. Each convolutional layer uses a stride of 2 to reduce the number of elements corresponding to the height and the number of elements corresponding to the width (the number of elements corresponding to the number of channels increases). Batch normalization is applied to the three convolutional layers. The activation function for each convolutional layer is LeakyReLU. The three convolution layers change the number of data elements in the order of "64*64*3", "32*32*64", "16*16*128", and "8*8*128". Let

A global average pooling layer 525 computes average values for each channel. In this example, the number of data elements output from the global average pooling layer 525 is 128 elements.

The fully bonded layer group 527 has three fully bonded layers arranged in order from the upstream side. The fully connected layer group 527 reduces the number of elements to one after increasing the number of elements. In this example, the three fully connected layers vary in number of elements from 128 to 512 to 256 to 1. Note that the number of elements may decrease monotonously. Dropout is also applied to the second fully connected layer during training. The activation function of the first two fully connected layers is ReLU. As the activation function of the final fully connected layer, a function suitable for generating identification data 529 (eg, sigmoid function or Tanh function) is used.

In this embodiment, the identification data 529 indicates a value of zero or more and one or less. "1" indicates "true" and "0" indicates "false". The identification data 529 indicates the degree of certainty that the input image data is image data of a computer graphics image.

A3. Configuration of the second generative model 600:
FIG. 3A is a block diagram showing an example of the second generative model 600. As shown in FIG. Image data I 61 d in the drawing is image data input to the second generative model 600 . In this embodiment, the image data I61d is first generated image data generated by the first generation model 510 (FIG. 2) (also referred to as first generated image data I61d). The first generated image data I61d represents the image I61 of the label sheet LBj. The second generative model 600 uses the first generated image data I61d to generate image data I62d representing the defect-free label sheet LBjx. As described above, the label sheet LBj of the image I61 of the first generated image data I61d may have various defects. The trained second generation model 600 generates image data I62d representing the image I62 of the label sheet LBjx without defects even if the label sheet LBj has defects. The position and size of the label sheet LBjx in the generated image I62 are the same as the position and size of the label sheet LBj in the input image I61. Hereinafter, the image data I62d generated by the second generation model 600 will also be referred to as the second generated image data I62d, and the image I62 of the second generated image data I62d will also be referred to as the second generated image I62. In this embodiment, the second generated image data I62d is the same color bitmap data as the first generated image data I61d. The number of elements (here, height*width*number of channels) of the second generated image data I62d is 256*512*3.

The second generative model 600 is constructed using a so-called autoencoder. In this embodiment, the configuration of the second generative model 600 is the same as the configuration of the first generative model 510 (FIG. 2). The second generative model 600 has a second encoder 602 and a second decoder 607 . The configuration of the second encoder 602 is the same as the configuration of the first encoder 512 (FIG. 2), and the configuration of the second decoder 607 is the same as the configuration of the first decoder 517 (FIG. 2). A plurality of connections 609 are provided between the second encoder 602 and the second decoder 607 . The second encoder 602 performs dimension reduction processing on the input image data to generate feature data 605 indicating features of the input image.

A4. First training process:
4 and 5 are flowcharts showing an example of the first training process for the first model 500. FIG. FIG. 5 shows the processing subsequent to FIG. The processor 210 (FIG. 1) executes the processes of FIGS. 4 and 5 according to the first program 231. FIG.

At S<b>210 , the processor 210 generates input image data representing the image of the label sheet read by the digital camera 100 . In this embodiment, the operator positions the printer (eg, printer DV) so that the label sheet (eg, label sheet LB (FIG. 1)) is positioned within the shooting range of the digital camera 100 . Here, defect-free label sheets are used. The processor 210 causes the digital camera 100 to photograph the label sheet and acquires photographed image data from the digital camera 100 . A photographed image is an image of a label sheet read by the digital camera 100 . The processor 210 performs cropping processing for extracting the label sheet area from the photographed image. In this embodiment, the processor 210 displays the captured image on the display unit 240, and the operator designates a label area indicating the label sheet in the captured image. Processor 210 uses the portion of the captured image data corresponding to the labeled region to generate input image data representing an image of the labeled region (also referred to as reference input image data). The processor 210 adjusts the height and width of the input image represented by the reference input image data to the height and width of the image accepted by the first model 500 .

At S215, the processor 210 generates a plurality of input image data representing different images from each other by performing a manipulation process, also called data augmentation, on the reference input image data. In this embodiment, the processor 210 randomly selects one or more processing operations from a plurality of predetermined candidate processing operations (for example, the processing operations are selected according to random numbers). Processor 210 then generates new input image data by performing the selected processing on the reference input image data. The processor 210 selects and executes processing operations multiple times.

In this embodiment, the plurality of candidate processing operations includes a plurality of defect forming operations for forming label sheet defects. A plurality of defect forming processes may include a process of deleting part of the character string of the label sheet (the color of the pixels corresponding to the deleted character is the same color as the color of the background portion in the vicinity of the deleted character). ). A plurality of fault-forming actions may include changing a character to another character. Defect forming operations may include removing portions of the image. Such a defect formation process can generate image data that would be generated if a label sheet with defects were photographed.

Further, the plurality of candidate processing processes include a plurality of condition change processes corresponding to changes in reading conditions of the label sheet. A plurality of condition change processes may include tone curve adjustment process, white balance adjustment process, and noise addition process. Such condition change processing can generate image data that can be generated when label sheets are photographed under various reading conditions.

Processor 210 generates a plurality of input image data representing various non-defective label sheets and a plurality of input image data representing various defective label sheets by performing various processing operations. Any input image data represents an image of an optically read label sheet. Note that the input image data of the defective label sheet may be generated by photographing the defective label sheet.

At S220, the processor 210 generates sets of image pairs of the input image data and the original image data representing the defect-free label sheets. Then, the processor 210 associates the input image data with the original image data for each image pair and stores them in the storage device 215 (nonvolatile storage device 230 in this embodiment). The stored sets of image pairs constitute Type 1 training data 590 . The original image data is bitmap data generated by rendering original vector data prepared in advance, and represents a computer graphics image of the label sheet. The original vector data is data prepared in advance for manufacturing the label sheet. The height and width of the image of the original image data are the same as the height and width of the first generated image generated by the first generation model 510 (FIG. 2), respectively. In this embodiment, the original image data is common to multiple sets of image pairs. The total number of original image data included in the type 1 training data 590 may be one. It should be noted that manufacturing errors can produce defective label sheets.

Note that in S210, the label sheet may be photographed a plurality of times. Shooting conditions (for example, brightness of a light source (not shown), etc.) may differ between multiple shots. Processor 210 may then generate a plurality of reference input image data. At S215, the processor 210 may generate a plurality of input image data using the plurality of reference input image data.

At S225, the processor 210 calculates a plurality of operational parameters of the first model 500 (FIG. 2), ie, a plurality of operational parameters of the first generative model 510 and the first discriminative model 520 (e.g., a plurality of convolution filters). weights and multiple biases). For example, each calculation parameter is set to a random value.

In S230, the processor 210 selects a subset composed of a plurality of target pairs, which are a plurality of image pairs to be processed, from the plurality of sets of image pairs included in the type 1 training data 590. FIG. A plurality of unprocessed image pairs are selected as the subset. Also, the total number of image pairs in the subset is predetermined. Hereinafter, it is assumed that the input image data I5id and the image data Iod in FIG. 2 indicate the target pair. Here, the image data Iod is the original image data (the image data Iod is also called the original image data Iod, and the image Io is also called the original image Io).

At S235, the processor 210 generates the first generated image data I51d by inputting the target pair of input image data I5id into the first generative model 510. FIG. Specifically, the processor 210 uses the calculation parameters of the first generative model 510 to calculate each layer of the first generative model 510 to generate the first generated image data I51d.

In S240, the processor 210 randomly determines a first partial area A1, which is a partial area in the first generated image I51. The processor 210 randomly determines a preselected parameter among a plurality of parameters (eg, position, shape, size, number, etc.) defining the first partial area A1. In this embodiment, the processor 210 generates a random number and determines the first position P1, which is the position of the first partial area A1, according to the random number. The shape and size of the first partial area A1 are predetermined. Processor 210 also determines one first partial area A1 for one first partial image PA1.

In S245, the processor 210 executes a first acquisition process using the first generated image data I51d and the first partial area A1 to obtain a first partial image representing at least part of the image of the first partial area A1. First partial image data PA1d of PA1 is obtained. In this embodiment, the processor 210 obtains the first partial image data PA1d of the first partial image PA1 representing the image of the first partial area A1 having the predetermined shape and size located at the first position P1. That is, processor 210 adopts the image of first partial area A1 as it is as first partial image PA1. Alternatively, the first acquisition process may include various image processing such as image rotation and image scaling (enlargement or reduction) (details below).

At S<b>250 , processor 210 generates identification data 529 by inputting first partial image data PA<b>1 d to first identification model 520 . Specifically, processor 210 calculates identification data 529 by calculating each layer of first identification model 520 using calculation parameters of first identification model 520 . Hereinafter, the identification data 529 obtained from the first partial image data PA1d (that is, the first generated image data I51d) will also be referred to as first identification data 529f.

At S255, the processor 210 randomly determines a second partial area A2, which is a partial area in the original image Io. The processor 210 randomly determines a preselected parameter among a plurality of parameters (eg, position, shape, size, number, etc.) defining the second partial area A2. In this embodiment, the processor 210 generates a random number and determines the second position P2, which is the position of the second partial area A2, according to the random number. The shape and size of the second partial area A2 are predetermined. Processor 210 also determines one second partial area A2 for one second partial image PA2.

In S260, the processor 210 executes a second acquisition process using the original image data Iod and the second partial area A2 to generate a second partial image PA2 showing at least part of the image of the second partial area A2. Acquire the second partial image data PA2d. In this embodiment, the processor 210 acquires the second partial image data PA2d of the second partial image PA2 representing the image of the second partial area A2 having the predetermined shape and size located at the second position P2. That is, processor 210 adopts the image of second partial area A2 as it is as second partial image PA2. Alternatively, the second acquisition process may include various image processing such as image rotation and image scaling (described in detail below).

At S<b>265 , processor 210 generates identification data 529 by inputting second partial image data PA<b>2 d to first identification model 520 . Specifically, processor 210 calculates identification data 529 by calculating each layer of first identification model 520 using calculation parameters of first identification model 520 . Hereinafter, the identification data 529 obtained from the second partial image data PA2d (that is, the original image data Iod) will also be referred to as second identification data 529r.

At S270 (FIG. 5), the processor 210 calculates an identification loss according to a predetermined identification loss function using the first identification data 529f and the second identification data 529r. The identification loss function may be calculated as a smaller value as the second identification data 529r is closer to “real (1)”, or as a smaller value as the first identification data 529f is closer to “false (zero)”. can be a function of For example, when the second identification data 529r is Dr and the first identification data 529f is Df, the identification loss function may be the following function Fa(Dr, Df).
Fa (Dr, Df) = (1-Dr) + (Df)

In this embodiment, the subset (FIG. 4: S230) includes multiple target pairs. Processor 210 performs the processing of S235-S265 for each of the plurality of target pairs. Then, in S270, the processor 210 calculates, as the identification loss, the total value of the plurality of losses obtained from the plurality of target pairs according to the identification loss function.

In S275, the processor 210 adjusts the multiple calculation parameters of the first discrimination model 520 according to a predetermined algorithm so that the discrimination loss becomes small. As an algorithm, for example, an algorithm using backpropagation and gradient descent may be employed. The processor 210 may also perform so-called Adam's optimization. In this example, processor 210 also applies batch normalization to each convolutional layer of convolutional layers 522 . Processor 210 applies dropout to the second fully connected layer of fully connected layers 527 .

The identification loss function is such that the closer the second identification data 529r is to "real (1)", the larger the value calculated, and the closer the first identification data 529f is to "false (zero)," the larger the calculated value is. can be a function. For example, the identification loss function may be the following function Fb(Dr, Df).
Fb(Dr, Df) = log(Dr) + log(1-Df)
In this case, the higher the discrimination loss, the better the performance of the first discrimination model 520 . The processor 210 adjusts a plurality of calculation parameters of the first discriminant model 520 so as to increase the discriminative loss. In this case, for the first identification model 520, the identification loss is not a loss that should be reduced, but an evaluation value that should be increased. A function −Fb(Dr, Df) obtained by adding a minus sign to the function Fb(Dr, Df) can be used as a loss function representing a loss to be reduced.

S280 and S285 are the same as S240 and S245 in FIG. 4, respectively. In S280 and S285, processor 210 determines a new first position P1 and acquires new first partial image data PA1d.

In S290, processor 210 generates new first identification data 529f by inputting new first partial image data PA1d to first identification model 520. FIG. Since the first discriminant model 520 has been trained in S275, the first discriminant data 529f calculated in S290 can be closer to "false (zero)."

At S295, the processor 210 calculates a production loss according to a predetermined production loss function using the first identification data 529f produced at S290. The generation loss function may be various functions that calculate a smaller value as the first identification data 529f is closer to "real (1)". For example, when the first identification data 529f is Df, the generation loss function may be the following function Fc(Df).
Fc(Df) = (1-Df)

In this example, the subset (FIG. 4: S230) includes multiple sets of target pairs, so at S235 the processor 210 generates multiple first generated image data. Processor 210 performs the processing of S280-S290 (FIG. 5) for each of the plurality of sets of first generated image data. Then, in S295, the processor 210 calculates, as the generated loss, the total value of the plurality of losses obtained from the plurality of first generated image data according to the generated loss function.

In S300, the processor 210 adjusts a plurality of calculation parameters of the first generative model 510 according to a predetermined algorithm so as to reduce the generative loss. As an algorithm, for example, an algorithm using backpropagation and gradient descent may be employed. The processor 210 may also perform so-called Adam's optimization. Also, in this example, the processor 210 performs four convolutional layers, 2nd through 5th, of the first encoder 512 and four transposed convolutional layers, 1st through 4th, of the first decoder 517. and to apply batch normalization. The processor 210 applies dropout to the first two transposed convolutional layers of the first decoder 517 .

At S305, the processor 210 determines whether an end-of-training condition is met. A training end condition may be any condition that indicates that the first model 500 has been properly trained. In this embodiment, the training end condition is input of an end instruction from the operator. Processor 210 randomly obtains a predetermined number of image pairs from the plurality of image pairs not used for training in Type 1 training data 590 (FIG. 1). The processor 210 inputs the input image data of each of the acquired image pairs to the first generative model 510 and acquires the first generative image data from the first generative model 510 . The processor 210 displays on the display unit 240 a plurality of pairs of the inputted input image and the outputted first generated image. The operator observes the display unit 240 to check whether the first generated image appropriately represents the computer graphics image of the same label sheet as the input image. The worker operates the operation unit 250 according to the confirmation result to input a training end instruction or a training continuation instruction.

Note that the training end condition may be another condition. For example, the termination condition may be that the discrimination loss and generation loss calculated using a first predetermined number of image pairs not used for training are less than or equal to respective predetermined thresholds. you can

If it is determined that the training has not ended (S305: No), the processor 210 proceeds to S230 of FIG. 4 and executes processing of a new subset. If it is determined that training has ended (S305: Yes), in S310 the processor 210 stores the trained first generative model 510 in the storage device 215 (here, the non-volatile storage device 230). Processor 210 then terminates the processing of FIGS. A trained first generative model 510 can convert a read image of a label sheet into a computer graphics image of the same label sheet. It should be noted that the processor 210 may delete the first discriminant model 520 from the non-volatile storage device 230 when the training is finished.

It should be noted that the above-described noise generated by the reading device and the reading conditions affect the entire read image (for example, the input image I5i (FIG. 2)). On the other hand, such effects on computer graphics images (eg, original image Io (FIG. 2)) are small. Therefore, even if only a portion of the read image and only a portion of the computer graphics image are available, it is easy to distinguish between the read image and the computer graphics image. Thus, the first discriminant model 520 (FIG. 2) is such that the input image is a computer graphics image, even if the partial image PA1, PA2 is input instead of the entire image I51, Io of the label sheet. It can be properly identified whether or not Also, in the training of this embodiment, the diversity of image data input to the first discriminant model 520 increases. For example, a plurality of second partial images PA2 representing different parts of the original image Io are input to the first identification model 520. FIG. Therefore, the speed of training of the first discriminant model 520 is reduced. As a result, the training of the first generative model 510 and the training of the first discriminative model 520 can be balanced. Therefore, in the training of this embodiment, the image I51 of the image data I51d generated by inputting the image data I5id representing the image I5i of the optically read label sheet into the first generation model 510 is generated by the computer of the label sheet. The first generative model 510 can be properly trained to approximate graphics images.

Further, in the training of the present embodiment, the first discrimination model 520 uses the first partial image data PA1d of the first partial image PA1 representing the image of the first partial area A1 in the first generated image I51 as fake image data. use. Also, the first identification model 520 uses the second partial image data PA2d of the second partial image PA2 representing the image of the second partial area A2 in the original image Io as the actual image data. The original image Io represents a label sheet with no defects. Therefore, the training of this embodiment can train the first generative model 510 so that the portion of the label sheet represented by the first partial image PA1 approaches the corresponding portion of the label sheet without defects. Various input image data I5id of various input images I5i including non-defective portions of the label sheet are input to the first generation model 510 . Therefore, the training of this embodiment can train the first generative model 510 to generate the first generative image I51 representing the same portion of the label sheet in the input image I5i as the defect-free portion. If the label sheet LBi of the input image I5i has a defect, the first generated image I51 may include a similar defective portion due to the influence of the defective portion included in the input image I5i. In the training of the present embodiment, the first partial area A1 can be determined as a part different from the part showing the defect in the first generated image I51. Accordingly, the first generative model 510 can be trained such that portions exhibiting similar defects in the first generated image I51 caused by portions exhibiting defects contained in the input image I5i remain intact. As described above, when the label sheet LBi of the input image I5i has a defect, the trained first generation model 510 generates the first generated image I51 representing the label sheet LBi having the same defect.

Processor 210 also randomly determines a first position P1 of first partial area A1 for first partial image PA1 (FIG. 4: S240). Processor 210 randomly determines a second position P2 of second partial area A2 for second partial image PA2 (S255). Accordingly, partial image data PA1d, PA2d of partial images PA1, PA2 at various positions P1, P2 are input to the first identification model 520. FIG. In this manner, proper training of the first generative model 510 and the first discriminative model 520 is possible as the diversity of image data input to the first discriminative model 520 is increased.

A5. Second training process:
FIG. 6 is a flowchart showing an example of the second training process for the second generative model 600. As shown in FIG. FIG. 3B is a block diagram showing an overview of the training of the second generative model 600. As shown in FIG. In this embodiment, the second generative model 600 is trained to generate defect-free label sheet image data when various label sheet image data are input. Processor 210 (FIG. 1) executes the process of FIG. 6 according to second program 232 .

In S410, the processor 210 generates a plurality of training image data using the original image data Iod described in FIG. 230). A plurality of stored training image data constitutes type 2 training data 690 (FIG. 1). In this embodiment, the processor 210 generates a plurality of training image data representing different images by performing a manipulation process, also called data augmentation, on the original image data Iod. Similar to S215 (FIG. 4), the processor 210 selects a processing process from a plurality of predetermined candidate processing processes and executes the selected processing process on the original image data Iod multiple times. The plurality of candidate processings used in S410 are composed of a plurality of processings for maintaining defect-free label sheets (for example, tone curve adjustment processing, white balance adjustment processing, noise addition processing, etc.). Processing that produces defects in the label sheet, such as processing text areas, is not used. Therefore, all of the plurality of training image data represent defect-free label sheets.

At S415, the processor 210 initializes a plurality of operational parameters of the second generative model 600 (eg, weights and biases of the convolution filter, etc.). For example, each calculation parameter is set to a random value.

In S420, the processor 210 selects a subset composed of a plurality of target training image data, which are a plurality of data to be processed, from a plurality of training image data included in the second type training data 690. FIG. A plurality of unprocessed training image data is selected as the subset. The total number of image data in the subset is predetermined.

At S425, the processor 210 generates the second generated image data I6xd by inputting the target training image data I6td (FIG. 3B) into the second generation model 600 (FIG. 3). Specifically, the processor 210 uses the calculation parameters of the second generative model 600 to calculate each layer of the second generative model 600 to generate the second generated image data I6xd.

At S430, the processor 210 calculates the reconstruction loss according to a predetermined reconstruction loss function using the target training image data I6td input to the second generative model 600 and the second generated image data I6xd. calculate. The reconstruction loss function can be various functions representing the difference between the two image data. In this example, the so-called cross-entropy is used as the reconstruction loss function. Alternatively, the sum-of-squares error may be used as the reconstruction loss function. In this example, the subset (S420) includes a plurality of target training image data. The processor 210 executes the process of S425 for each of the plurality of target training image data to generate a plurality of second generated image data. In S430, the processor 210 calculates the total value of multiple losses L2 obtained from multiple pairs of the target training image data and the second generated image data according to the reconstruction loss function as the reconstruction loss.

In S435, processor 210 adjusts a plurality of calculation parameters of second generative model 600 according to a predetermined algorithm so that reconstruction loss is reduced. As an algorithm, for example, an algorithm using backpropagation and gradient descent may be employed. The processor 210 may also perform so-called Adam's optimization. Also, in this embodiment, the processor 210 performs four convolutional layers, 2nd through 5th, of the second encoder 602 and four transposed convolutional layers, 1st through 4th, of the second decoder 607. and to apply batch normalization. The processor 210 applies dropout to the first two transposed convolutional layers of the second decoder 607 .

At S440, the processor 210 determines whether an end-of-training condition is met. A training end condition may be any condition that indicates that the second generative model 600 has been properly trained. In this embodiment, the training end condition is input of an end instruction from the operator. Processor 210 randomly acquires a predetermined number of image data from a plurality of training image data not used for training in type 2 training data 690 (FIG. 1). The processor 210 inputs the obtained plurality of image data to the second generative model 600 and obtains the plurality of second generated image data from the second generative model 600 . Processor 210 displays a plurality of pairs of input images and output second generated images on display unit 240 . The operator observes the display unit 240 to check whether the second generated image appropriately represents the image of the same label sheet as the input image. The worker operates the operation unit 250 according to the confirmation result to input a training end instruction or a training continuation instruction.

Note that the training end condition may be another condition. For example, the termination condition may be that the reconstruction loss calculated using a second predetermined number of training image data not used for training is less than or equal to a predetermined reconstruction threshold.

If it is determined that the training has not been completed (S440: No), the processor 210 moves to S420 and processes a new subset. If it is determined that the training has ended (S440: Yes), in S445 the processor 210 stores the trained second generative model 600 in the storage device 215 (here, the non-volatile storage device 230). Processor 210 then terminates the process of FIG. As described above, the second generative model 600 is trained to generate image data of the same label sheet without defects from image data of various label sheets without defects. Therefore, even if the input image data I61d (FIG. 3A) represents the defective label sheet LBj, the trained second generative model 600 can generate the image data representing the defective label sheet LBjx. Generate I62d.

A6. Inspection process:
FIG. 7 is a flowchart illustrating an example of inspection processing. Inspection processing is executed by the image processing device 200 (FIG. 1). Processor 210 executes inspection processing according to third program 233 .

In S<b>510 , the processor 210 causes the digital camera 100 to photograph the label sheet of interest, which is the label sheet to be inspected, and acquires photographed image data of interest from the digital camera 100 . FIG. 8 is a schematic diagram of the inspection process. The image data Ixd in the figure is the photographed image data of interest Ixd of the photographed image of interest Ix. The photographed image of interest Ix represents a portion of the printer DV including the label sheet of interest LBx. In the example of FIG. 8, the attention label sheet LBx has a character string defect (details will be described later).

In S515, the processor 210 performs cropping processing for extracting the region of the label sheet of interest from the photographed image of interest. In this embodiment, the processor 210 uses a machine learning model called an object detection model (eg, YOLOv4 (You only look once), SSD (Single Shot MultiBox Detector), R-CNN (Region Based Convolutional Neural Networks), etc.). to detect the area of the target label sheet LBx. The processor 210 uses the portion of the photographed image data of interest Ixd that indicates the detected area to generate the input image data of interest Ix0d representing the label sheet of interest LBx. The processor 210 adjusts the height and width of the input image of interest Ix0 represented by the input image data of interest Ix0d to the height and width of the image accepted by the first model 500 . Any other method may be used to detect the area of the target label sheet LBx. For example, the processor 210 may detect the area of the attention label sheet LBx by pattern matching using a predetermined reference pattern.

In S<b>520 , the processor 210 generates first generated image data of interest Ix1d by inputting the input image data of interest to the first generative model 510 . The processor 210 uses the calculation parameters of the first generative model 510 to calculate each layer of the first generative model 510 to generate the first generated image data of interest Ix1d. The first attention generated image Ix1 of the first attention generated image data Ix1d is an image of the attention label sheet LBx, such as a computer graphics image.

In S<b>530 , the processor 210 generates the second generated image data of interest Ix2d by inputting the first generated image data of interest Ix1d to the second generation model 600 . The processor 210 uses the calculation parameters of the second generative model 600 to calculate each layer of the second generative model 600 to generate the second generated image data of interest Ix2d. The second generated image of interest Ix2 of the second generated image of interest Ix2d represents the label sheet LBxm with no defects (also referred to as the correct label sheet LBxm). In the example of FIG. 8, the correct label sheet LBxm represents three character strings T1, T2, and T3. On the other hand, as indicated by the input image of interest Ix0, the label sheet of interest LBx represents two character strings T1 and T2x. The rightmost character T2e of the second character string T2x is an incorrect "9" instead of the correct "2" of the second character string T2 on the correct label sheet LBxm. On the attention label sheet LBx, the third character string T3 is replaced with a blank area Ax.

In S535, the processor 210 generates intermediate difference image data Ix3d of an intermediate difference image Ix3 indicating the difference between the first generated image of interest Ix1 and the second generated image of interest Ix2. Each pixel of the intermediate difference image Ix3 indicates the difference between the first color value of the corresponding pixel of the first generated image of interest Ix1 and the second color value of the corresponding pixel of the second generated image of interest Ix2. there is In this embodiment, the first color value and the second color value are RGB values. The processor 210 calculates the L1 distance between the first color value and the second color value as the pixel value of the pixel in the intermediate difference image Ix3. Note that the pixel value of the pixels of the intermediate difference image Ix3 may be any value indicating the difference between the first generated image of interest Ix1 and the second generated image of interest Ix2 instead of the L1 distance (for example, L2 distance, luminance value difference calculated from RGB values, etc.).

The intermediate difference image Ix3 may contain various noises. For example, the color values of pixels indicating the boundary between an object such as a character and the background may differ between the first generated image of interest Ix1 and the second generated image of interest Ix2. The intermediate difference image Ix3 can represent such color value deviations. Therefore, in this embodiment, the processor 210 generates differential image data Ix4d of the final differential image Ix4 by executing binarization processing of the intermediate differential image data Ix3d. The threshold value for the binarization process is experimentally determined in advance so that the value of a pixel showing a defect is set to 1, and the value of each pixel showing no defect and a pixel showing noise is set to 0. be done. In the example of FIG. 8, the difference image Ix4 represents the third character string T3 missing from the target input image Ix0 and the difference image T2w of the rightmost character of the second character string T2x.

In S540, the processor 210 calculates the degree of abnormality DA by executing the dimension reduction process of the difference image data Ix4d. The degree of abnormality DA indicates the degree of defect of the label sheet of interest LBx represented by the input image data of interest Ix0d. The dimension reduction process may be various processes for calculating the degree of anomaly DA that indicates the feature of the difference image data Ix4d. In this embodiment, the processor 210 uses t-SNE (t-distributed Stochastic Neighbor Embedding) to generate two-dimensional data composed of a first value V1 and a second value V2 from the differential image data Ix4d. do. FIG. 8 shows an outline of a graph showing two-dimensional data. The horizontal axis indicates the first value V1, and the vertical axis indicates the second value V2. A point of interest Pi indicates two-dimensional data calculated from the differential image data Ix4d. The reference point Ps is a representative point of a plurality of two-dimensional data Pr obtained from a plurality of photographed images of label sheets without defects by the same processing as the processing of S515-S540. In this embodiment, the reference point Ps is the barycentric point of the plurality of two-dimensional data Pr and is determined in advance. The distance DAd is the distance between the reference point Ps and the target point Pi (sum of square error in this embodiment). Note that the distance DAd may be various values indicating the distance between the two points Ps and Pi (for example, L2 distance, L1 distance, etc.).

The processor 210 calculates the degree of abnormality DA by inputting the distance DAd into a predetermined evaluation function. The evaluation function may be various functions that calculate the degree of abnormality DA that increases as the distance DAd increases. Here, when the entire range of the distance DAd is divided into three ranges, a short range, a medium range, and a long range, the anomaly degree DA is close to zero in the short range, and the anomaly degree DA is close to zero in the long range. Close to the maximum value (for example, 100), in the middle range, it is preferable that the degree of anomaly DA changes greatly according to the change in the distance DAd. A sigmoid function or a tanh function, for example, can be used as such an evaluation function.

Thus, in this embodiment, the processor 210 executes dimension reduction processing using t-SNE and an evaluation function to calculate the degree of anomaly DA from the differential image data Ix4d. In this embodiment, the degree of anomaly DA is a value of zero or more and 100 or less.

In S545, the processor 210 (FIG. 1) displays the difference image Ix4 and the degree of abnormality DA on the display unit 240. FIG. FIG. 9A is a schematic diagram showing an example of an image displayed on the display unit 240. FIG. The display unit 240 displays a difference image Ix4 and a character string DAt indicating the degree of abnormality DA. In FIG. 9A, the degree of abnormality DA is 80, and "%" is shown as the unit of the degree of abnormality DA. By observing the display unit 240, the operator can easily recognize the defective portion of the attention label sheet LBx and the degree of abnormality.

In the embodiment of FIG. 7, the processor 210 executes the processes of S550, S555 and S560 in parallel with the processes of S530-S545. FIG. 10 is a schematic diagram of the inspection process. FIG. 10 outlines S550 and S555. In S550 (FIG. 7), the processor 210 performs character recognition processing on the first generated image data of interest Ix1d, and obtains a character string table of interest Tx that lists the character strings represented by the label sheet of interest LBx. The character recognition processing may be various known processing. For example, an optical character recognition engine called "Tesseract OCR" from Google may be used. In this embodiment, the processor 210 uses the first generated image data of interest Ix1d instead of the input image data of interest Ix0d containing noise for character recognition. Therefore, the processor 210 can obtain proper character recognition results.

FIG. 10 shows an example of the noted character string table Tx. As illustrated, the target character string table Tx indicates a first character string T1 and a second character string T2x. The rightmost character T2e of the second string T2x is a false "9". The third string T3 is not recognized.

In S555, the processor 210 compares the noted character string table Tx and the original character string table To to generate a defective character string table Td showing a list of defective character strings. The original character string table To shows a list of character strings represented by label sheets with no defects, and is prepared in advance. The processor 210 compares the noted character string table Tx and the original character string table To, and detects defects in character string units. In the example of FIG. 10, the defect character string table Td indicates a second character string T2x and a third character string T3. The rightmost character T2e of the second string T2x is a false "9".

In this way, the defect character string table Td is a character string included in the target character string table Tx that is at least partially different from the corresponding character string included in the original character string table To (for example, the second contains the string T2x). The defective character string table Td also includes character strings (for example, the third character string T3) that are included in the original character string table To but not included in the target character string table Tx. Although illustration is omitted, if the noted character string table Tx includes a character string that is not included in the original character string table To, the defective character string table Td includes that character string.

In S560, processor 210 (FIG. 1) displays the character string indicated by defect character string table Td on display unit 240. FIG. FIG. 9B is a schematic diagram showing an example of an image displayed on the display unit 240. As shown in FIG. The display section 240 displays character strings T2x and T3 included in the defect character string table Td. By observing the display unit 240, the operator can easily recognize the defects included in the noted label sheet LBx.

When the processing of S530-S545 and the processing of S550-S560 are completed, the processor 210 ends the inspection processing of FIG. Note that the processor 210 may execute the processing of S530-S545 and the processing of S550-S560 sequentially, not in parallel.

As described above, in this embodiment, the processor 210 of the image processing apparatus 200 executes the following image processing. In S520 of FIG. 7, the processor 210 inputs the input image data of interest Ix0d representing the optically read label sheet of interest LBx (FIG. 8) to the first generation model 510, thereby corresponding to the input image data of interest Ix0d. 1st generated image data of interest Ix1d is generated. 2, 4 and 5, the first generative model 510 inputs the image data I5id representing the optically read label sheet image I5i (Fig. 2) to the first generative model 510. It is a generative model trained so that the image I51 of the image data I51d generated by the above approaches the computer graphics image Io of the label sheet.

Further, in S530 of FIG. 7, the processor 210 inputs the first generated image data of interest Ix1d (FIG. 8) to the second generative model 600, thereby generating a second generated image of interest representing the image Ix2 of the object of interest without defects. Generate image data Ix2d. As described with reference to FIG. 6, the second generative model 600 generates a fault-free label sheet image I62 by inputting the image data I61d (FIG. 3A) representing the image of the label sheet into the second generative model 600. is a generative model trained to generate image data I62d representing

In addition, in S535 of FIG. 7, the processor 210 generates a difference image of a difference image Ix4 indicating the difference between the image Ix1 of the first generated image data of interest Ix1d (FIG. 8) and the image Ix2 of the second generated image data of interest Ix2d. Generate data Ix4d.

As described in FIGS. 2, 5 and 6, first generative model 510 is generated by training first model 500 having first generative model 510 and first discriminative model 520 . Here, the first generative model 510 is an example of a generative network, the first discriminant model 520 is an example of a discriminative network, and the first model 500 is a generative adversarial network having a generative network and a discriminative network. For example.

As described in S250 and S260 of FIG. 4 and FIG. ing. S235-S245, as described with reference to FIG. A first partial image PA1, which is part of the image I51, is shown. S250-S260, as described in FIG. 2, the second partial image data PA2d represents the second partial image PA2 which is a portion of the second image Io which is the training computer graphics image Io.

As described above, the diversity of the image data PA1d, PA2d (FIG. 2) input to the first discrimination model 520, which is an example of a discrimination network, is increased, and the training speed of the first discrimination model 520 is suppressed. A balance can be achieved between training the first generative model 510 and the first discriminative model 520 of the example network. Therefore, as shown in FIG. 8, the first generative model 510 is based on the image data Ix1d generated by inputting the image data Ix0d representing the image Ix0 of the optically read label sheet LBx to the first generative model 510. is properly trained to approximate the computer graphics image of the label sheet. Since the first generated image data of interest Ix1d generated by the first generation model 510 is input to the second generation model 600, the second generation model 600 is a second generation model representing the image of the label sheet of interest without defects. Noted generated image data Ix2d can be generated appropriately. Therefore, the difference image data Ix4d of the difference image Ix4 indicating the difference between the image Ix1 of the first generated image data of interest Ix1d and the image Ix2 of the second generated image data of interest Ix2d is obtained regardless of the conditions for reading the label sheet of interest LBx. Therefore, it is possible to appropriately express the defect of the attention label sheet LBx represented by the attention input image data Ix0d.

Further, in the present embodiment, in S540 (FIG. 7), the processor 210 reduces the degree of abnormality DA of the attention label sheet LBx represented by the attention input image data Ix0d by executing the dimension reduction process of the difference image data Ix4d. calculate. As described above, the differential image data Ix4d adequately represents the defect of the attention label sheet LBx, so the processor 210 can use the differential image data Ix4d to calculate an appropriate degree of abnormality DA.

Further, as described in S240 and S255 in FIG. 4 and in FIG. 2, the first position P1 of the first partial area A1 on the first image I51 is randomly determined, and the second A second position P2 of the partial area A2 is randomly determined. In training the first model 500 (FIGS. 4, 5), the processor 210 converts the partial image data PA1d, PA2d of the partial images PA1, PA2 of the partial areas A1, A2 at the various positions P1, P2 into the first discriminant model Enter 520. This increases the diversity of the image data input to the first discriminative model 520, so that proper training of the first generative model 510 and the first discriminative model 520 is possible.

Also, as mentioned above, FIGS. 4 and 5 show an example of a training process for a generative adversarial network having a generative network and a discriminative network. This training process includes the following processes. In S235, the processor 210 generates the first image data I51d of the first image I51 by inputting the training input image data I5id into the first generation model 510, which is an example of a generation network. At S240, the processor 210 randomly determines the first partial area A1 in the first image I51 (in this embodiment, the first position P1 of the first partial area A1 is randomly determined). In S245, processor 210 converts first partial image data PA1d representing first partial image PA1 showing at least part of the image of first partial area A1 in first image I51 to first image data I51d and the first part. obtained using the area A1. At S250, the processor 210 obtains the first identification data 529f by inputting the first partial image data PA1d into the first identification model 520, which is an example of an identification network. In S255, the processor 210 randomly determines the second partial area A2 in the second image Io of the second image data Iod associated with the input image data I5id (in this embodiment, the second partial area A2 is A second position P2 is randomly determined). In S260, processor 210 converts second partial image data PA2d representing second partial image PA2 showing at least part of the image of second partial area A2 in second image Io to second image data Iod and the second part. obtained using area A2. At S265, the processor 210 obtains the second identification data 529r by inputting the second partial image data PA2d into the first identification model 520. FIG. At S270, S275, the processor 210 adjusts the parameters of the first discriminative model 520 by using the first discriminative data 529f and the second discriminative data 529r. As described above, the variety of images input to the identification network for training of the adversarial generative network is improved, so that the identification network and, in turn, the adversarial generative network having the identification network and the generative network can be appropriately trained. It should be noted that the use of the adversarial generation network is not limited to the inspection of defects in objects such as label sheets, and may be any use.

B. Variant:
(1) A plurality of input image data representing different images are preferably used for training the first model 500 (FIGS. 4 and 5). Preferably, the plurality of input image data includes a plurality of input image data representing different defects of the label sheet. Moreover, it is preferable that the plurality of input image data include a plurality of input image data corresponding to reading conditions different from each other.

Further, the computer graphics image data associated with the input image data may represent the same label sheet as the label sheet represented by the input image data. That is, if the input image data represents a label sheet with a defect, the computer graphics image data associated with the input image data may represent the label sheet with the same defect. By using such a pair of input image data of interest and computer graphics image data for training, the trained first generation model 510 can, when image data representing a defective label sheet is input, Data representing computer graphics images of label sheets with the same defects can be suitably generated.

(2) In the training of the first model 500 (FIGS. 4 and 5), the processor 210 generates multiple It is preferable to input a plurality of data PA1d of first partial image PA1 to first identification model 520 . Here, the processor 210 preferably obtains data of the plurality of first partial images PA1 using the plurality of first generated image data. Alternatively, processor 210 may acquire data of a plurality of first partial images PA1 using one piece of first generated image data. Here, in S240 (FIG. 4), processor 210 calculates the number of first partial images PA1 obtained from one first generated image (that is, the number of first partial areas A1 on the first generated image) as , may be randomly determined. A plurality of first positions P1 different from each other may be predetermined instead of being randomly determined. Also, in S240, the processor 210 randomly determines the size of the first partial region A1 on the first generated image (the unit is, for example, the number of pixels in a predetermined direction (eg, width direction)). you can Also, the processor 210 may randomly determine the shape of the first partial region A1 on the first generated image from a plurality of predetermined shapes (eg, square, rectangle, rhombus, circle, ellipse, etc.). Generally, at S240, the processor 210 randomly determines one or more of a plurality of parameters including the position, shape, size and number of the first partial regions A1 on the first generated image. you can This further improves the diversity of the image data input to the first discriminant model 520 . Note that the remaining parameters may be determined in advance. Here, the processor 210 may randomly determine a parameter to be randomly determined among the plurality of parameters at S240.

Further, the first acquisition process of S245 is various processes for acquiring the first partial image data PA1d of the first partial image PA1 representing at least part of the image of the first partial area A1 in the first generated image. good. The first acquisition process may include, for example, one or both of image rotation and image scaling (enlargement or reduction). Scaling an image changes the pixel density of the image. The first acquisition processing is not limited to rotation and scaling, and may include various other image processing. The first acquisition process is preferably configured so that the first partial image PA1 generated by the first acquisition process shows the same image as at least part of the image of the first partial area A1. In the first acquisition process, the processor 210 selects one or more preselected parameters from among the one or more parameters used in the image processing of the first acquisition process (eg, a plurality of parameters including rotation angle and scaling ratio). Parameters (eg, rotation angle and/or scaling ratio) may be randomly determined. The scaling ratio is the ratio of the size of the image after scaling to the size of the image before scaling (the unit is, for example, the number of pixels in a predetermined direction (eg, width direction)). In the first acquisition process, processor 210 may randomly select an image process to perform from a plurality of image processes including rotation and scaling. In the first acquisition process, the processor 210 may randomly select a parameter to be determined randomly among the one or more parameters used in the image processing of the first acquisition process. By using randomly determined parameters in the image processing of the first acquisition process, the diversity of the image data input to the first discriminative model 520 is further enhanced. Note that the remaining parameters may be determined in advance.

In either case, the first acquisition process may comprise adjusting the first partial image PA1 to the discriminative input image, which is the image accepted by the first discriminative model 520 . Processor 210 may adjust the size (more generally, pixel density) of first partial image PA1, for example, by pixel density adjustment or convolution processing. Further, when the shape of first partial image PA1 before adjustment is different from the shape of the identification input image, processor 210 extracts a portion having the same shape as the shape of the identification input image from first partial image PA1 before adjustment. Thus, the adjusted first partial image PA1 may be generated. In this way, the first partial image PA1 may represent all or part of the image of the first partial area A1 in the first generated image. When the first acquisition process including scaling is executed, the size of the image of the first partial area A1 in the adjusted first partial image PA1 is increased as the scaling ratio increases. Image PA1 may be adjusted.

Similarly, processor 210 first identifies a plurality of data PA2d of a plurality of second partial images PA2 representing a plurality of images of a plurality of second partial areas A2 at a plurality of second positions P2 (FIG. 2) different from each other. Inputs to model 520 are preferred. Here, the processor 210 may obtain data of the plurality of second partial images PA2 using one piece of computer graphics image data (for example, image data Iod). Here, in S255 (FIG. 4), processor 210 calculates the number of second partial images PA2 obtained from one computer graphics image (that is, the number of second partial areas A2 on the computer graphics image) as , may be randomly determined. Alternatively, processor 210 may acquire data of multiple second partial images PA2 using multiple computer graphics image data. The plurality of second positions P2 different from each other may be determined in advance instead of being randomly determined. Also, in S255, the processor 210 randomly determines the size of the second partial region A2 on the computer graphics image (the unit is, for example, the number of pixels in a predetermined direction (eg, width direction)). you can Also, the processor 210 may randomly determine the shape of the second partial area A2 on the computer graphics image from a plurality of predetermined shapes (eg, square, rectangle, rhombus, circle, ellipse, etc.). Generally, at S255, the processor 210 randomly determines one or more of a plurality of parameters including the position, shape, size and number of the second partial regions A2 on the computer graphics image. you can This further improves the diversity of the image data input to the first discriminant model 520 . Note that the remaining parameters may be determined in advance. Here, the processor 210 may randomly determine a parameter to be randomly determined among the plurality of parameters at S255.

The second acquisition process of S260 is various processes for acquiring the second partial image data PA2d of the second partial image PA2 representing at least part of the image of the second partial area A2 in the computer graphics image. good. The second acquisition process is preferably configured such that the second partial image PA2 generated by the second acquisition process shows the same image as at least part of the image of the second partial area A2. The second acquisition process may be the same as the first acquisition process, and may include, for example, one or both of image rotation and image scaling. In the second acquisition process, the processor 210 selects one or more preselected parameters from among the one or more parameters used in the image processing of the second acquisition process (eg, a plurality of parameters including rotation angle and scaling ratio). Parameters (eg, rotation angle and/or scaling ratio) may be randomly determined. In the second acquisition process, processor 210 may randomly select an image process to perform from a plurality of image processes including rotation and scaling. In the second acquisition process, the processor 210 may randomly select a parameter to be determined randomly among the one or more parameters used in the image processing of the second acquisition process. By using randomly determined parameters in the image processing of the second acquisition process, the diversity of image data input to the first discriminative model 520 is further enhanced. Note that the remaining parameters may be determined in advance.

In either case, the second acquisition process may include adjusting the second partial image PA2 to a discriminative input image acceptable by the first discriminative model 520 . Processor 210 may adjust the size (more generally, pixel density) of second partial image PA2, for example, by adjusting the pixel density or by convolution processing. Further, when the shape of the second partial image PA2 before adjustment is different from the shape of the identification input image, processor 210 extracts a portion having the same shape as the shape of the identification input image from the second partial image PA2 before adjustment. Thus, the adjusted second partial image PA2 may be generated. Thus, the second partial image PA2 may represent all or part of the image of the second partial area A2 in the computer graphics image. When the second acquisition process including scaling is performed, the size of the image of the second partial area A2 in the adjusted second partial image PA2 is increased as the scaling ratio increases. Image PA2 may be adjusted.

(3) The first model 500 may be various adversarial generative networks having a generative network and an identification network instead of the networks described in FIG. 2 and the like. For example, the height and width of the image input to the first generative model 510 may be any predetermined height and width. The total number of convolutional layers of the first encoder 512 of the first generative model 510 may be any number greater than or equal to one. The size of the filters in each convolutional layer may be of various sizes. The first encoder 512 may include fully connected layers. The convolutional layers may be omitted from the first encoder 512 . Similarly, the total number of transposed convolutional layers of the first decoder 517 of the first generative model 510 may be any number greater than or equal to one. The size of the filters in each transposed convolutional layer may be of various sizes. The first decoder 517 may include fully connected layers. From the first decoder 517, the transposed convolution layer may be omitted.

The total number of convolution layers of the convolution layer group 522 of the first discriminant model 520 may be any number equal to or greater than one. The size of the filters in each convolutional layer may be of various sizes. The total number of fully bonded layers in fully bonded layer group 527 may be any number equal to or greater than one. The number of nodes in each fully connected layer can be of various values. One or two of the three modules of the convolutional layer group 522, the global average pooling layer 525, and the fully connected layer group 527 may be omitted.

Batch normalization may be applied to various layers of the first model 500 . Dropouts may be applied to various layers of the first model 500 .

(4) The second generative model 600 is not limited to the networks described with reference to FIGS. 3A, 3B, etc., and may be any network that generates a defect-free image from an input image. For example, the configuration of the second generative model 600 may differ from the configuration of the first generative model 510 . Also, the second generative model 600 may be a generative network generated by training a generative adversarial network having a generative network and a discriminating network. Batch normalization may be applied to various layers of the second generative model 600 . Dropout may be applied to various layers of the second generative model 600 .

(5) The process of calculating the degree of anomaly DA from the difference image data Ix4d (FIG. 7: S540) may be various processes of calculating the degree of anomaly DA indicating the degree of failure. For example, the processor 210 uses t-SNE to generate three-dimensional data from the difference image data Ix4d, and uses the distance between the reference point and the point of interest in the three-dimensional space to calculate the degree of anomaly DA. good. Processor 210 may also use t-SNE to directly compute a single value from difference image data Ix4d. Then, the processor 210 may calculate the degree of abnormality DA by inputting one calculated value into the evaluation function. Also, processor 210 may use various dimensionality reduction processes (eg, principal component analysis) instead of t-SNE.

(6) The inspection process may be any process that generates information (for example, the difference image Ix4, the degree of abnormality DA, the defect character string table Td, etc.) regarding defects in the label sheet, instead of the process shown in FIG. . For example, S540 may be omitted. In this case, the processor 210 may display the difference image Ix4 on the display unit 240 in S545. Further, in the process of generating the difference image data (S535), the binarization process (FIG. 8) of the intermediate difference image data Ix3d may be omitted, and the intermediate difference image data Ix3d may be used as the difference image data. Also, the processing of S550-S560 may be omitted. Alternatively, the processing of S530-S545 may be omitted.

In addition, in S545 and S560, the method of presenting information to the user may be any method capable of transmitting information to the user instead of displaying information. For example, processor 210 may output audio describing information.

(7) The shape of the label sheet may be any shape, such as a triangle, a square, a polygon with 5 or more corners, a circle, or an ellipse. In either case, the image data may represent a rectangular image. At S240 and S255 of FIG. 4, the processor 210 preferably determines the area of the partial image so that the entire extracted partial image is included on the label sheet.

(8) The object to be image-processed is not limited to a label sheet attached to a product such as a printer, but may be a label directly printed on the product. The object is not limited to a label, but may be a tag attached to a product. The object may be a container containing the product or a wrapping paper for wrapping the product. Products related to objects are not limited to printers, and may be arbitrary products such as sewing machines and cutting machines. In general, the object can be any object that represents a character. The number of characters represented by an object may be any integer greater than or equal to one.

(9) The reading device for optically reading an object is not limited to the digital camera 100, but may be any reading device configured to optically read an object using a photoelectric conversion element such as a CCD or CMOS. (e.g. scanner).

(10) The first training process (eg, FIGS. 4 and 5), the second training process (eg, FIG. 6), and the inspection process (eg, FIG. 7) may be performed by different image processing devices. Any image processing device may be a type of device other than a personal computer (eg, digital camera, scanner, smart phone). In addition, a plurality of devices (for example, computers) that can communicate with each other via a network may share the image processing function of the image processing device, and provide the image processing function as a whole ( A system including these devices corresponds to the image processing device).

In each of the above embodiments, part of the configuration implemented by hardware may be replaced with software, or conversely, part or all of the configuration implemented by software may be replaced with hardware. good too. For example, the first generative model 510 of FIG. 1 may be implemented by dedicated hardware circuitry.

In addition, when part or all of the functions of the present invention are realized by a computer program, the program is provided in a form stored in a computer-readable recording medium (for example, a non-temporary recording medium). be able to. The program can be used while being stored in the same or different recording medium (computer-readable recording medium) as when it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but also internal storage devices such as various ROMs in computers, and hard disk drives that are connected to computers. An external storage device may also be included.

Although the present invention has been described above based on examples and modifications, the above-described embodiments of the present invention are intended to facilitate understanding of the present invention, and are not intended to limit the present invention. The present invention may be modified and improved without departing from its spirit, and the present invention includes equivalents thereof.

DESCRIPTION OF SYMBOLS 100...Digital camera 200...Image processing apparatus 210...Processor 215...Storage device 220...Volatile storage device 230...Nonvolatile storage device 231...First program 232...Second program 233...Third program Program 240 Display unit 250 Operation unit 270 Communication interface 500 First model 510 First generation model 512 First encoder 515 First feature data 517 First decoder 519 ... connection, 520 ... first identification model, 522 ... convolution layer group, 525 ... global average pooling layer, 527 ... fully connected layer group, 529 ... identification data, 529f ... first identification data, 529r ... second identification data, 590 First type training data 600 Second generation model 602 Second encoder 607 Second decoder 609 Connection 690 Second type training data

Claims (11)

An image processing method for processing image data, comprising:
A first generation step of generating first generated image data of interest corresponding to the input image data of interest by inputting the input image data of interest representing the optically read object of interest into the first generation model, , the first generation model is such that an image of image data generated by inputting image data representing an optically read image of an object into the first generation model is converted into a computer graphics image of the object. the first generating step, which is a generative model trained to approximate;
A second generation step of generating second generated image data of interest representing an image of an object of interest without defects by inputting the first generated image data of interest to a second generation model, wherein the second generation model is a generative model trained to generate image data representing an image of a defect-free object by inputting image data representing an image of the object into the second generative model; When,
a third generating step of generating difference image data of a difference image indicating a difference between the image of the first generated image data of interest and the image of the second generated image data of interest;
with
wherein the first generative model is the generative network generated by training a generative adversarial network having a generative network and a discriminative network;
The identification network generates a first partial image representing at least part of an image of a first partial area in a first image, which is an image of image data generated by inputting input image data for training to the generation network. and second partial image data representing a second partial image representing at least a part of the image of the second partial area in the second image, which is the computer graphics image for training. are trained by inputting into a discriminative network,
Image processing method. The image processing method according to claim 1, further comprising:
a calculating step of calculating the degree of abnormality of the object of interest represented by the input image data of interest by executing dimension reduction processing of the difference image data;
Image processing method. The image processing method according to claim 1 or 2,
The position of the first partial area on the first image is randomly determined,
The position of the second partial area on the second image is randomly determined,
Image processing method. The image processing method according to any one of claims 1 to 3,
The shape of the first partial area on the first image is randomly determined,
the shape of the second partial area on the second image is randomly determined;
Image processing method. The image processing method according to any one of claims 1 to 4,
The size of the first partial area on the first image is randomly determined,
the size of the second partial region on the second image is randomly determined;
Image processing method. The image processing method according to any one of claims 1 to 5,
the number of the first partial images obtained from the first image is randomly determined;
the number of the second partial images obtained from the second image is randomly determined;
Image processing method. The image processing method according to any one of claims 1 to 6,
the first partial image data is obtained by performing a process that includes rotating an image by a randomly determined angle;
wherein the second partial image data is obtained by performing a process that includes rotating the image by a randomly determined angle;
Image processing method. The image processing method according to any one of claims 1 to 7,
the first partial image data is obtained by performing a process including scaling the image by a randomly determined scaling ratio;
wherein the second partial image data is obtained by performing a process including scaling the image by a randomly determined scaling ratio;
Image processing method. A computer program for a computer that processes image data, comprising:
A first generation function for generating first generated image data of interest corresponding to the input image data of interest by inputting the input image data of interest representing an optically read object of interest into a first generation model, , the first generation model is such that an image of image data generated by inputting image data representing an optically read image of an object into the first generation model is converted into a computer graphics image of the object. the first generative function, which is a generative model trained to approximate;
A second generation function for generating second generated image data of interest representing an image of an object of interest without defects by inputting the first generated image data of interest to a second generation model, the second generation function comprising: is a generative model trained to generate image data representing an image of a defect-free object by inputting image data representing an image of the object into the second generative model; When,
a third generation function for generating difference image data of a difference image indicating a difference between the image of the first generated image data of interest and the image of the second generated image data of interest;
is realized on a computer,
wherein the first generative model is the generative network generated by training a generative adversarial network having a generative network and a discriminative network;
The identification network generates a first partial image representing at least part of an image of a first partial area in a first image, which is an image of image data generated by inputting input image data for training to the generation network. and second partial image data representing a second partial image representing at least a part of the image of the second partial area in the second image, which is the computer graphics image for training. is trained by inputting into a discriminative network,
computer program. An image processing device for processing image data,
A first generating unit that generates first generated image data of interest corresponding to the input image data of interest by inputting the input image data of interest representing an optically read object of interest to a first generation model, , the first generation model is such that an image of image data generated by inputting image data representing an optically read image of an object into the first generation model is converted into a computer graphics image of the object. the first generator, which is a generative model trained to approximate;
A second generation unit for generating second generated image data of interest representing an image of an object of interest without defects by inputting the first generated image data of interest to a second generation model, the second generation unit comprising: is a generative model trained to generate image data representing an image of a defect-free object by inputting image data representing an image of the object into the second generative model; When,
a third generator for generating difference image data of a difference image indicating a difference between the image of the first generated image data of interest and the image of the second generated image data of interest;
with
wherein the first generative model is the generative network generated by training a generative adversarial network having a generative network and a discriminative network;
The identification network generates a first partial image representing at least part of an image of a first partial area in a first image, which is an image of image data generated by inputting input image data for training to the generation network. and second partial image data representing a second partial image representing at least a part of the image of the second partial area in the second image, which is the computer graphics image for training. is trained by inputting into a discriminative network,
Image processing device. A method of training a generative adversarial network having a generative network and a discriminative network, comprising:
generating first image data for a first image by inputting training input image data into a generation network;
randomly determining a first partial region in the first image;
obtaining first partial image data representing a first partial image showing at least part of an image of the first partial area in the first image using the first image data and the first partial area process and
obtaining first identification data by inputting the first partial image data into an identification network;
randomly determining a second partial region in a second image of the second image data associated with the input image data;
Obtaining second partial image data representing a second partial image showing at least part of an image of the second partial area in the second image using the second image data and the second partial area process and
obtaining second identification data by inputting the second partial image data into the identification network;
adjusting parameters of the identification network by using the first identification data and the second identification data;
A training method comprising:

Images