JP2021111228A - Learning device, learning method, and program - Google Patents

Abstract

To enhance quality for expansion data generated by expanding data, for learning.SOLUTION: A model is trained to output a determination result against inputted data. An edition parameter for a plurality of learning data obtained by editing original data expressing determination targets is determined for each of the original data. Based on the parameter, the plurality of learning data expressing each of the determination subjects is generated, from the original data. Each of the plurality of learning data is used to train the model.SELECTED DRAWING: Figure 1

Description

The present invention relates to learning devices, learning methods, and programs, and particularly to techniques for extending learning data used for machine learning.

A method of training a model such as a neural network using training data is known in order to perform tasks such as data detection or classification or data-based authentication. In general, a large amount of learning data is required to achieve sufficient performance, and therefore, a method for efficiently generating learning data is required.

The method of increasing the learning data in a pseudo manner is called by expressions such as "data expansion", "data augmentation", or "data padding" (hereinafter referred to as "data expansion"). For example, Patent Document 1 proposes a method of performing data expansion by performing vertical-left-right symmetry conversion and image rotation on a learning image used for learning.

Further, in Non-Patent Document 1, in the task of detecting a general object, a learning image having a size of 256 × 256 pixels is prepared, and a partial area having a size of 224 × 224 pixels is cut out from the learning image to expand the data. I'm proposing a way to do it.

Japanese Unexamined Patent Publication No. 2004-361092

K. Chatfield et al. "Return of the Devil in the Details: Delving Deep into Convolutional Nets", arXiv: 1405.331v4 5th Nov 2014 (https://arxiv.org/pdf/1405.3531.pdf)

The quality of the data generated by data expansion (hereinafter referred to as "extended data") affects the quality of the model obtained by training. That is, if the quality of the extended data does not meet the quality required for the learning data, the extended data becomes noise, which is a factor of lowering the quality and efficiency of learning.

In the prior art, the quality of extended data may vary. For example, when data expansion is performed by extracting a partial area from an image, a partial area that does not include an object to be detected or classified may be extracted as extended data, that is, the expanded data may become noise. ..

An object of the present invention is to improve the quality of extended data generated by data expansion for learning.

In order to achieve the object of the present invention, for example, the learning device according to the embodiment of the present invention has the following configurations. That is,
A learning device that uses training data to train a model that outputs judgment results for input data.
A determination means for determining the editing parameters of a plurality of training data obtained by editing the original data representing the determination target for each original data, and
A generation means for generating a plurality of learning data representing each of the determination targets from the original data based on the parameters.
A learning means for learning the model using each of the plurality of learning data, and
To be equipped.

It is possible to improve the quality of extended data generated by data expansion for learning.

The block diagram of the control part 401 of the learning apparatus which concerns on one Embodiment. A flowchart showing an example of a processing flow of a learning method according to an embodiment. The figure which shows the example of the accuracy evaluation result of a model. The block diagram which shows the structural example of the learning apparatus which concerns on one Embodiment. The figure explaining the relationship between a basic image and an extended image. The figure which shows an example of the relationship between the blur intensity and the shift width. The figure explaining the relationship between the original image and the learning image. A block diagram of a basic configuration of a computer used in one embodiment. The figure explaining the cut-off area of an extended image. The figure which shows the example of the extended image corresponding to three different basic images. The figure explaining the method of estimating the distance between center points using a reference value. The figure which shows an example of the user interface for adjusting a shift width. The figure which shows an example of the relationship between the likelihood and the shift width. The figure which shows the example of the reference value attached to the original image. The figure which shows an example of the user interface for removing a training image.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

[Embodiment 1]
The learning device according to the present embodiment determines editing parameters for generating extended data according to the original data used for learning. The extended data generated by such processing is used for learning a model such as a neural network that outputs a determination result for the input data.

In the prior art, data expansion is performed by the same method for all original data. Therefore, when the properties of the original data vary, for example, the position of the subject fluctuates between the original images, it is easy to generate extended data that becomes noise. On the other hand, in the present embodiment, the parameters for data expansion are determined according to the contents of the individual original data. According to such an embodiment, it is possible to reduce the extended data that becomes noise and improve the quality of the extended data.

In the following, a case of learning a model that outputs a judgment result for the data of the original image including the image to be judged will be described as an example. The data obtained from the entire original image is called "original data" and will be used in the following description. This model is a neural network model for face recognition that estimates a person in an image, and can estimate the person's name in the image by face image processing. Even if the person's name is unknown, it may be possible to presume to identify each individual. First, an outline of a processing example described below will be shown.
(1) For learning the face recognition model, a basic image that can be used as a part of the learning data is created by using the original image which is the original data in which the face of a person is captured. The basic image is a partial image cut out from the original image. In addition, for the evaluation of the face recognition model, evaluation data is created using an image showing a person's face.
(2) By shifting the position of the basic image in which the face of a person is captured, an extended image, which is extended data, is generated. The extended image is generated by shifting the position to be cut out from the original image in various ways. The extended image generated in this way is also used for learning.

FIG. 4 shows a configuration example of a monitoring system according to an embodiment of the present invention, which has a function as a learning device capable of performing data expansion and learning of a model. This learning device can train a model that outputs a determination result for the input data. The monitoring system shown in FIG. 4 has a control unit 401 and a learning unit 402.

The control unit 401 generates a plurality of learning data from the original data, as will be described later. For example, the control unit 401 can receive an image from the data holding unit 407, generate a learning image using the received image, and transmit it to the data holding unit 407. Further, the control unit 401 may generate evaluation data for evaluating the determination accuracy of the model. For example, the control unit 401 can generate an evaluation image using the image received from the data holding unit 407 and transmit it to the data holding unit 407.

The learning unit 402 trains a model that outputs a determination result for the input data by using each of the plurality of learning data. For example, the learning unit 402 can receive a learning image from the data holding unit 407 and train the model using the received learning image. In the learning of the model, a set of the learning data and the teacher data showing the correct judgment result for the learning data can be used. In the present embodiment, the determination result for the original data can be predetermined, and the learning unit 402 outputs the same determination result so that the model in which each of the plurality of learning data generated from the original data is input outputs the same determination result. Can train the model. The learning unit 402 can transmit the trained model to the data holding unit 407.

The monitoring system according to the embodiment of the present invention may further include an evaluation unit 403. The evaluation unit 403 can evaluate the determination accuracy of the model after training by using the evaluation data. For example, the evaluation unit 403 can receive the trained model and the evaluation image from the data holding unit 407. Further, the evaluation unit 403 can evaluate the performance of the model after training using the evaluation image, and can transmit the result of the performance evaluation to the data holding unit 407.

The monitoring system according to the embodiment of the present invention may further include a display unit 404, an operation input unit 405, and a data holding unit 407. The display unit 404 can receive and display the learning image from the data holding unit 407. In addition, the display unit 404 can display an interface for setting parameters for generating training data. The operation input unit 405 can set parameters for generating learning data. For example, the user can perform operations such as mouse operation or touch panel operation while looking at the display unit 404, and the operation input unit 405 can set parameters according to the user operation.

As shown in FIG. 4, each of the above-mentioned parts can be connected via a network such as a LAN (Local Area Network) 406.

In this embodiment, each of the processing units shown in FIGS. 1 and 4 may be realized by a computer or by dedicated hardware. In this embodiment, at least part of the processing is performed by a computer. Further, the learning device or the monitoring system according to the embodiment of the present invention may be composed of, for example, a plurality of information processing devices connected via a network.

FIG. 8 is a diagram showing a basic configuration of a computer. In FIG. 8, the processor 810 is, for example, a CPU and controls the operation of the entire computer. The memory 820 is, for example, a RAM, and temporarily stores programs, data, and the like. The computer-readable storage medium 830 is, for example, a hard disk or a CD-ROM, and stores programs, data, and the like for a long period of time. In the present embodiment, the program that realizes the functions of each part stored in the storage medium 830 is read into the memory 820. Then, the processor 810 operates according to the program on the memory 820, so that the functions of each processing unit are realized.

In FIG. 8, the input interface 840 is an interface for acquiring information from an external device. Further, the output interface 850 is an interface for outputting information to an external device. The bus 860 connects the above-mentioned parts and enables data exchange.

FIG. 1 is a block diagram showing a detailed configuration example of the control unit 401. The control unit 401 receives the original image 104 as an input, generates a learning image 106, and outputs the learning image 106. The control unit 401 has a determination unit that determines a parameter indicating a difference between the plurality of training data for each original data representing the determination target. In the example of FIG. 1, the range setting unit 101 and the parameter setting unit 102 can determine this parameter. Further, the control unit 401 has an extension unit that generates a plurality of learning data representing each determination target from the original data based on the parameters. In the example of FIG. 1, the image generation unit 103 and the detection unit 107 can function as extension units for generating learning data.

The detection unit 107 detects the face region from the original image 104. The detection unit 107 can detect 0 or 1 face region from the original image 104. By the way, a large amount of training images are used for learning the model. It is important that the position of the face is accurately determined for the trained image as well, but since the number of trained images is usually enormous, it is difficult for the operator to visually set the face region. Therefore, the detection unit 107 detects the face region from the original image 104. For example, the detection unit 107 can detect a face region by using an image processing method that automatically detects a rectangular region of an object from an image with a likelihood.

In the present embodiment, the detection unit 107 detects the face region by using the method described in Redmon (J. Redmon et al. "YOLOv3: An Incremental Improvement", arXiv: 1804.02767), but other methods may also be used. good. When using the method disclosed in Redmon, the number of rectangles representing the face area output is unknown. Therefore, the detection unit 107 can exclude the original image 104 from the learning target when the number of rectangles output to the original image 104 is 0. Further, when the number of output rectangles is one, the detection unit 107 can use the output rectangles as they are as a face area. On the other hand, when the number of output rectangles is two or more, the detection unit 107 can select one rectangle having the highest likelihood of representing a face and use it as a face region.

FIG. 7A represents a region 702, which is a face region detected from the original image 701 and the original image 104. The image of such a region 702 can be used for training the model. Hereinafter, one image (data) representing a determination target obtained from the original image (original data) by a specific method such as an automatic detection technique is referred to as a basic image (basic data). The basic image is an image of a partial region including an image of a determination target detected from the original image. The learning image (learning data) used for learning the model can include such a basic image (basic data). The learning device according to the present embodiment may acquire a basic image (basic data), and in this case, the basic image (basic data) can be used as the original image (original data). On the other hand, in the present embodiment, in order to increase the learning data, an extended image (extended data) is generated in addition to the basic image (basic data). For example, as shown in FIG. 5, a plurality of extended images 502 corresponding to one basic image 501 can be generated.

The range setting unit 101 determines a parameter indicating a difference between the plurality of learning data for each original data representing the determination target. In the present embodiment, the range setting unit 101 indicates the position shift amount of the image to be determined between the plurality of learning images, and sets the shift width used for generating the extended image. This shift width is used to determine the cropping position of the extended image, as will be described later. Therefore, it can be said that this shift width is a parameter representing the difference in the position of the image representing the determination target between the basic image and the extended image, or between the extended images.

The range setting unit 101 can determine the parameter based on the estimation error of the position of the image to be determined detected from the original image and the position of the partial region from which the basic image is extracted. In the present embodiment, the range setting unit 101 estimates and estimates the distance between the center point of the face and the center point of the face area (hereinafter, referred to as the distance between the center points) in the face region detected by the detection unit 107. The shift width is set according to the distance between the center points. Here, it is considered desirable to improve the accuracy of the model that the amount of cut-off of the face (the size of the part of the face that is not in the image area) in the extended image is as small as possible, while the amount of position shift is as large as possible. Be done. The larger the position shift amount, the higher the possibility that the face is cut off (a part of the face disappears from the image). Therefore, the face cut-off amount and the position shift amount are in a trade-off relationship. When the same shift amount is used, the larger the distance between the center points, the more likely it is that a part of the face will be missing from the expanded image. Therefore, the range setting unit 101 can set the shift width so that the shift amount becomes smaller as the distance between the center points becomes larger.

Further, the range setting unit 101 can determine the parameters based on the image of the determination target included in the original image. For example, the range setting unit 101 can estimate the blur intensity in the face region detected by the detection unit 107, and set the shift width according to the estimated blur intensity. In general, the higher the blur intensity of an image, the higher the difficulty of the face detection process, and the lower the position accuracy of the detected face region tends to be. That is, there is a positive correlation between the blur intensity and the distance between the center points, and it can be considered that the higher the blur intensity, the larger the distance between the center points.

In this embodiment, the range setting unit 101 is described by Masashi Nishiyama et al., "PSF estimation for blur removal in face recognition", IPSJ Research Report Computer Vision and Image Media (CVIM) 2008 (3 (2008-CVIM-161)). )), P. 61-68 is used to estimate the blur intensity, but other methods may be used. The range of blur intensity is between 0 and 100. Then, the range setting unit 101 can set the shift width so that the shift amount becomes smaller as the blur strength becomes larger. For example, the range setting unit 101 can set the shift width so that the magnitude is inversely proportional to the blur strength. In the present embodiment, the range setting unit 101 sets the shift width by using the correspondence relationship between the blur strength and the shift width, which is set in advance as shown in Table 601 shown in FIG. 6 (A).

The parameter setting unit 102 determines the parameters used to generate the extended image. In the case of the present embodiment, the parameter setting unit 102 determines the parameter according to the shift width set by the range setting unit 101 according to the original data (for example, the blur intensity of the face region). For example, the parameter setting unit 102 can determine the cutout position of the extended image from the original image 104. In the case of the present embodiment, the parameter setting unit 102 shifts the face area up, down, left, or right on the original image 104 according to the shift width determined by the range setting unit 101 according to the blur intensity. Determine the cropping position of the extended image.

The parameter setting unit 102 can further determine the number of extended images generated from one original image. In the present embodiment, the number of extended images generated is 4. Further, as shown in FIG. 9, the region 901 for generating the extended image may include a region outside the original image 701. Therefore, the parameter setting unit 102 can further set the method of complementing the pixel values of the cut-off region of the extended image (the portion where there is no data corresponding to the original image 701). For example, in the extended image, the pixel value of the region corresponding to the outside of the original image 701 can be complemented by the median value range of the pixel value of the original image 701. For example, when the pixel value of the original image 701 is a positive integer and its range is in the range of 0 to 255, the pixel value can be set to 127 or 128. As another complementary method, a method using background image data prepared in advance, a method of reducing the shift width so as not to include the outer portion of the original image 701, or the like can be used.

The image generation unit 103 generates data of a plurality of learning images including the image of the determination target from the data of the original image including the image of the determination target based on the parameters. In the present embodiment, the image generation unit 103 generates an extended image from the original image 104 by using the parameters output by the parameter setting unit 102. Further, the image generation unit 103 can generate a basic image from the original image 104 according to the detection result by the detection unit 107. The training image 106 (training data) used for training the model can include such an extended image (extended data) and a basic image (basic data).

Hereinafter, an example of a method for generating an extended image will be described. In the present embodiment, the extended image corresponds to the image of the region after editing in which the position of the partial region from which the basic image is extracted is shifted according to the shift width. More specifically, the image of the region after shifting the position of the partial region from which the basic image is extracted in the vertical direction (vertical direction) and the horizontal direction (horizontal direction) of the original image by the distance according to the shift width is expanded. It is an image. For example, in FIG. 7, the area 702 is a cut-out area for generating a basic image, and is a face area detected by the detection unit 107. Further, the region 703 indicates an example of a cutout region for generating an extended image, and the distance 704 indicates a shift width. In the examples of FIGS. 7A to 7D, four areas for generating an extended image are generated by shifting the area 702 by a distance 704 according to the shift width in the four directions of lower right, lower left, upper left, and upper right. A cutout area is obtained. For example, region 703 is obtained by shifting region 702 in the lower right direction of the image by a distance of 704. Similarly, region 705, region 706, and region 707 each indicate the remaining three cropped regions for generating an extended image.

FIG. 10 shows the relationship between the basic image and the extended image generated from three different original images. The blur intensity of the basic image whose image ID is AAA is estimated to be 10, and the shift width is set to 10 based on Table 601. The basic image having an image ID of BBB has a higher blur intensity than the basic image having an image ID of AAA and is estimated to be 50, so the shift width is smaller and is set to 6. Since the basic image having the image ID of CCC has a higher blur intensity than the basic image having the image ID of BBB and is estimated to be 90, the shift width is further reduced and is set to 2. However, the method of generating the extended image is not limited to the above method, and the extended image can be generated by, for example, editing the basic image (or the original image) such as rotation, enlargement, or reduction processing. .. Similarly, in this case as well, the processing parameters can be changed according to the basic image (or the original image).

The learning unit 402 trains the model using each of the plurality of learning data. For example, the learning unit 402 can receive a plurality of learning images from the data holding unit 407 and learn the face recognition neural network. The learning image used by the learning unit 402 for learning includes an extended image, and may further include a basic image. In this embodiment, the neural network described in Schroff (F. Schroff et al. "FaceNet: A Unified Embedding for Face Recognition and Clustering", arXiv: 1503.03832) is used, and the learning unit 402 uses the method described in Schroff. The neural network is trained by using it, but the model used and the learning method thereof are not limited to this.

The evaluation unit 403 evaluates the determination accuracy of the model after learning by the learning unit 402 using the evaluation data. The evaluation unit 403 can perform accuracy evaluation using evaluation data prepared separately from the original data for learning. For example, the evaluation unit 403 can receive the model obtained by learning by the learning unit 402 and the evaluation image generated by the control unit 401 from the data holding unit 407. Then, the evaluation unit 403 inputs the evaluation image, which is an image of the face region, into the model, and compares the output with the person's name (that is, the correct determination result for the evaluation image) associated with the evaluation image in advance to evaluate the accuracy. It can be performed.

Hereinafter, the processing flow of the learning method according to the present embodiment will be described with reference to the flowchart of FIG.

In step S201, the evaluation unit 403 acquires an evaluation image. In the present embodiment, a digital image obtained by photographing a person's face with a digital camera is used as a material for creating an evaluation image. In addition, each digital image shows one person, and the person's name is known. The evaluation image can be created by the operator visually setting a face area on the digital image in order to accurately determine the position of the face and cutting out the image of the set face area. The face area is a rectangular area provided on the digital image, and one face area is set for each digital image. The created evaluation image can be stored in the data holding unit 407 together with information representing a person's name (that is, a correct determination result for the evaluation image). The evaluation unit 403 can acquire such an evaluation image together with information representing a person's name.

In step S202, the detection unit 107 detects the face region for each of the plurality of original images 104 as described above. The image of this face area is used as the basic image.

In step S203, the range setting unit 101 sets the shift width used when generating the extended image for each of the plurality of original images 104 as described above.

In step S204, the parameter setting unit 102 sets parameters for each of the plurality of original images 104 based on the shift width set in step S203.

In step S205, the image generation unit 103 generates an extended image for each of the plurality of original images 104 based on the parameters set in step S204.

In step S206, the learning unit 402 trains the model using the learning image (the basic image obtained in step S202 and the extended image obtained in step S205).

In step S207, the evaluation unit 403 evaluates the accuracy of the model after the training is performed in step S206.

In step S208, the evaluation unit 403 determines whether or not to continue the learning process based on the result of the accuracy evaluation obtained in step S207. For example, the learning process can be continued when the accuracy is less than the threshold value. When continuing the learning process, the process returns to step S203. If the learning process is not continued, the process of FIG. 2 ends.

According to the above embodiment, parameters for generating an extended image are set for each original image. By using such a parameter, an extended image can be generated by processing according to the characteristics of the original image. That is, the apparatus can generate an extended image suitable for learning the model according to how the determination target is captured in the original image. Therefore, in the present embodiment, it is possible to reduce the extended image that becomes noise, improve the learning efficiency, and generate a highly accurate model, as compared with the extended image generated under uniform conditions.

(Modification example)
In the above embodiment, the shift width is determined according to the original data, for example, according to the blur intensity of the original image. On the other hand, the shift width may be changed so that the accuracy of the model is further improved. In one embodiment, the range setting unit 101 updates parameters such as the shift width according to the determination accuracy of the model after learning. For example, the range setting unit 101 can further determine the shift width based on the accuracy evaluation result generated in step S207. Hereinafter, such an embodiment will be described.

In the following example, a constant fluctuation range (the maximum shift width of 1 to 10 is used in the following example) is set for the shift amount, and a shift width that gives a highly accurate model is searched within this fluctuation range. .. The range setting unit 101 determines the shift width based on each of the maximum shift width and the original image (for example, the blur intensity of the face region). Here, the relationship between the maximum shift width, the blur strength, and the shift width as shown in Table 601 can be predetermined as a table or a calculation formula. That is, the range setting unit 101 can determine the parameters corresponding to the original data representing the determination target according to each of the plurality of criteria. Then, while changing the maximum shift width, the image generation unit 103 generates extended data, the learning unit 402 learns the model, and the evaluation unit 403 evaluates the model. By such a method, the optimum maximum shift width that gives a model with higher accuracy can be determined.

As an example, a method of determining the optimum maximum shift width by searching for the optimum value by the mountain climbing method in the direction in which the maximum shift width decreases, with the largest candidate of the maximum shift width as the initial value, will be described. In the example of FIG. 3B, 10 is set as the initial value of the maximum shift width. In the first step S203, the range setting unit 101 determines the shift width according to each blur intensity of the original image according to Table 601 that defines the reference when the maximum shift width is 10. Further, in the second and subsequent steps S203, the range setting unit 101 determines the shift width based on each of the original images while reducing the maximum shift width. For example, in the fourth step S203, 7 is set as the maximum shift width. At this time, the range setting unit 101 determines the shift width according to the blur intensity of the original image according to Table 602, which defines the reference when the maximum shift width is 7 shown in FIG. 6 (B).

In steps S204 to S207, the same process is performed according to the shift width determined in step S203. For example, in step S206, the learning unit 402 trains the model using each of the plurality of learning images generated by the image generation unit 103 based on parameters such as the updated shift width. In the present embodiment, the evaluation unit 403 can obtain the positive authentication rate when the erroneous authentication rate is 0.0001% as the accuracy evaluation result of the model. Table 302 of FIG. 3B shows an example of the accuracy evaluation result obtained by performing the processing of steps S203 to S208 in three loops. Table 302 shows that the accuracy evaluation results were 94%, 91%, and 90%, respectively, when the maximum shift widths were 8, 9, and 10.

In such an embodiment, in step S208, the evaluation unit 403 can determine whether or not to continue the process based on the relationship between the maximum shift width and the accuracy. If the optimum value is not found, for example, if the accuracy evaluation result obtained in the current loop is better than the accuracy evaluation result obtained in the previous loop, the process returns to step S203. Further, when the optimum value is found, for example, when the accuracy evaluation result obtained in the previous loop is better than the accuracy evaluation result obtained in the current loop, the process of FIG. 2 ends.

For example, Table 301 in FIG. 3A shows an example of the accuracy evaluation result obtained when the evaluation unit 403 evaluates each of the maximum shift widths from 1 to 10. In such a case, the processing of steps S203 to S208 is performed in 6 loops from the case where the maximum shift width is 10 to the case where the maximum shift width is 5. Since the accuracy evaluation result when the maximum shift width is 6 is better than the accuracy evaluation result when the maximum shift width is 5, the optimum value of the maximum shift width is determined to be 6. In this way, the evaluation unit 403 can select one from a plurality of criteria that determine the parameters corresponding to the original data according to the determination accuracy of the model after learning according to each criterion. However, the method for determining the maximum shift width is not limited to this method, and for example, another optimum value search algorithm may be used to determine the optimum value for the maximum shift width.

[Embodiment 2]
In the second embodiment, an extended image is generated according to the use of the model. According to such a configuration, high-quality extended data can be obtained by determining parameters for data expansion according to both the contents of individual samples used for training and the contents of tasks performed using a model. Can be generated. Hereinafter, the difference between the first embodiment and the second embodiment will be described. Unless otherwise specified, the hardware configuration and functional configuration are the same as those in the first embodiment.

In Embodiment 1, the model was used to authenticate a person. Further, the candidate value of the shift width when generating the extended image is 10 kinds of integers from 1 to 10 as shown in Table 601, and is preset according to the original data (for example, blur intensity). rice field.

On the other hand, in the second embodiment, a plurality of uses of the model are defined in advance. For example, the type of use of the model may be selected from "face recognition" and "face detection". The user can set the type of use by the user input via the operation input unit 405. In addition, the range setting unit 101 can further determine the parameters according to the use of the model. For example, the range setting unit 101 can determine the parameters according to both the blur strength and the application of the model. As an example, when the use is "face recognition", the range setting unit 101 can determine the shift width by using Table 601 for the "face recognition" use shown in FIG. 6 (A). Further, when the application is "face detection", the range setting unit 101 can determine the shift width by using Table 603 for the "face detection" application shown in FIG. 6 (C).

[Embodiment 3]
In the first embodiment, the parameter (shift width) is determined according to the blur intensity of the image to be determined included in the original image. On the other hand, the parameters may be determined by using an element different from the blur strength or by combining a plurality of elements. For example, as an alternative to blur intensity, the position, orientation, or both of the determination targets in the original image may be used. Further, as an alternative to the blur intensity, information indicating the imaging environment such as the brightness of the environment may be used. The brightness of the environment can be estimated by a method such as analyzing the pixel value of the background area near the determination target. These factors, as well as the blur intensity, can be used to estimate the distance between center points. For example, the closer the position of the determination target is to the center of the image and the closer the orientation of the determination target is to the front, the lower the difficulty of the determination target detection process, and therefore the distance between the center points is estimated to be small. Further, the brighter the environment, the lower the difficulty of the detection process of the determination target, so it is estimated that the distance between the center points is small.

Further, as an alternative to the blur intensity, the metadata associated with the original image or the basic image may be used. For example, as described above, when a basic image is generated by detecting a judgment target (for example, a face region) from the original image, a detector that outputs the likelihood that the image of the judgment target in the original image represents the judgment target. Can be used. This likelihood can be used as metadata. Such a likelihood, like the blur intensity, can also be used to estimate the distance between center points. That is, the lower the likelihood, the higher the difficulty of the detection process of the determination target, so it is estimated that the distance between the center points is large. For example, if the range of face likelihood is a value between 0 and 100, the magnitude of face likelihood for the trained image generated from the original image may be 90. In such an example, the range setting unit 101 can set the shift width corresponding to the likelihood based on the table shown in FIG. In the example of FIG. 13, when the likelihood is 90, the shift width 10 is used.

The type of metadata is not limited to likelihood. For example, information indicating the imaging conditions when the original image is captured may be used as metadata. Examples of imaging conditions include camera position / orientation data, camera gain value at the time of shooting, and the like. For example, if the camera position / orientation data indicates that the judgment object was taken from near the front, the distance between the center points is presumed to be small, so the shift width is increased, and in other cases, the center. Since it is estimated that the distance between points is large, the shift width can be reduced. If the camera gain value at the time of shooting is small, the distance between the center points is estimated to be small, so the shift width is increased. In other cases, the distance between the center points is estimated to be large, so the shift width is increased. Can be made smaller.

[Embodiment 4]
In the first embodiment, parameters for generating extended data are determined for each original image by using elements such as blur intensity for estimating the distance between center points. However, the present invention is not limited to such a form. For example, when a part of the original image group is accompanied by a reference value indicating a judgment target area, the original image group is divided into several subsets, and extended data is used for each subset using the reference value. The parameters to generate can be determined. Hereinafter, such a method will be described focusing on the difference between the first embodiment and the fourth embodiment. Unless otherwise specified, the hardware configuration and functional configuration are the same as those in the first embodiment.

First, the reference value will be described with reference to FIG. The image 1401 is the original image, the area 1402 is the area of the image (face) to be determined detected by the apparatus by the method using the detection unit 107, and the area 1403 is the face area indicated by the reference value. The reference value represents the position of the image to be determined detected by a method having a higher accuracy than the method of detecting the region 1402. The reference value may be data generated by human visual inspection, or may be the output of a detector that takes a long time to process but has very high accuracy. In the following, for the sake of simplification of the description, it is assumed that the region 1402 and the region 1403 have the same shape. Further, the distance 1404 and the distance 1405 indicate the positional error between the region 1402 and the region 1403 in the vertical direction and the horizontal direction on the image, respectively. Such a positional error can be used to estimate the distance between center points for images having similar characteristics.

The procedure for estimating the distance between the center points in the present embodiment will be described with reference to FIG. First, it is divided into a plurality of original image groups (subsets) based on the characteristics of the plurality of original image groups and images. For example, the original image group may include an image captured by the camera C1 and an image captured by the camera C2. The two cameras C1 and C2 are fixedly installed in different environments. In this case, the original image group can be divided into a subset A composed of images taken by the camera C1 and a subset B composed of images taken by the camera C2. Here, the original image is divided into two subsets A and B for the sake of simplification of the description, but the original image group may be divided into three or more subsets. Further, the division method is not limited to this method, and for example, the original image group may be divided into an image group captured in the daytime and an image group captured in the nighttime.

Subsequently, reference values are given to a part of the images of the subset A and a part of the images of the subset B. Further, each of the subsets A and B is separated into an image group with a reference value and an image group without a reference value. Next, for the reference valued image group for the subset A, the positional error (that is, the distance 1404, 1405) between the face region and the region specified by the reference value is statistically analyzed. For example, a histogram of the vertical error (distance 1404) on the image can be created, and the median value thereof can be used as the vertical shift width for the reference value-less pixel group of the subset A. Similarly, the median of the lateral error (distance 1405) on the image can be the lateral shift width for the reference-less pixel group of the subset A. Similarly, for the subset B, the shift widths in the vertical direction and the horizontal direction can be obtained.

By such a method, based on the error between the position of the image of the judgment target detected by the device and the position of the image of the judgment target indicated by the reference value for the image group with the reference value, the same subset Parameters for image groups without reference values can be determined. According to the above method, it is possible to create a table in which different shift widths are set for each of the subsets. In this case, the image generation unit 103 can generate an extended image from the reference value-less pixel group of each subset according to the set shift width in the same manner as in the first embodiment. As in Table 601, different shift widths may be set for each subset of images having a predetermined range of blur intensity.

[Embodiment 5]
The learning device according to the above embodiment further has a configuration for visually confirming the generated extended image, a configuration for interactively adjusting various parameters, and a configuration for removing a part of the learning image. May include. Hereinafter, such a configuration will be described. Unless otherwise specified, the hardware configuration and functional configuration are the same as those in the first embodiment.

(Configuration for visually confirming the generated extended image)
The learning device may have a function of displaying the extended image in order to visually confirm that the intended extended image is generated. For example, when the extended image is created based on the blur intensity of the image as in the first embodiment, the display unit 404 can perform the display as shown in FIG.

(Configuration for interactive adjustment of various parameters)
In order to interactively adjust the parameters used for generating the extended image while visually confirming the result of image generation, the learning device may have a function for manually adjusting the parameters. For example, after the extended image is obtained according to the first embodiment, the user may adjust the shift width for each extended image. In this case, the display unit 404 can display the user interface shown in FIG. This user interface may include elements for displaying parameters such as shift width. The user interface may also include an element for receiving user input for adjusting parameters and an element for displaying a learning image generated according to the adjusted parameters. For example, in the user interface shown in FIG. 12, when the user manually changes the shift width, an extended image according to the changed shift width is displayed.

In FIG. 12, the shift width is set to 6 according to the first embodiment, and the extended image 1604 generated according to the shift width and the corresponding basic image are displayed. When changing the shift width, the user inputs a new shift width in the box 1601 and presses the generation start button 1602. Here, the user has entered 3 as the new shift width, and the new extended image 1605 generated according to this shift width is displayed. The user visually observes the displayed extended image 1604 and extended image 1605, and presses the button 1606 when he / she wants to change the shift width, and presses the button 1607 when he / she does not want to change the shift width. If the user wants to consider a shift width of a different size, he / she can enter the new shift width in the box 1601 again and repeat the above procedure.

(Structure for removing the learning image)
The user may remove a specific image such as an image determined to be unsuitable for learning when the learning image (basic image and extended image) is visually confirmed. In this case, the display unit 404 can display the user interface shown in FIG. This user interface contains elements for receiving user input to delete the training image. For example, the user interface shown in FIG. 15 has a check box for selecting each image of the basic image and the extended image. By checking the check box, the user can remove the checked image from the image used for learning.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

101: Range setting unit, 102: Parameter setting unit, 103: Image generation unit, 107: Detection unit, 401: Control unit, 402: Learning unit, 403: Evaluation unit

Claims (20)

A learning device that uses training data to train a model that outputs judgment results for input data.
A determination means for determining the editing parameters of a plurality of training data obtained by editing the original data representing the determination target for each original data, and
A generation means for generating a plurality of learning data representing each of the determination targets from the original data based on the parameters.
A learning means for learning the model using each of the plurality of learning data, and
A learning device characterized by being provided with. An evaluation means for evaluating the determination accuracy of the model after learning is further provided by using the evaluation data.
The determination means updates the parameters according to the determination accuracy of the model after learning.
The learning according to claim 1, wherein the learning means trains the model using each of the plurality of learning data generated by the generation means based on the updated parameters. Device. The determination means determines the parameter corresponding to the original data representing the determination target according to each of the plurality of criteria.
The learning apparatus according to claim 2, wherein the evaluation means selects one from the plurality of criteria according to the determination accuracy of the model after learning according to the respective criteria. The model outputs the judgment result for the data of the original image including the image to be judged, and outputs the judgment result.
The generation means is characterized in that, based on the parameter, data of a plurality of learning images including the image of the determination target is generated from the data of the original image including the image of the determination target. The learning device according to any one of 3 to 3. The learning device according to claim 4, wherein the parameter is a position shift amount of the image to be determined between the plurality of learning images. The generation means, as at least a part of the data of the plurality of training images,
Data of a basic image which is an image of a partial region including the image of the determination target detected from the original image and
The data of the extended image which is the image of the region after shifting the position of the partial region according to the parameter, and
The learning device according to claim 4 or 5, wherein the learning apparatus is generated. The learning apparatus according to claim 6, wherein the extended image includes an image of the region after the position of the partial region is shifted in the vertical direction and the horizontal direction of the image by a distance according to the parameter. The determination means is
6. The parameter is determined based on an estimation error between the position of the image of the determination target detected from the original image and the position of the partial region including the image of the determination target. Or the learning device according to 7. The learning device according to any one of claims 4 to 8, wherein the determination means determines the parameters based on the image of the determination target included in the original image. 4. The determination means is characterized in that the parameter is determined based on the blur intensity, position, or orientation of the image of the determination target, or the likelihood that the image of the determination target represents the determination target. 9. The learning device according to any one of 9. The learning device according to any one of claims 4 to 10, wherein the determination means determines the parameters based on information indicating the imaging conditions or the imaging environment of the original image. The determination means is
The plurality of original images are classified into a plurality of image groups according to the characteristics of the images, and the plurality of original images are classified into a plurality of image groups.
The position of the image of the determination target detected by the method by the apparatus and the position of the image of the determination target detected by a method with higher accuracy than the method for a part of the images included in one image group. The learning apparatus according to any one of claims 4 to 7, wherein the parameters for the remaining images included in the one image group are determined based on the error of. The learning device according to any one of claims 4 to 12, further comprising a display means for displaying the learning image generated according to the parameters. The display means displays a user interface including an element for receiving a user input for adjusting the parameter and an element for displaying the learning image generated according to the adjusted parameter. , The learning device according to claim 13. The learning device according to claim 13, wherein the display means displays a user interface including an element for receiving a user input for deleting the learning image. The judgment result for the original data is predetermined.
The learning means according to claims 1 to 15, wherein the learning means trains the model so that the model to which each of the plurality of learning data is input outputs the predetermined determination result. The learning device according to any one item. The learning device according to any one of claims 1 to 16, wherein the determination means further determines the parameters according to the use of the model. The learning device according to any one of claims 1 to 17, wherein the model is a neural network. It is a learning method that uses training data to train a model that outputs judgment results for input data.
A process of determining editing parameters of a plurality of learning data obtained by editing the original data representing the determination target for each original data, and
A step of generating a plurality of learning data representing each of the determination targets from the original data based on the parameters, and
A process of learning the model using each of the plurality of training data, and
A learning method characterized by including. A program for causing a computer to function as each means of the learning device according to any one of claims 1 to 18.

Images