JP7085605B2 - Model evaluation system, model evaluation method and model evaluation program - Google Patents

Description

The present invention relates to a model evaluation system, a model evaluation method, and a model evaluation program that support evaluation of a trained model generated by machine learning.

In recent years, a technique of recognizing contents such as images by a computer using a trained model generated by using deep learning has been used. However, since the trained model is a black box judgment, it is difficult to explain the basis of the judgment as to how the computer recognizes the content. For this reason, it may be difficult to operate in situations where the basis for judgment is unknown.

Therefore, a technique for explaining the basis of judgment is also being studied (Non-Patent Documents 1 and 2). The technique described in Non-Patent Document 1 creates "visual explanations" for decisions from large classes of CNN-based models and makes them more transparent. Here, the gradient of any target concept is used to inject into the final convolution layer and generate a coarse localization map that highlights important areas in the image to predict the concept.

Further, in the technique described in Non-Patent Document 2, LIME (Local Interpretable Model-agnostic Explainations) is used to explain the prediction of the classifier in an interpretable manner by locally learning a model that can be interpreted centering on the prediction. Use.

Cornell University, October 7, 2016, Ramprasaath R. et al., "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization", [online], arxiv.org site, [September 22, 2nd year of Reiwa] Search], Internet <https://arxiv.org/pdf/1610.02391.pdf> Cornell University, February 16, 2016, Marco Tulio Ribeiro et al., "" Why Should I Trust You ": Explaining the Predictions of Any Classifier", [online], arxiv.org site, [Reiwa September 22, 2016 Search], Internet <https://arxiv.org/pdf/1602.04938.pdf>

However, since the technique described in Non-Patent Document 1 modifies the inside of deep learning, it cannot be applied to a trained model that has already been trained. Further, in the techniques described in Non-Patent Documents 1 and 2, only the area of interest can be known. In addition, in any of the techniques, since the image is prepared by a person, arbitrariness cannot be excluded.

The model evaluation system that solves the above problems includes an evaluation target storage unit that records a trained model and a control unit that outputs a recognition result using the trained model. Then, the control unit generates a plurality of sample contents, inputs the sample contents into the trained model recorded in the evaluation target storage unit, and acquires the certainty of the recognition result of the sample contents. , The evaluation result regarding the feature content in the trained model is output by using the sample content according to the certainty.

According to the present invention, a trained model generated by machine learning can be evaluated.

Explanatory drawing of the model evaluation system of 1st Embodiment. Explanatory drawing of the hardware configuration of 1st Embodiment. Explanatory drawing of the processing procedure of 1st Embodiment. Explanatory drawing of the black-and-white image of the first embodiment. Explanatory drawing of the processing procedure of 2nd Embodiment. Explanatory drawing of masking of 2nd Embodiment. Explanatory drawing of clustering of 2nd Embodiment.

(First Embodiment)
The first embodiment of the model evaluation system, the model evaluation method, and the model evaluation program will be described with reference to FIGS. 1 to 4. In this embodiment, a trained model that is generated by machine learning using teacher information, inputs predetermined content (image), and outputs a recognition result (text) is evaluated.
As shown in FIG. 1, the model evaluation system of the present embodiment uses a user terminal 10 and a support server 20 connected via a network.

(Hardware configuration example)
FIG. 2 is a hardware configuration example of the information processing apparatus H10 that functions as a user terminal 10, a support server 20, and the like.

The information processing device H10 includes a communication device H11, an input device H12, a display device H13, a storage device H14, and a processor H15. Note that this hardware configuration is an example, and may have other hardware.

The communication device H11 is an interface that establishes a communication path with another device and executes data transmission / reception, such as a network interface or a wireless interface.

The input device H12 is a device that receives input from a user or the like, and is, for example, a mouse, a keyboard, or the like. The display device H13 is a display, a touch panel, or the like that displays various information.

The storage device H14 is a storage device that stores data and various programs for executing various functions of the user terminal 10 and the support server 20. An example of the storage device H14 is a ROM, a RAM, a hard disk, or the like.

The processor H15 controls each process (for example, a process in the control unit 21 described later) in the user terminal 10 and the support server 20 by using the programs and data stored in the storage device H14. Examples of the processor H15 include a CPU, an MPU, and the like. The processor H15 expands a program stored in a ROM or the like into a RAM and executes various processes corresponding to various processes. For example, when the application program of the user terminal 10 and the support server 20 is started, the processor H15 operates a process for executing each process described later.

The processor H15 is not limited to the one that performs software processing for all the processing executed by itself. For example, the processor H15 may include a dedicated hardware circuit (for example, an integrated circuit for a specific application: ASIC) that performs hardware processing for at least a part of the processing executed by the processor H15. That is, the processor H15 is (1) one or more processors that operate according to a computer program (software), (2) one or more dedicated hardware circuits that execute at least a part of various processes, or ( 3) It can be configured as a circuitry including a combination thereof. The processor includes a CPU and a memory such as a RAM and a ROM, and the memory stores a program code or a command configured to cause the CPU to execute a process. Memory or computer readable media includes any available medium accessible by a general purpose or dedicated computer.

(Functions of each information processing device)
The user terminal 10 of FIG. 1 is a computer terminal used by a person in charge of evaluating a trained model.

The support server 20 is a computer system for supporting the evaluation of the trained model. The support server 20 includes a control unit 21, an evaluation target storage unit 22, and a feature information storage unit 23.

The control unit 21 performs a process described later (a process including an image processing step, a prediction step, an evaluation step, a cluster analysis step, and the like). By executing the model evaluation program for this purpose, the control unit 21 functions as an image processing unit 211, a prediction unit 212, an evaluation unit 213, a cluster analysis unit 214, and the like.

The image processing unit 211 executes a process of adjusting an image used for evaluation. The image processing unit 211 holds data regarding an end condition for terminating the feature content generation process. As the end condition, for example, a case where the number of feature images recorded in the feature information storage unit 23 reaches a predetermined number can be used.

The prediction unit 212 executes a process of outputting a prediction result using the trained model.
The evaluation unit 213 executes a process of evaluating the trained model. The evaluation unit 213 holds data on the reference value for comparison with the certainty output by the trained model.

The cluster analysis unit 214 executes a process of grouping feature images rather than a clustering process. For this clustering process, for example, a k-means method using a recognition result and a feature image can be used, but the clustering process is not limited to the k-means method.

The trained model to be evaluated is recorded in the evaluation target storage unit 22. This trained model is recorded when the trained model to be evaluated is acquired from the user terminal 10. In the present embodiment, as the trained model to be evaluated, a character recognition model that recognizes the text included in the image, which is a prediction model (network) generated by deep learning, is used. Here, as the text included in the image, for example, the number "5" can be used.

A feature image management record is recorded in the feature information storage unit 23. This feature image management record is recorded when the feature content generation process is executed. Data related to the feature image and the recognition result are recorded in the feature image management record.

In the feature image data area, data related to the feature image (feature content) in which the certainty of the recognition character is equal to or higher than the reference value is recorded.
In the recognition result data area, data related to characters in which the feature image is recognized with a certainty higher than the reference value is recorded. For example, when "5" is recognized in the image with a certainty of the reference value or more, the number "5" is recorded as the recognition result.

Next, a processing procedure for evaluating the trained model in the system configured as described above will be described.
(Feature content generation processing)
First, the feature content generation process will be described with reference to FIG.

Here, the control unit 21 of the support server 20 executes a black-and-white image generation process (step S101). Specifically, the image processing unit 211 of the control unit 21 acquires the trained model from the user terminal 10 and records it in the evaluation target storage unit 22. Then, the image processing unit 211 generates an arbitrary black-and-white image (sample content). For example, in a bitmap of a predetermined size, a black-and-white image in which white pixels and black pixels are randomly arranged is generated.
For example, as shown in FIG. 4, it is assumed that a black-and-white image 500 is generated.

Next, the control unit 21 of the support server 20 randomly executes the pixel selection process (step S102). Specifically, the image processing unit 211 of the control unit 21 randomly selects pixels in the generated black-and-white image. In this embodiment, one pixel is selected in the bitmap of the black-and-white image generated in step S101.
For example, in FIG. 4, the pixel 501 of the black-and-white image 500 is selected.

Next, the control unit 21 of the support server 20 executes the pixel inversion process (step S103). Specifically, the image processing unit 211 of the control unit 21 inverts the black and white of the selected pixel. As a result, if the selected pixel is a white pixel, it is inverted to a black pixel, and if it is a black pixel, a black-and-white image (sample content) inverted to a white pixel is generated.
Here, in FIG. 4, a black-and-white image 510 is generated by black-and-white inversion of black pixels 501 to white.

Next, the control unit 21 of the support server 20 executes the prediction process (step S104). Specifically, the prediction unit 212 of the control unit 21 inputs the generated black-and-white image 510 into the trained model of the evaluation target storage unit 22. Then, the prediction unit 212 acquires the recognition result and the certainty output by the trained model.

Next, the control unit 21 of the support server 20 executes a determination process as to whether or not the certainty is equal to or higher than the reference value (step S105). Specifically, the prediction unit 212 of the control unit 21 compares the certainty output by the trained model with the reference value.

When it is determined that the certainty is equal to or higher than the reference value (when "YES" in step S105), the control unit 21 of the support server 20 executes the feature image registration process (step S106). Specifically, the image processing unit 211 of the control unit 21 generates a feature image management record associated with the recognition result using the black-and-white image input to the trained model as the feature image, and records it in the feature information storage unit 23.

On the other hand, when it is determined that the certainty is less than the reference value (when "NO" in step S105), the control unit 21 of the support server 20 skips the feature image registration process (step S106).

Next, the control unit 21 of the support server 20 executes a determination process as to whether or not it is terminated (step S107). Specifically, the image processing unit 211 of the control unit 21 counts the number of records of the feature image management record of the same recognition result. Then, when the number of records satisfies the end condition, it is determined to end.

When it is determined that the number of records does not satisfy the end condition and it is not the end (when "NO" in step S107), the control unit 21 of the support server 20 randomly selects pixels (step S102) and thereafter. repeat.
For example, as shown in FIG. 4, in the black-and-white image 510, the black-and-white image 520 is generated by selecting the pixel 502 and inverting the black-and-white image.

On the other hand, when it is determined that the end is completed (when "YES" in step S107), the control unit 21 of the support server 20 executes the feature image acquisition process (step S108). Specifically, the cluster analysis unit 214 of the control unit 21 extracts all the feature image management records from the feature information storage unit 23, and acquires the feature images recorded in the feature image management records.

Next, the control unit 21 of the support server 20 executes a clustering process of the feature image (step S109). Specifically, the cluster analysis unit 214 of the control unit 21 groups the feature images by clustering processing.

Next, the control unit 21 of the support server 20 executes the output processing of the clustering result (step S110). Specifically, the evaluation unit 213 of the control unit 21 outputs the feature image of each group generated by clustering to the user terminal 10.

According to this embodiment, the following effects can be obtained.
(1-1) In the present embodiment, the control unit 21 of the support server 20 randomly executes a pixel selection process (step S102), a pixel inversion process (step S103), and a prediction process (step S104). This makes it possible to evaluate the trained model by calculating the certainty while partially changing the image.

(1-2) In the present embodiment, the control unit 21 of the support server 20 executes a determination process as to whether or not the certainty is equal to or higher than the reference value (step S105). Then, when it is determined that the certainty is equal to or higher than the reference value (when "YES" in step S105), the control unit 21 of the support server 20 executes the feature image registration process (step S106). This makes it possible to search for a characteristic image that outputs the recognition result according to the certainty.

(1-3) In the present embodiment, the control unit 21 of the support server 20 executes a clustering process of the feature image (step S109). As a result, even when a plurality of feature images are acquired for the recognition result, the features grouped by clustering can be output.

(Second Embodiment)
Next, a second embodiment of the model evaluation system, the model evaluation method, and the model evaluation program will be described. In the first embodiment, clustering is performed on the feature images recorded in the feature information storage unit 23. In the second embodiment, the feature area evaluation process is executed in which the feature area (feature area) is specified in the feature image and changed so as to perform clustering. The same parts as those in the first embodiment are designated by the same reference numerals, and detailed description thereof will be omitted.

In this case, the image processing unit 211 of the control unit 21 executes a process of masking a part of the feature image. Then, the image processing unit 211 holds data regarding the necessity determination condition for determining the necessity of the masking process. As the necessity determination condition, for example, a dispersion value that evaluates the similarity of each feature image can be used. In this case, if the variance value is within the necessity reference value, it is determined that the mask processing is unnecessary.

Further, the feature area management record is recorded in the feature information storage unit 23. The feature area management record is recorded when the feature area evaluation process is executed. Data related to the feature area image and the recognition result are recorded in the feature area management record.

In the feature area image data area, data related to an image in a region of the feature image that affects the recognition result is recorded.
In the recognition result data area, data related to the text (here, numbers) recognized by the feature area is recorded.

(Characteristic area evaluation processing)
Next, the feature region evaluation process will be described with reference to FIG.
First, the control unit 21 of the support server 20 executes a feature image acquisition process (step S201). Specifically, the image processing unit 211 of the control unit 21 extracts all the feature image management records from the feature information storage unit 23, and acquires the feature images recorded in the feature image management records.

Next, the control unit 21 of the support server 20 executes a determination process as to whether or not the masking process is necessary (step S202). Specifically, the image processing unit 211 of the control unit 21 acquires the clustering result in the feature content generation processing. Then, the image processing unit 211 compares the feature amounts of the feature images and calculates the variance value of the similarity. Then, the image processing unit 211 compares the dispersion value with the necessity reference value.

When the dispersion value exceeds the necessity reference value and it is determined that masking is necessary (step S202), the control unit 21 of the support server 20 repeats the following processing for each feature image.

Here, first, the control unit 21 of the support server 20 executes the partial masking process (step S203). Specifically, the image processing unit 211 of the control unit 21 generates a mask image (mask content) that masks the feature image by using a black mask having a size of 1/4 of the entire bitmap of the feature image. For example, a mask image in which a black mask is placed at the upper left of the feature image is generated.
As shown in FIG. 6, a mask image 610 in which the black mask M1 is arranged is generated for the feature image 600.

Next, the control unit 21 of the support server 20 executes the calculation process of the certainty (step S204). Specifically, the prediction unit 212 of the control unit 21 inputs the mask image to the trained model recorded in the evaluation target storage unit 22. In this case, the prediction unit 212 outputs the recognition result and the certainty of the mask image.

Next, the control unit 21 of the support server 20 executes a temporary storage process for reducing the certainty (step S205). Specifically, the evaluation unit 213 of the control unit 21 acquires the certainty of the mask image output by the prediction unit 212 with respect to the recognition result of the feature image. Next, the evaluation unit 213 calculates the difference value between the certainty of the feature image and the certainty of the mask image. Then, the evaluation unit 213 temporarily stores the difference value of the certainty in the memory in association with the image of the area (mask area) masked by the black mask in the feature image.

Next, the control unit 21 of the support server 20 executes a determination process as to whether or not masking has been completed (step S206). Specifically, the image processing unit 211 of the control unit 21 determines that the masking is completed when masking is performed for all the arrangements in the feature image. For example, when masking is started from the upper left of the feature image, it is determined that the masking is completed when the black mask reaches the lower right of the feature image.

Here, if it is determined that masking has not been completed (in the case of "NO" in step S206), the control unit 21 of the support server 20 executes the partial masking process (step S203) and subsequent processes. In this case, a mask image is generated by moving the black mask by one pixel (one row or one column).

As shown in FIG. 6, a mask image 620 is generated by moving the black mask M1 with respect to the mask image 610. The control unit 21 of the support server 20 repeats the generation of the mask image 630 sequentially after the processing of steps S203 to 206 is completed for the mask image 620. Then, in the mask image 640 where the black mask reaches the lower right of the feature image 600, it is determined that the masking is completed.

When it is determined that the masking is finished (when "YES" in step S206), the control unit 21 of the support server 20 executes the specification process of the feature area where the decrease in certainty is the largest (step S207). Specifically, the evaluation unit 213 of the control unit 21 specifies the mask area of the maximum value as the feature area among the difference values temporarily stored in the memory. Then, the evaluation unit 213 generates a feature area management record in which the image of the feature area is associated with the recognition result and is recorded, and records the image in the feature information storage unit 23.
Then, the control unit 21 of the support server 20 repeats the above process until all the feature images are completed.

When the iterative processing for all the feature images is completed, the control unit 21 of the support server 20 executes the acquisition process of the feature area (step S208). Specifically, the cluster analysis unit 214 of the control unit 21 extracts all the feature area management records from the feature information storage unit 23, and acquires an image of the feature area recorded in the feature area management record.

Next, the control unit 21 of the support server 20 executes a clustering process of the feature area (step S209). Specifically, the cluster analysis unit 214 of the control unit 21 groups the feature area images by clustering processing. This makes it possible to identify similar feature region images in a common recognition result.
Here, as shown in FIG. 7, the groups G1 to G3 are generated by the clustering process of the feature region.

Next, the control unit 21 of the support server 20 executes the output processing of the clustering result (step S211). Specifically, the cluster analysis unit 214 of the control unit 21 generates an average image of the feature region image for each group generated by clustering. Then, the cluster analysis unit 214 outputs the average image associated with the recognition result to the user terminal 10.

Here, as shown in FIG. 7, average images 701 to 703 of the feature region images belonging to each group G1 to G3 are generated and output to the user terminal 10.
On the other hand, when it is determined that the masking process is unnecessary (when "NO" in step S202), the control unit 21 of the support server 20 uses the feature image recorded in the feature information storage unit 23 as in step S109. , The clustering process of the feature image is executed (step S210).

According to the present embodiment, in addition to the above-mentioned effects (1-1) to (1-3), the following effects can be further obtained.
(2-1) In the present embodiment, the control unit 21 of the support server 20 executes a determination process as to whether or not the masking process is necessary (step S202). Thereby, it is possible to determine the necessity of executing the feature area evaluation process based on the generation status of the feature image.

(2-2) In the present embodiment, the control unit 21 of the support server 20 has a partial masking process (step S203), a certainty calculation process (step S204), and a temporary storage process for reducing the certainty (step S205). To execute. This makes it possible to evaluate the influence on the recognition result of the past region even in the feature image.

(2-3) In the present embodiment, the control unit 21 of the support server 20 executes the process of specifying the characteristic region having the greatest decrease in certainty (step S207). This makes it possible to identify the region of the feature image that most affects the output of the trained model.

(2-4) In the present embodiment, the control unit 21 of the support server 20 executes the clustering process of the feature area (step S209). As a result, even when a plurality of feature area images are acquired for the recognition result, the features grouped by clustering can be output.

This embodiment can be modified and implemented as follows. The present embodiment and the following modified examples can be implemented in combination with each other within a technically consistent range.
-In the first embodiment, the trained model that recognizes the characters included in the image as the content is evaluated. The object of application of the present invention is not limited to image recognition. For example, it may be applied to speech recognition that converts a sound signal into text, or a trained model that recognizes emotions based on sentences.

-In the first embodiment, the control unit 21 of the support server 20 executes a black-and-white image generation process (step S101). The first image is not limited to black and white images. For example, an image whose entire surface is white or black may be used. Further, an image with high certainty of the recognition result may be used as the first image.

-In the first embodiment, the control unit 21 of the support server 20 randomly executes the pixel selection process (step S102). Pixel selection is not limited to random if the sample content can be changed exhaustively. Further, the pixel to be selected is not limited to one, and a plurality of pixels may be selected at the same time.

-In the first embodiment, the control unit 21 of the support server 20 executes the pixel inversion process (step S103). If changes can be made comprehensively, it is not limited to pixel inversion. In the case of a trained model that performs image recognition using a color image, the control unit 21 of the support server 20 sequentially changes, for example, the RGB value of each pixel.

Further, in the case of a trained model that converts an audio signal into text, the control unit 21 of the support server 20 converts the audio signal into a frequency and randomly changes the coefficient of each frequency, for example.
Further, in the case of a trained model that performs natural language processing to acquire some recognition result from a sentence, for example, the word included in the sentence is changed. In this case, in order to generate a plurality of sample contents, another word is acquired from the dictionary storage unit in which the word is recorded in order to replace the word contained in the sentence.

-In the first embodiment, the control unit 21 of the support server 20 executes a determination process as to whether or not it is terminated (step S107). Here, when the number of records satisfies the end condition, it is determined to end. The termination condition is not limited to this. For example, the control unit 21 of the support server 20 may repeat the black-and-white image generation process (step S101) and use the number of repetitions as the end condition. In this case, the control unit 21 of the support server 20 compares the certainty of the sample content generated in advance with the certainty of the subsequent sample content, and when the certainty is lowered, the preceding sample is found. Judge the content as a feature image. Then, the process from the black-and-white image generation process (step S101) is repeated again, and when the number of repetitions reaches a predetermined number, it is determined to end.

-In the second embodiment, the control unit 21 of the support server 20 executes a determination process as to whether masking is necessary (step S202). This determination is not limited to the case where the necessity reference value is used. For example, the necessity of masking may be determined based on the input result of the determination of the person in charge at the user terminal 10. In this case, the image processing unit 211 outputs a confirmation screen for confirming the necessity of mask processing to the user terminal 10.

-In the second embodiment, the control unit 21 of the support server 20 executes a temporary storage process for reducing the certainty (step S205). Here, the feature area 23 in which the decrease in certainty is equal to or higher than the reference value may be recorded in the feature information storage unit 23. In this case, the evaluation unit 213 holds data on the lowering reference value for determining the characteristic region. Then, the evaluation unit 213 compares the difference value between the certainty of the feature image and the certainty of the mask image and the reduction reference value, and records the feature image having the difference value equal to or higher than the reduction reference value.

-In the second embodiment, the control unit 21 of the support server 20 executes the partial masking process (step S203). Specifically, a black mask of a size is used for 1/4 of the entire bitmap of the feature image. The mask size is not limited to this. For example, the mask size may be changed according to the black-and-white dispersion situation in the feature image. In this case, if the index value indicating the dispersion status is equal to or less than the reference value, the mask size is increased.

10 ... user terminal, 20 ... support server, 21 ... control unit, 211 ... image processing unit, 212 ... prediction unit, 213 ... evaluation unit, 214 ... cluster analysis unit, 22 ... evaluation target storage unit, 23 ... feature information storage unit ..

Claims (4)

An evaluation target storage unit that records the trained model,
It is a model evaluation system that evaluates the trained model by providing a control unit that outputs a recognition result using the trained model.
The control unit
Generate multiple sample contents by randomly changing only a part of the content of a predetermined size .
Each of the sample contents is input to the trained model recorded in the evaluation target storage unit, and the certainty of the recognition result of the sample contents is acquired.
A model evaluation system characterized in that the sample content whose certainty is equal to or higher than a reference value is specified as a feature content, and the result of clustering the feature content is output as an evaluation result regarding the feature content. The control unit
A mask content is generated by masking a part of the feature content in the mask area.
The mask content is applied to the trained model recorded in the evaluation target storage unit to calculate the certainty.
The difference value between the certainty of the feature content and the certainty of the mask content is calculated.
The mask area is specified according to the magnitude of the difference value, and the mask area is specified.
The model evaluation system according to claim 1 , wherein the result of clustering the mask area is further output. An evaluation target storage unit that records the trained model,
It is a method of evaluating the trained model by using a model evaluation system that includes a control unit that outputs a recognition result using the trained model and evaluates the trained model.
The control unit
Generate multiple sample contents by randomly changing only a part of the content of a predetermined size .
Each of the sample contents is input to the trained model recorded in the evaluation target storage unit, and the certainty of the recognition result of the sample contents is acquired.
A model evaluation method characterized in that the sample content whose certainty is equal to or higher than a reference value is specified as a feature content, and the result of clustering the feature content is output as an evaluation result regarding the feature content. An evaluation target storage unit that records the trained model,
It is a model evaluation program that evaluates the trained model by using a model evaluation system that has a control unit that outputs a recognition result using the trained model and evaluates the trained model.
The control unit
Generate multiple sample contents by randomly changing only a part of the content of a predetermined size .
Each of the sample contents is input to the trained model recorded in the evaluation target storage unit, and the certainty of the recognition result of the sample contents is acquired.
A model evaluation program for specifying the sample content whose certainty is equal to or higher than the reference value as the feature content and functioning as a means for outputting the result of clustering the feature content as an evaluation result for the feature content.

Images