JP7532159B2 - Information processing device, information processing method, and program - Google Patents

Description

The present disclosure relates to an information processing device, an information processing method, and a program.

Annotation tools that assign correct answer information (correct answer labels) to target data to be learned are used as programs for creating training data used in machine learning. There is a demand for higher accuracy of the assigned correct answer information and for a larger amount of data to be used for training data used in machine learning. This is because using highly accurate and large amount of training data for training makes it possible to create a higher performance machine learning model (for example, a machine learning model with more accurate estimation results).

Annotation tools may have a function for efficiently assigning more accurate correct answer information to target data, for example. For example, a function may be provided that uses a machine learning model for a similar task that has been trained in advance using other training data to analyze data to be annotated, and automatically presents candidates for correct answer information to be assigned as annotations based on the results of the analysis. Another example is the function disclosed in Patent Literature 1. This function calculates the reliability of the work related to the annotation from multiple indicators, and suggests redoing the work related to the annotation if the reliability is below a threshold. By using this function to improve the accuracy of the work related to the annotation, it is possible to create training data in which more accurate correct answer information is assigned.

JP 2019-101560 A

However, since the task of annotating large amounts of data tends to require a large amount of labor (human costs), in recent years, outsourcing services for annotation tasks have been provided. In such a situation where outsourcing services are used for annotation tasks, the accuracy of the training data (e.g., the reliability of the correct answer information added) may be guaranteed, for example, by reviewing the validity of the annotation results.

On the other hand, the difficulty of reviewing the validity of annotation results may vary depending on the data being reviewed. In such a situation, for example, a reviewer with low review proficiency may be assigned annotation results with high review difficulty, making it difficult for them to perform a more accurate review, which may result in a decrease in the accuracy of the training data. In light of this, there is a need for a mechanism that can estimate the difficulty of reviewing the validity of annotation results, thereby making it possible to both improve the efficiency of the review work and the accuracy of the training data.

In consideration of the above problems, the present invention aims to make it possible to estimate the difficulty of reviewing the validity of annotation results in a more suitable manner.

The information processing device according to the present invention is characterized by comprising an extraction means for extracting information from annotated data, the information relating to an index of the difficulty of reviewing the validity of the annotation results for the data, and a calculation means for calculating the difficulty based on the information extracted from the data.

According to the present invention, it is possible to estimate the difficulty of reviewing the validity of annotation results in a more suitable manner.

FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. FIG. 2 is a block diagram illustrating an example of a functional configuration of an information processing device. FIG. 11 is a diagram showing an example of a user interface. FIG. 13 is a diagram showing an example of a correct answer information file. FIG. 2 is a diagram showing an example of a structure of a database related to parameter storage. 11 is a flowchart showing an example of processing by an information processing device. 11 is a flowchart showing an example of processing by an information processing device. 11 is a flowchart showing an example of processing by an information processing device. FIG. 11 is a block diagram showing another example of a functional configuration of the information processing device. 10 is a flowchart showing another example of the process of the information processing device. 10 is a flowchart showing another example of the process of the information processing device.

A preferred embodiment of the present disclosure will be described in detail below with reference to the attached drawings. Note that in this specification and drawings, components having substantially the same functional configuration are designated by the same reference numerals to avoid redundant description.

In addition, in this disclosure, an existing trained machine learning model is also referred to as a "pre-trained model." Data that is the subject of annotation work by an operator is also referred to as "annotation target data." Annotation information that is assigned to annotation target data through annotation work is also referred to as "correct answer information." The combination of annotation target data and correct answer information is also referred to as "annotation work result data."

The information processing method according to the present embodiment, which will be described below, can be realized, for example, by an information processing device (computer) described with reference to FIG. 1 executing a predetermined program. The information processing method according to the present embodiment may be realized by a single information processing device, or may be realized by multiple information processing devices working together. When the information processing method according to the present embodiment is realized by multiple information processing devices working together, for example, the functions for realizing the information processing method may be assigned so as to be distributed among multiple information processing devices. As another example, the processing load related to the realization of at least some of the functions for realizing the information processing method according to the present embodiment may be distributed among multiple information processing devices.

In addition, when multiple information processing devices operate in cooperation with each other, the multiple information processing devices are connected to each other via a predetermined network so that they can transmit and receive information to each other. In this case, the type of the network is not particularly limited as long as it is possible to connect the multiple information processing devices to each other. As a specific example, the above network may be the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), a public line (e.g., a telephone line, a mobile communication line, etc.), etc. As another example, the above network may be a dedicated line, an ATM (Asynchronous Transfer Mode), a frame relay line, a cable television line, a wireless communication line for data broadcasting, etc. In addition, the above network may be a wireless network or a wired network. In addition, the above network may include multiple different types of networks. As a specific example, communication between multiple information processing devices may be relayed by another communication device. In this case, different types of networks may be applied to the communications between the other communication device and each of the multiple information processing devices.

<Hardware Configuration>
An example of a hardware configuration of an information processing device 100 according to an embodiment of the present disclosure will be described with reference to Fig. 1. As shown in Fig. 1, the information processing device 100 according to the embodiment includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103. The information processing device 100 also includes an auxiliary storage device 104, an input device interface 105, an output device interface 106, and a network interface 107. The CPU 101, the ROM 102, the RAM 103, the auxiliary storage device 104, the input device interface 105, the output device interface 106, and the network interface 107 are connected to each other via a bus 108.

The CPU 101 is a central processing unit that controls various operations of the information processing device 100. For example, the CPU 101 may control the operation of the entire information processing device 100. The ROM 102 stores control programs and boot programs that can be executed by the CPU 101. The RAM 103 is the main storage memory of the CPU 101, and is used as a work area or a temporary storage area for expanding various programs.

The auxiliary storage device 104 stores various data and programs. The auxiliary storage device 104 is realized by a storage device capable of temporarily or persistently storing various data, such as a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). As another example, the auxiliary storage device 104 may be realized by an optical disk such as a flexible disk (FD) or a compact disk (CD), a magnetic or optical card, an IC card, a memory card, or the like.

The input device interface 105 is an interface for connecting an input device 109 that receives various instructions from a user to the information processing device 100. For example, a pointing device, a keyboard, a touch panel, etc. may be applied as the input device 109. Note that the device applied as the input device 109 may be changed as appropriate depending on the method for recognizing instructions from the user. As a specific example, when instructions from a user are recognized by the voice spoken by the user, a sound collection device for collecting the voice, such as a microphone, may be applied as the input device 109. Also, multiple types of devices may be applied as the input device 109.

The output device interface 106 is an interface for connecting an output device 110, which presents various information to the user, to the information processing device 100. The output device 110 may be realized, for example, by a display device such as a display that presents various information to the user by visually displaying the information. The device applied as the output device 110 may be changed as appropriate depending on the method of presenting information to the user. As a specific example, when information is presented to the user by sound such as voice or electronic sound, an audio output device such as a speaker or headphones capable of outputting the sound may be applied as the output device 110. Furthermore, multiple types of devices may be applied as the output device 110.

The network interface 107 is used for communication with an external device via a network 111 (e.g., the Internet, etc.). The device used as the network interface 107 may be changed as appropriate depending on the type of communication path and the communication method applied.

The CPU 101 deploys a program stored in the ROM 102 or the auxiliary storage device 104 in the RAM 103 and executes this program, thereby realizing the functional configurations shown in Figures 2 and 9 and the processes shown as flowcharts in Figures 6 to 8, 10, and 11.
As a storage medium for supplying the above-mentioned program, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, etc. As other examples of the storage medium, it is also possible to use a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory card, a ROM, a DVD, etc.
Furthermore, the above programs may be executed directly by a computer, or may be executed under the management of underlying software such as an OS (Operating System) that runs on the computer.
Furthermore, the program read from the storage medium may be processed by a function expansion board mounted on the computer, a function expansion unit connected to the computer, or the like.

First Embodiment
As a first embodiment of the present disclosure, an example of an information processing device that estimates a degree of difficulty of reviewing the validity of an annotation result for target data will be described below.

(Functional configuration)
First, an example of the functional configuration of an information processing device according to this embodiment will be described with reference to Fig. 2. In the following, the information processing device according to this embodiment will also be referred to as "information processing device 200" for convenience in order to distinguish it from information processing devices according to other embodiments.

The information processing device 200 includes a data input unit 201, an index extraction unit 202, a parameter storage unit 203, and a review difficulty calculation unit 204.

The data input unit 201 accepts input of a series of data (hereinafter also referred to as "annotation work result data") including data to be annotated (e.g., an image) and correct answer information that has been annotated to the data. For example, a user interface that accepts instructions from a user is used to input the annotation work result data.

3 is a diagram showing an example of a user interface 300 for receiving input of annotation work result data. The user interface 300 includes input areas 301 and 302 and an enter button 303.
The input area 301 is an input area for receiving, from the user, the specification of the path of a file related to the result of adding correct answer information to data to be annotated (hereinafter, also referred to as a "correct answer information file").
The input area 302 is an input area for receiving, from the user, a specification of a path of a file corresponding to data (such as an image) to be annotated.
The enter button 303 is a button for accepting an instruction from the user to confirm the information input in the input areas 301 and 302, respectively.

When the confirm button 303 is pressed based on the operation of the pointer 310 with the file paths input into the input areas 301 and 302, the data input unit 201 acquires the information (paths) input into the input areas 301 and 302. The data input unit 201 then acquires the result of assigning correct answer information to the data to be annotated by reading out the correct answer information file based on the information (path) input into the input area 301. The data input unit 201 also acquires the data by reading out the file of the data to be annotated based on the information (path) input into the input area 302. In this way, the data input unit 201 acquires the annotation work result data specified by the user.

The operation of the pointer 310 can be realized, for example, by a pointing device connected to the information processing device 200 as the input device 109. Of course, as long as it is possible to realize the operation of the pointer 310, the method is not necessarily limited to a method using a pointing device.
In addition, in the example shown in Fig. 3, an example of a case where the paths of the file corresponding to the data to be annotated and the correct answer information file are specified separately has been described, but the method of specification is not limited as long as these files can be read. For example, in an example described later with reference to Fig. 4, the correct answer information file includes information regarding the path of the file corresponding to the data to be annotated. In this case, by accepting the specification of the path of the correct answer information file, the correct answer information file and the file of the data to be annotated specified by the information regarding the path included in the correct answer information file may be read.

An example of a correct answer information file will now be described with reference to Fig. 4. In the correct answer information file 400 shown in Fig. 4, various information related to the result of assigning correct answer information to data to be annotated is recorded in a markup language called XML.
Specifically, the correct answer information file 400 indicates by the annotation element that this file contains information on the annotation result. In the example shown in Fig. 4, the folder element and the filename element specify a file corresponding to the data (e.g., an image) to be annotated. Specifically, in the example shown in Fig. 4, an image is set as the data to be annotated, and the file of the image is specified by the folder element and the filename element.
The size element specifies the size of the data to be annotated (image size). Specifically, the width element specifies the width of the data to be annotated. The height element specifies the height of the data to be annotated.
In addition, the object element specifies various information related to the correct answer information given to the data to be annotated. Specifically, the name element indicates what correct answer information (information indicating what the target indicates) has been given. In addition, the bndbox element indicates in which area of the data to be annotated (e.g., an image) the bounding box to which the annotation (correct answer information) specified by the name element has been given is placed. For example, in the example shown in FIG. 4, the xmin element and the xmax element indicate the area in the x direction (width direction) where the bounding box is placed, and the ymin element and the ymax element indicate the area in the y direction (height direction) where the bounding box is placed.

Of course, the above is merely one example, and as long as the data input unit 201 is capable of acquiring the data to be annotated and the result of assigning correct answer information to the data, the method is not limited to the example described above.
As a specific example, data to be annotated may be acquired as an image file, and the result of adding correct answer information to the data may be acquired as data in a so-called txt format.
As another example, data to be annotated and the result of adding correct answer information to the data may be stored in a so-called database. In this case, the data input unit 201 may obtain data to be annotated that satisfies a predetermined condition (e.g., a condition specified by a user) and the result of adding correct answer information to the data from the database.
The data format of the correct answer information file is not particularly limited. As a specific example, the YOLO format or PASCAL VOC format may be adopted as the data format of the correct answer information file.

The index extraction unit 202 performs a predetermined analysis process on the target data (e.g., data received by the data input unit 201) to extract information about an index of the difficulty of reviewing the validity of the annotation results from the data. An example of the information about the index will be described in detail later.

The parameter storage unit 203 stores information about parameters related to the calculation of the difficulty of reviewing the validity of the annotation result. Note that there is no particular limitation on the method of storing the information about the parameters. As a specific example, a so-called database may be used to store the information about the parameters.

For example, Fig. 5 is a diagram showing an example of a database structure for storing information related to parameters for calculating the review difficulty level regarding the validity of the annotation result. In the example shown in Fig. 5, parameter names and values set for the parameters are stored in association with each other.
As a specific example, the data in the first line indicates that the parameter name is the weight for the number of bounding boxes and that the value is 0.1, the data in the second line indicates that the parameter name is the weight for the number of bounding boxes that overlap with others and that the value is 0.3, and the data in the third line indicates that the parameter name is the weight for the total overlap area of the bounding boxes and that the value is 0.4.

5 is merely an example, and the method of storing the information on the parameters is not particularly limited as long as the information on the parameters can be stored. As a specific example, the information on the various parameters exemplified above may be stored in a setting file in txt format or xml format.
In addition, the parameters listed in Fig. 5 are merely examples, and the parameters applied to the calculation of the difficulty level are not particularly limited as long as they can be used to calculate the difficulty level of the review regarding the validity of the annotation result. In addition, although three parameters are shown in the example shown in Fig. 5, the number of parameters used to calculate the difficulty level of the review is not necessarily limited to three. As a specific example, the number of parameters used to calculate the difficulty level of the review may be more or less than the example shown in Fig. 5.

The review difficulty calculation unit 204 calculates the difficulty of reviewing the validity of the annotation results for the target data based on information related to the indexes extracted from the data by the index extraction unit 202. In addition, at this time, the review difficulty calculation unit 204 may calculate the difficulty of the review by using parameters stored in the parameter storage unit 203. As a specific example, the review difficulty calculation unit 204 may calculate the difficulty of the review by weighting the information related to the indexes extracted by the index extraction unit 202 based on the parameters stored in the parameter storage unit 203.

(process)
Next, an example of processing by the information processing device 200 according to this embodiment will be described with reference to FIGS.

First, an example of the overall processing flow of the information processing device 200 according to this embodiment will be described with reference to FIG.
In S601, the data input unit 201 receives input of annotation work result data, thereby acquiring the annotation work result data.
In S602, the index extraction unit 202 performs a predetermined analysis process on the annotation work result data acquired in S601, thereby extracting information on an index of the difficulty of reviewing the validity of the annotation result from the data. Details of the process of S602 will be described separately later with reference to FIG. 7.
In S603, the degree of difficulty of reviewing the validity of the annotation result for the target data is calculated based on the information on the index extracted in S602. Note that details of the process of S603 will be described separately later with reference to FIG.

Next, an example of the details of the process of S602 in FIG. 6 will be described with reference to FIG.
In S701, the index extraction unit 202 acquires information regarding the number of bounding boxes from the annotation work result data acquired in S601 of Fig. 6. For example, when the annotation work result data is data in the XML format shown in Fig. 4, the index extraction unit 202 may acquire the number of bodbox elements set in the data as information regarding the number of bounding boxes.
Of course, the above is merely an example, and the method is not limited as long as it is possible to obtain information regarding the number of bounding boxes from the annotation work result data.

In S702, the metric extraction unit 202 acquires information regarding the number of bounding boxes that overlap with other bounding boxes from the annotation work result data.
As a specific example, when the annotation work result data is data in the XML format shown in Fig. 4, the index extraction unit 202 may recognize the bounding box set based on the xmin, ymin, xmax, and ymax elements below each bndbox element. In this case, for example, the index extraction unit 202 first sets an x-axis range with xmin as the minimum and xmax as the maximum, and a y-axis range with ymin as the minimum and ymax as the maximum, for some bndbox elements. Then, the index extraction unit 202 may recognize that two bindings overlap when the xmin or xmax of another bndbox element is within the x-axis range and the ymin or ymax is within the y-axis range.
As another example, the index extraction unit 202 may assign, to each pixel of an image that is the data to be annotated, the ID of a bounding box that includes the pixel within its range, and then extract pixels to which multiple bounding box IDs have been assigned. In this case, the index extraction unit 202 may extract the number of combinations of overlapping bounding boxes by extracting combinations of bounding box IDs assigned to each extracted pixel. The number of combinations extracted at this time corresponds to the number of bounding boxes that overlap with other bounding boxes.
Of course, the above is merely an example, and the method is not limited as long as it is possible to obtain information regarding the number of bounding boxes that overlap with other bounding boxes from the annotation work result data.

In S703, the index extraction unit 202 acquires information on the area of the portion where a plurality of bounding boxes overlap (for example, the total area of the portion where a plurality of bounding boxes overlap) from the annotation work result data.
As a specific example, the index extraction unit 202 extracts two values that are neither maximum nor minimum among the xmin and xmax of the two binding boxes recognized as overlapping in S702 as both ends of one side of the target range for which the area is to be calculated. Similarly, the index extraction unit 202 extracts two values that are neither maximum nor minimum among the ymin and ymax of the two binding boxes as both ends of the other side of the target range for which the area is to be calculated. Then, the index extraction unit 202 may calculate the area of the region defined by the extracted sides as the area of the portion where the two binding boxes overlap. In this manner, the index extraction unit 202 calculates the above-mentioned area for each pair of bounding boxes recognized as overlapping. Then, the index extraction unit 202 may acquire a value obtained by adding up the areas calculated for each pair as information on the area of the portion where multiple bounding boxes overlap.
As another example, the index extraction unit 202 may count pixels in a portion where multiple bounding boxes overlap, thereby acquiring information on the area of the portion. Specifically, the index extraction unit 202 first assigns, to each pixel of an image that is data to be annotated, the ID of a bounding box in which the pixel is included. Next, for a series of binding box combinations that were recognized as overlapping in S702, the index extraction unit 202 counts the number of pixels to which the ID of each bounding box included in the combination is assigned. Then, the index extraction unit 202 may calculate the area of the portion where multiple bounding boxes overlap, based on the total value of the number of pixels counted for each combination.
Of course, the above is merely one example, and the method is not limited as long as it is possible to obtain information regarding the area of the portion where multiple bounding boxes overlap from the annotation work result data.

Next, an example of the details of the process of S603 in FIG. 6 will be described with reference to FIG.
In S801, the review difficulty level calculation unit 204 acquires the parameters stored in the parameter storage unit 203.
In S802, the review difficulty calculation unit 204 calculates the difficulty of a review regarding the validity of the annotation result for the target data, based on the information on the indexes extracted in S602 of Fig. 6 and the parameters acquired in S801. As a specific example, the review difficulty calculation unit 204 may multiply each index by the above parameters as a weight, and calculate the difficulty of the review based on the result of adding up each index multiplied by the weight.
Of course, the above is merely an example, and the method is not particularly limited as long as it is possible to calculate the difficulty of the review based on the above indicators and parameters.

Second Embodiment
Next, as a second embodiment of the present disclosure, another example of an information processing device that estimates the difficulty of a review regarding the validity of the annotation result for the target data will be described below. The information processing device according to this embodiment differs from the information processing device 200 according to the first embodiment described above in that a pre-learning model constructed based on prior machine learning is used to estimate the difficulty of the review. Therefore, hereinafter, the information processing device according to this embodiment will be described with a focus on the parts that are different from the information processing device 200 according to the first embodiment described above, and a detailed description of the parts that are substantially similar to the information processing device 200 will be omitted.

(Functional configuration)
First, an example of the functional configuration of an information processing device according to this embodiment will be described with reference to Fig. 9. In the following, the information processing device according to this embodiment will also be referred to as an "information processing device 900" for convenience in order to distinguish it from information processing devices according to other embodiments.

The information processing device 900 differs from the information processing device 200 described with reference to FIG. 2 in that it includes a learning model reading unit 901 and an estimation unit 902. Therefore, hereinafter, the functional configuration of the information processing device 900 will be described, focusing in particular on the parts that differ from the functional configuration of the information processing device 200 described with reference to FIG. 2, and detailed descriptions of parts that are substantially similar to the information processing device 200 will be omitted.

The learning model reading unit 901 reads a pre-learning model that has been constructed in advance based on machine learning related to adding annotations to target data. The read pre-learning model is stored in a predetermined storage area (for example, a storage area such as the RAM 103 or auxiliary storage device 104 provided in the information processing device 900).

The estimation unit 902 uses the loaded pre-learning model to estimate the labels to be assigned as annotations to the annotation work result data (particularly, the data to be annotated) that has been accepted as input.

In addition, in this embodiment, the parameter storage unit 203 stores, as information on parameters, information on parameters related to the weighting of numbers for which the estimation result using the pre-learning model is ambiguous. In addition, the parameter storage unit 203 stores, as information on parameters, information on parameters related to the weighting of numbers for which there is a difference between the annotation result data and the estimation result of the pre-learning model. Of course, the above is merely an example, and the parameter storage unit 203 may hold parameters other than the parameters exemplified above.

(process)
Next, an example of processing of the information processing device 900 according to this embodiment will be described with reference to FIGS.

First, an example of the overall processing flow of the information processing device 900 according to this embodiment will be described with reference to FIG. 10. The example shown in FIG. 10 differs from the example described with reference to FIG. 6 in that it includes processing of S1001 and the content of processing of S602, but the processing of S601 and S603 is substantially similar to the example described with reference to FIG. 6. Therefore, hereinafter, the overall processing of the information processing device 900 will be described, focusing in particular on the parts that differ from the processing of the information processing device 200 described with reference to FIG. 6, and detailed descriptions of the parts that are substantially similar to the information processing device 200 will be omitted.

In S1001, the estimation unit 902 uses the pre-learning model read by the learning model reading unit 901 to estimate a label to be assigned as an annotation to the annotation work result data received as input by the data input unit 201.
As a specific example, the estimation unit 902 may use image data cut out from the range specified by the bndbox element for each object element defined in the correct answer information file 400 (XML file) described with reference to Figure 4 as the subject of the above estimation.
Of course, the above is merely an example, and the target of the estimation of the label to be assigned as the annotation using the pre-learning model is not particularly limited. For example, the estimation of the label to be assigned as the annotation using the pre-learning model may be performed for the entire image data that is the data to be annotated.

Next, referring to FIG. 11, an example of the details of the processing of S602 shown in FIG. 10 will be described. Note that the example shown in FIG. 11 differs from the example described with reference to FIG. 7 in that it includes processing of S1101 and S1102, but the processing of S701 to S703 is substantially similar to the example described with reference to FIG. 7. Therefore, hereinafter, the processing of S602 shown in FIG. 10 will be described, focusing in particular on the parts that differ from the processing of the information processing device 200 described with reference to FIG. 7, and a detailed description of the parts that are substantially similar to the information processing device 200 will be omitted.

In S1101, the index extraction unit 202 acquires information on the number of ambiguous estimation results based on the estimation results in S1001. Specifically, the index extraction unit 202 may acquire information on the number of bounding boxes for which the score related to the likelihood of the estimation result of the label assigned as an annotation based on the pre-learning model is equal to or less than a threshold, out of the series of target bounding boxes.

In S1102, the index extraction unit 202 compares the estimation result in S1001 with the annotation work result data acquired in S601, and acquires information regarding the number of labels (labels assigned as annotations) that differ between the two. Specifically, for each bounding box, the index extraction unit 202 compares the label assigned as an annotation to the bounding box with the estimation result of the label assigned as an annotation for the bounding box. Then, when there is a difference between the assigned label and the estimation result of the label with a higher score (e.g., the label with the highest score) among the estimation results, the index extraction unit 202 determines that there is a difference between the two.

Of course, the above is merely an example, and the method is not particularly limited as long as it is possible to determine whether there is a difference between the estimation result in S1001 and the annotation work result data acquired in S601. As a specific example, prior to the user annotating the target data, the data may be annotated based on the estimation result of the label to be annotated to the data using a pre-learning model. After the user annotates the data, information according to the determination result of whether the annotation assigned based on the estimation result has been corrected may be retained. This makes it possible to determine whether there is a difference between the estimation result of the label to be assigned as an annotation using the pre-learning model and the label assigned as an annotation by the user based on the retained information.

In this way, the index extraction unit 202 obtains information regarding the number of bounding boxes that are determined to be different between the estimated result of the label to be assigned as an annotation using the pre-training model and the label assigned as an annotation by the user.

<Other embodiments>
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a recording medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

100, 200 Information processing device 202 Index extraction unit 203 Parameter storage unit 204 Review difficulty calculation unit

Claims (19)

An extraction means for extracting information relating to an index of difficulty of reviewing the validity of the annotation result for the data from the annotated data;
A calculation means for calculating the difficulty level based on the information extracted from the data;
An information processing device comprising: The calculation means calculates the difficulty level based on the information extracted from the data and a parameter related to the calculation of the difficulty level that is set in advance.
2. The information processing apparatus according to claim 1, The extraction means extracts information relating to a number of bounding boxes from the data;
The calculation means calculates the difficulty level based on information regarding the number of the extracted bounding boxes.
3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer. The extraction means extracts information regarding the number of bounding boxes that overlap with other bounding boxes from the data;
The calculation means calculates the difficulty level based on information regarding the number of the extracted bounding boxes.
4. The information processing device according to claim 1, wherein the information processing device is a computer. The extraction means extracts information regarding an area of an overlapping portion of a plurality of bounding boxes from the data;
The calculation means calculates the degree of difficulty based on the extracted information about the area.
5. The information processing device according to claim 1, wherein the information processing device is a computer. A means for loading a trained machine learning model;
an estimation means for estimating a label to be assigned as an annotation to the data based on the loaded machine learning model;
6. The information processing device according to claim 1, further comprising: The extraction means extracts information on the number of estimation results of the labels that have a score on a likelihood that the labels indicate annotation targets that is equal to or less than a threshold; and
The calculation means calculates the difficulty level based on information regarding the number of estimation results of the label for which the extracted score is equal to or less than a threshold.
7. The information processing apparatus according to claim 6, The extraction means extracts information regarding a difference between a label assigned as an annotation to the data and an estimation result of a label assigned as an annotation to the data based on the machine learning model;
The calculation means calculates the degree of difficulty based on the extracted information on the difference.
8. The information processing apparatus according to claim 6, wherein the information processing apparatus is a computer. a receiving means for receiving an input of the data,
The extraction means extracts information regarding the index from the data whose input has been accepted.
9. The information processing device according to claim 1, The computer
an extraction step of extracting information from the annotated data relating to an index of difficulty of reviewing the validity of the annotation result for the data;
a calculation step of calculating the difficulty level based on the information extracted from the data;
13. An information processing method comprising: The calculation step calculates the difficulty level based on the information extracted from the data and a parameter related to the calculation of the difficulty level that is set in advance.
11. The information processing method according to claim 10. The extracting step includes extracting information about a number of bounding boxes from the data;
The calculation step calculates the difficulty level based on information regarding the number of the extracted bounding boxes.
12. The information processing method according to claim 10 or 11. The extracting step extracts information about the number of bounding boxes that overlap with other bounding boxes from the data;
The calculation step calculates the difficulty level based on information regarding the number of the extracted bounding boxes.
13. The information processing method according to claim 10, wherein the first and second information processing units are connected to each other. The extraction step includes extracting information relating to an area of overlapping portions of a plurality of bounding boxes from the data;
The calculation step calculates the degree of difficulty based on the extracted information about the area.
14. The information processing method according to claim 10, An import step to import a trained machine learning model;
an estimation step of estimating a label to be assigned as an annotation to the data based on the loaded machine learning model;
The information processing method according to any one of claims 10 to 14, further comprising: The extraction step includes extracting information on the number of estimation results of the labels that have a score on a likelihood that the labels indicate annotation targets that is equal to or less than a threshold;
The calculation step calculates the difficulty level based on information regarding the number of estimation results of the label whose extracted score is equal to or less than a threshold.
16. The information processing method according to claim 15, The extraction step extracts information regarding a difference between a label annotated to the data and an estimation result of a label to be annotated to the data based on the machine learning model;
The calculation step calculates the degree of difficulty based on the extracted information about the difference.
17. The information processing method according to claim 15 or 16. A receiving step of receiving an input of the data,
The extraction step extracts information about the index from the data whose input has been accepted.
The information processing method according to any one of claims 10 to 17. A program for causing a computer to function as each of the means of an information processing device according to any one of claims 1 to 9.

Images