JP2019036087A - Generation device, method for generation, generation program, learning data, and model - Google Patents

Abstract

A generation device, a generation method, a generation program, learning data, and a model capable of appropriately estimating a ratio of each class of target information to be classified. A generation apparatus includes an acquisition unit and a generation unit in a control unit. The acquisition unit 131 acquires target information to be classified and ratio information indicating a ratio classified into each class of target information by each of a plurality of users. Based on the target information and the ratio information acquired by the acquisition unit 131, the generation unit 132 calculates a model that estimates the ratio at which one target information is classified into each class when the one target information is input. Generate. [Selection] Figure 4

Description

  The present invention relates to a generation device, a generation method, a generation program, learning data, and a model.

  In recent years, crowdsourcing has been known in which work (tasks) is requested to an unspecified number of people (users) using the Internet. For example, in such crowdsourcing, a technique for efficiently utilizing the orderer's human resources is provided.

JP 2014-153756 A

  However, with the above-described conventional technology, it may be difficult to appropriately estimate the ratio of classification into each class of target information. For example, in crowdsourcing, simply requesting a user to perform a task of classifying target information such as an image into a desired class, when target information that requires new classification occurs, etc. It is difficult to properly estimate the proportion of classes. For example, if the consensus (agreement) of the answer results obtained from the user who requested the task is estimated, if the consensus cannot be obtained, it cannot be used as data. For example, when using information only when consensus cannot be obtained, it is difficult to estimate a useful amount such as a tendency of an answer from target information.

  The present application has been made in view of the above, and provides a generation device, a generation method, a generation program, learning data, and a model that can appropriately estimate the ratio classified into each class of target information. Objective.

  The generation device according to the present application is acquired by the acquisition unit that acquires target information to be classified, and ratio information indicating a ratio classified into each class of the target information by each of a plurality of users, and the acquisition unit. A generation unit that generates a model for estimating a ratio of the one target information classified into each class when one target information is input based on the target information and the ratio information; It is characterized by having.

  According to one aspect of the embodiment, there is an effect that it is possible to appropriately estimate the ratio classified into each class of the target information.

FIG. 1 is a diagram illustrating an example of a generation process according to the embodiment. FIG. 2 is a diagram illustrating an example of an estimation process according to the embodiment. FIG. 3 is a diagram illustrating a configuration example of a generation system according to the embodiment. FIG. 4 is a diagram illustrating a configuration example of the generation apparatus according to the embodiment. FIG. 5 is a diagram illustrating an example of a learning data storage unit according to the embodiment. FIG. 6 is a diagram illustrating an example of the model information storage unit according to the embodiment. FIG. 7 is a diagram illustrating an example of a user information storage unit according to the embodiment. FIG. 8 is a diagram illustrating an example of the estimated information storage unit according to the embodiment. FIG. 9 is a flowchart illustrating an example of the generation process according to the embodiment. FIG. 10 is a flowchart illustrating an example of the estimation process according to the embodiment. FIG. 11 is a hardware configuration diagram illustrating an example of a computer that realizes the function of the generation device.

  In the following, a generation apparatus, a generation method, a generation program, learning data, and a form for implementing a model (hereinafter referred to as “embodiment”) according to the present application will be described in detail with reference to the drawings. Note that the generation device, generation method, generation program, learning data, and model according to the present application are not limited by this embodiment. In the following embodiments, the same portions are denoted by the same reference numerals, and redundant description is omitted.

(Embodiment)
[1. Generation process)
First, an example of the generation process according to the embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of a generation process according to the embodiment. FIG. 1 illustrates a case where the generation apparatus 100 generates a model based on target information to be classified and correct answer information indicating a ratio at which the target information is classified into each class. Hereinafter, the target information associated with the correct answer information is also referred to as “learning data”. In the example of FIGS. 1 and 2, a case where the target information is image information (hereinafter also simply referred to as “image”) will be described later as an example. The target information is not limited to images, but may be various information such as character information or article content combining images and character information.

  Here, a model (learning device) generated by the generation device 100 in FIG. 1 will be briefly described. The model generated by the generation apparatus 100 is, for example, a model in which a plurality of nodes that output calculation results for input data are connected in multiple layers, and a model in which image features abstracted by supervised learning are learned It is. For example, the model is a neural network in which layers having a plurality of nodes are connected in multiple stages, and may be a DNN (Deep Neural Network) realized by a so-called deep learning technique. Image features include not only the specific features that appear in the image, such as the presence / absence of characters in the image, color, composition, etc., but also what the object is being imaged and what kind of user the image is It is a concept that also includes the characteristics of an abstracted (meta-) image such as the atmosphere of the image.

  For example, the model is generated by the following learning method using the deep learning technique. For example, in the model, the connection coefficient between each node is initialized, and images having various characteristics are input. Then, the model is generated by a process such as back propagation (error back propagation method) that corrects a parameter (connection coefficient) so that an error between the output of the model and the input image is reduced. For example, the model is generated by performing processing such as back propagation so as to minimize a predetermined loss function such as an error function. By repeating the processing as described above, the model can output an output that can better reproduce the input image, that is, a feature of the input image.

  Note that the model learning technique is not limited to the technique described above, and any known technique can be applied. In addition, any method can be applied to the image input method for the model, the format of the data output by the model, the content of features that are explicitly learned for the model, and the like. That is, the generation apparatus 100 can use an arbitrary model as long as it can calculate a feature amount indicating a feature abstracted from an image.

  In FIG. 1, the generation device 100 generates models M1 to M3 and the like based on a so-called convolutional neural network (CNN) that repeats convolution and pooling of a local region of an input image. Hereinafter, the convolutional neural network may be referred to as CNN. For example, the models M1 to M3 by CNN have output invariance with respect to positional variations such as characters and imaging objects included in the image in addition to the function of extracting and outputting features from the image. Therefore, the models M1 to M3 and the like can accurately calculate the abstract features of the image. As described above, when “model M * (* is an arbitrary numerical value)” is described, this indicates that the model is a model identified by the model ID “M *”. For example, when “model M1” is described, the model is a model identified by the model ID “M1”.

[Configuration of generation system]
First, the generation system 1 shown in FIG. 3 will be described prior to the description of FIG. As illustrated in FIG. 3, the generation system 1 includes a terminal device 10 and a generation device 100. The terminal device 10 and the generation device 100 are connected via a predetermined network N so as to be communicable by wire or wirelessly. FIG. 3 is a diagram illustrating a configuration example of a generation system according to the embodiment. Note that the generation system 1 illustrated in FIG. 3 may include a plurality of terminal devices 10 and a plurality of generation devices 100.

  The terminal device 10 is an information processing device used by a user. For example, the user is a plurality of workers who perform tasks such as crowdsourcing. The terminal device 10 is realized by, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. In the example shown in FIG. 1, the case where the terminal device 10 is a smartphone used by the user is shown. Hereinafter, the terminal device 10 may be referred to as a user. That is, hereinafter, the user can be read as the terminal device 10. Specifically, FIG. 1 illustrates a case where the terminal device 10 is a smartphone used by a user identified by the user ID “U1” (hereinafter, may be referred to as “user U1”).

  In the example illustrated in FIG. 1, the terminal device 10 will be described as terminal devices 10-1 to 10-5 in accordance with a user who uses the terminal device 10. For example, the terminal device 10-1 is the terminal device 10 used by the user U1. For example, the terminal device 10-2 is the terminal device 10 used by the user U2. Hereinafter, the terminal devices 10-1 to 10-5 are referred to as the terminal device 10 when they are not particularly distinguished. Note that, as described above, “user U * (* is an arbitrary numerical value)” indicates that the user is a user identified by the user ID “U *”. For example, when “user U1” is described, the user is a user identified by the user ID “U1”.

  The generation apparatus 100 is an information processing apparatus that generates a model based on target information to be classified and ratio information indicating a ratio classified into each class of elephant information by each of a plurality of users. For example, when one piece of target information is input, the generation apparatus 100 generates a model that estimates the ratio at which the one piece of target information is classified into each class. Moreover, the generation apparatus 100 estimates the ratio of classification into each class with respect to new target information by inputting the target information into the model.

  First, in the example illustrated in FIG. 1, the generation device 100 collects information for generating correct answer information. Specifically, the generating apparatus 100 classifies the image group IML1 such as the images IM101 to IM103 into users (workers) by crowdsourcing, and acquires the answer. In the example of FIG. 1, the generation apparatus 100 is assumed to cause the user to perform a task of classifying an image as either a dog or a cat.

  For example, the generation device 100 provides the target information to the terminal device 10-1 used by the user U1 who is a worker (Step S11-1). In the example of FIG. 1, the generation device 100 provides an image IM101 that is target information to the terminal device 10-1. Then, the generation apparatus 100 acquires an answer to the image IM101 from the user U1 (Step S12-1). In the example of FIG. 1, the generation apparatus 100 acquires a reply from the user U1 that the image IM101 is “cat”.

  Further, for example, the generation device 100 provides the target information to the terminal device 10-2 used by the user U2 who is a worker (Step S11-2). In the example of FIG. 1, the generation device 100 provides an image IM101 that is target information to the terminal device 10-2. Then, the generation device 100 acquires an answer to the image IM101 from the user U2 (Step S12-2). In the example of FIG. 1, the generation apparatus 100 acquires a reply from the user U2 that the image IM101 is “cat”.

  Further, for example, the generation device 100 provides the target information to the terminal device 10-3 used by the user U3 who is a worker (Step S11-3). In the example of FIG. 1, the generation device 100 provides an image IM101 that is target information to the terminal device 10-3. Then, the generation apparatus 100 acquires a response to the image IM101 from the user U3 (Step S12-3). In the example of FIG. 1, the generation apparatus 100 acquires a reply from the user U3 that the image IM101 is “dog”.

  Further, for example, the generation device 100 provides the target information to the terminal device 10-4 used by the user U4 who is a worker (Step S11-4). In the example of FIG. 1, the generation device 100 provides the terminal device 10-4 with an image IM101 that is target information. Then, the generation device 100 acquires an answer to the image IM101 from the user U4 (Step S12-4). In the example of FIG. 1, the generation apparatus 100 obtains an answer from the user U4 that the image IM101 is “dog”.

  Further, for example, the generation device 100 provides the target information to the terminal device 10-5 used by the user U5 who is a worker (Step S11-5). In the example of FIG. 1, the generation device 100 provides an image IM101 that is target information to the terminal device 10-5. Then, the generation device 100 acquires an answer to the image IM101 from the user U5 (Step S12-5). In the example of FIG. 1, the generation apparatus 100 acquires a reply from the user U5 that the image IM101 is “cat”.

  Hereinafter, when it demonstrates without distinguishing step S11-1-S11-5, it will generically call step S11. In addition to steps S11-1 to S11-5, provision of target information to each user may be performed a plurality of times for each image IM102, 103, etc. of the image group IMF1. In the following description, steps S12-1 to S12-5 will be collectively referred to as step S12 when not described. In FIG. 1, five users U1 to U5 are illustrated. However, the generation apparatus 100 is not limited to the users U1 to U5, and the target information for a large number of users (for example, 1 million users, 10 million users, etc.) Get an answer. For example, the generation device 100 may provide the image group IMF1 to the terminal device 10 in step S11, and acquire a user's answer to each of the images IM101 to IM103 and the like included in the image group IMF1 in step S12.

  And the production | generation apparatus 100 produces | generates the ratio information which shows the ratio classified into each class of object information based on the information acquired by step S12. Further, the generation device 100 adds a combination of the target information and the total information including the ratio information as correct answer information as learning data (step S13). Specifically, the generation apparatus 100 adds data DT101 to DT103 and the like corresponding to each of the images IM101 to IM103 that are target information to the learning data storage unit 121.

  As described above, when “data DT * (* is an arbitrary numerical value)” is described, it indicates that the data is data identified by the data ID “DT *”. For example, when “data DT1” is described, the data is data identified by the data ID “DT1”.

  “Data ID” shown in the learning data storage unit 121 in FIG. 1 indicates identification information for identifying data. “Target information” shown in the learning data storage unit 121 in FIG. 1 indicates target information included in the data identified by the data ID. The “aggregation information” shown in the learning data storage unit 121 in FIG. 1 indicates information (aggregation information) obtained by aggregating answers to tasks performed by workers by crowdsourcing.

  “Correct information (ratio information)” in “aggregated information” shown in the learning data storage unit 121 in FIG. 1 indicates correct information (ratio information) corresponding to data identified by the data ID. For example, “correct answer information (ratio information)” indicates a ratio of answers of each class in all answers of workers.

  The “number of users” in the “aggregated information” indicates the number of users who answered about the corresponding target information. “Cat (CL1)” in the “aggregated information” indicates the number of users who answered as cats for the corresponding target information. In the example of FIG. 1, “cat (CL1)” in “aggregated information” also includes information for identifying a user who has made a reply with a cat regarding the corresponding target information. Further, “dog (CL2)” in the “aggregated information” indicates the number of users who have made a reply with the dog regarding the corresponding target information. In the example of FIG. 1, “dog (CL2)” in “aggregated information” also includes information for identifying a user who made a reply with a dog regarding the corresponding target information.

  For example, in the example illustrated in FIG. 1, the data identified by the data ID “DT101” (data DT101) indicates that the target information to be classified is the image IM101. In addition, in the data DT101, the ratio of correct information classified as “cat” is “0.58 (58%)”, and the ratio classified as “dog” is “0.42 (42%)”. It shows that. Data DT101 indicates that the number of users is 1000. The data DT101 indicates that the number of users who answered “cat” is 580, and the user includes the user U1, the user U2, and the like. The data DT101 indicates that the number of users who answered “dog” is 420, and the user includes the user U3 and the user U4. That is, the data DT101 indicates that, for the image IM101, out of 1000 users (workers), 580 responded as cats and 420 responded as dogs. Therefore, in the data DT101, the ratio of the cat in the correct answer information is “580/1000 = 0.58”, and the ratio of the dog in the correct answer information is “420/1000 = 0.42”.

  The data identified by the data ID “DT102” (data DT102) indicates that, for the image IM102, 12000 out of 2000 users (workers) responded as cats and 300 responded as dogs. . Therefore, in the data DT102, the ratio of the cat in the correct answer information is “1700/2000 = 0.85”, and the ratio of the dog in the correct answer information is “300/2000 = 0.15”.

  Further, the data (data DT103) identified by the data ID “DT103” indicates that, for the image IM103, among 10000 users (workers), 5500 responded as cats and 4500 responded as dogs. . Therefore, in the data DT103, the ratio of the cat in the correct answer information is “5500/10000 = 0.55”, and the ratio of the dog in the correct answer information is “4500/10000 = 0.45”.

  As described above, the image group IMF1 used as learning data includes the images IM101 and IM103 that have a small difference in worker classification results between classes, and the image IM102 that has a large difference in worker classification results between classes. . For example, the image group IMF1 in FIG. 1 includes images such as the image IM101 and the image IM103 in which the difference in the ratio of each class is less than a predetermined threshold (for example, 0.1 or 0.2). Further, for example, the image group IMF1 in FIG. 1 includes images such as the image IM102 in which the difference in the ratio of each class is equal to or greater than a predetermined threshold (for example, 0.1 or 0.2). Therefore, the generation device 100 learns (generates) a model using the ratio information as correct answer information. This point will be described in detail below.

  The generation device 100 generates a model based on learning data including the images IM101 to IM103 as described above (step S14). For example, the generating apparatus 100 generates a model by performing learning using the data DT101 to DT103 in the learning data storage unit 121 as learning data (teacher data).

  In the example illustrated in FIG. 1, the generation apparatus 100 generates a model using ratio information indicating a ratio classified into each class such as the images IM101 to IM103 and the images IM101 to IM103. Here, the ratio information (hereinafter referred to as “correct answer information RDT101”) indicating the ratio of the generation apparatus 100 classified into each class of the image IM101 and the image IM101 will be described as an example.

  First, the image IM101 is input to the model M1. As a result, the model M1 outputs a score corresponding to each class. In the example of FIG. 1, the model M1 outputs a score corresponding to a cat (class CL1) and a score corresponding to a dog (class CL2).

  As described above, for example, the generation device 100 learns and generates the model M1 using a deep learning technique. For example, the generating apparatus 100 uses, as learning data, a combination of the image IM101 and correct answer information including a cat (class CL1) ratio “0.58” and a dog (class CL2) ratio “0.42”. For example, the correct answer information RDT101 includes information indicating the ratio “0.58” of the cat (class CL1) and information indicating the ratio “0.42” of the dog (class CL2). For example, the generation device 100 corrects the parameter (connection coefficient) so that the error between the output (score of each class) in the model M1 and the ratio (value) of each class included in the learning data is reduced. The model M1 is learned by performing processing such as (error back propagation method). For example, the generation apparatus 100 generates the model M1 by performing processing such as back propagation so as to minimize a predetermined error (loss) function.

  For example, the generation apparatus 100 uses an error function L as shown in the following equation (1). As shown in the following formula (1), for example, in the case of an N-class classification problem, the generation device 100 uses cross entropy as an error function. The error function L may be any function as long as it represents the certainty of the identification result. For example, the error function L may be entropy obtained from the identification probability. For example, the error function L may be any function as long as it indicates the accuracy of recognition of the model M1.

  Here, “x” in the above formula (1) and the following formulas (3) to (6) indicates an image. For example, in the example shown in FIG. 1, “x” in the above formula (1) and the following formulas (3) to (6) corresponds to the image IM. Also, 1 to N assigned to the variable “n” correspond to each class identified (classified) by the model M1. For example, the model M1 corresponding to the above equation (1) indicates that N classes are identified. For example, “Cat (Class CL1)” and “Dog (Class CL2)” correspond to each class.

Further, “t _n (x)” in the above formula (1) and the following formulas (4) and (5) indicates the ratio of the class n (any one of 1 to N) into which the image IM101 is classified. For example, “t _n (x)” in the above formula (1) indicates a ratio corresponding to the class n in the correct answer information RDT101. In this case, for example, when the target corresponding to class 1 is “cat”, “t ₁ (x)” is “0.58 (58%)”. For example, “t _n (x)” indicating the ratio of class n (any one of 1 to N) has a relationship as shown in the following formula (2).

Here, as shown in the above formula (2), the total value of “t _n (x)” indicating the ratio of the class n (any one of 1 to N) into which the image is classified is “1”. For example, the total value of “t ₁ (x)” and “t ₂ (x)” indicating the ratio of two classes into which the image IM101 is classified is “1”. For example, in the image IM101, “t ₁ (x)” when the object corresponding to class 1 is “cat” is “0.58 (58%)”, and the object corresponding to class 2 is “dog”. In this case, “t ₂ (x)” is “0.42 (42%)”. In this case, “t ₁ (x) + t ₂ (x)” becomes “0.58 + 0.42”, that is, “1”. In this way, each of “t _n (x)” indicating the ratio of class n (any one of 1 to N) is a value such that the total of classes 1 to N as a whole is “1”. For example, “t _n (x)” indicating the ratio of each class n (any one of 1 to N) is a value such that the sum is “1”.

Further, “p _n (x)” in the above formula (1) and the following formulas (3) and (4) is based on the output of the model M1 for the class n (any one of 1 to N) in the image IM101. Indicates the percentage. For example, “p _n (x)” in the above equation (1) indicates a ratio corresponding to the class n output by the model M1. For example, when the object corresponding to class 1 is “cat”, “p ₁ (x)” is “0.55 (55%)” or “0.57 (57%)” according to the learning of the model M1. ", Etc."

Further, “p _n (x)” in the above formula (1) is defined by a Softmax function as shown in the following formula (3) with the probability of class n with respect to x.

The function “f _n ” in the above equation (3) is a score of class n output by CNN (model M1). “Θ” is a parameter of CNN (model M1). The function “exp” is an exponential function. In this case, the gradient of the error function L (1) shown in the above equation (1) is calculated by the following equation (4).

As shown in the above equation (4), in all classes 1 to N, when p _n (x) = t _n (x), the gradient of the error function L (x) becomes 0 and becomes an extreme value. . For example, the generation device 100 performs feedback processing so that the gradient of the error function L (x) becomes zero. For example, when the generation apparatus 100 repeats the above-described processing, the model M1 can appropriately output a score indicating the ratio at which the input image is classified into each class.

  In the above example, the image IM101 is taken as an example, and the case where processing is performed for each target information using the above formula (1) is shown. However, the generation apparatus 100 is configured as shown in the following formula (5), An error function L corresponding to all images may be used.

  For example, 1 to M assigned to the variable “x” in the equation (5) corresponds to each of a plurality of images such as the images IM101 to IM103 included in the image group IMF1. For example, the generation apparatus 100 performs a feedback process so that the gradient of the error function L in the above equation (5) becomes 0, so that the model M1 indicates the ratio at which the input image is classified into each class. A score can be output appropriately.

  Further, when classifying the handling information into two classes, the following formula (6) may be used instead of the above formula (5).

For example, in the example of FIG. 1, “t _A (x)” in the above formula (6) indicates the ratio at which the image IM101 is classified as a cat (class CL1). In this case, for example, “t _A (x)” corresponding to the ratio classified as a cat (class CL1) in the correct answer information RDT101 is “0.58 (58%)”. Further, for example, “(1-t _A (x))” becomes “0.42 (= 1−0.58) (42%)”, and the ratio of being classified as a dog (class CL2) in the correct answer information RDT101 Corresponding to

Further, “p _A (x)” in the above formula (6) indicates a ratio based on the output of the model M1 for the cat (class CL1) in the image IM101.

  Note that the model learning technique is not limited to the technique described above, and any known technique can be applied. Each model may be generated using various conventional techniques relating to machine learning as appropriate. For example, the model generation may be performed using a technique related to machine learning of supervised learning such as SVM (Support Vector Machine). Further, for example, the model generation may be performed using a technique related to machine learning of unsupervised learning. For example, the generation of the model may be performed using a deep learning technique. For example, the model generation may be performed by appropriately using various deep learning techniques such as RNN (Recurrent Neural Network) and CNN. The description relating to the generation of the model is merely an example, and the generation of the model may be performed by a learning method appropriately selected according to information that can be acquired. That is, the generation apparatus 100 generates the model M1 by any method as long as the model M1 can be learned so as to output the score corresponding to the correct answer information when the target information included in the learning data is input. May be performed.

  In the example of FIG. 1, the generation apparatus 100 generates a model (model M1) identified by the model ID “M1” as shown in the model information storage unit 122 by the process as described above. In addition, as shown in the model information storage unit 122 in FIG. 1, the model M1 is used for estimation of whether the image is classified into one of two classes of dogs and cats. This is a model used in the above, and indicates that the specific model data is “model data MDT1”. For example, the generation apparatus 100 inputs image information to the model M1, and causes the model M1 to output a score indicating the ratio of the input image information classified into each class. Based on the score output by the model M1 Estimate the proportion of images classified into each class.

  As described above, the generation device 100 learns using a learning data in which image information and correct answer information are associated with each other, so that a model that can appropriately estimate a ratio classified into each class of target information. Can be generated. Therefore, by using the model generated as described above, for example, the generation apparatus 100 can accurately estimate the ratio classified into each class of an image (dog cat classification).

  For example, in crowdsourcing, if the user simply asks the user to perform a task of classifying target information such as images into a desired class and estimates the consensus of the obtained answer results, Can not be used as. The consensus referred to here may be, for example, a match (agreement) between the user's answers due to bias in the answers. For example, the consensus may be that a certain number of users have made a certain answer. For example, the consensus may be that a certain answer (for example, “cat”) is given by a predetermined percentage (70%) or more of users. For example, when using information only when consensus cannot be obtained, it is difficult to estimate a useful amount such as a tendency of an answer from target information.

  For example, in conventional learning, images such as the image IM101 and the image IM103 in the example of FIG. 1 that have no difference in the proportion of each class according to worker classification are not used for model generation. For example, in conventional learning, learning is performed using only an image such as the image IM102 in which the difference in the ratio of each class is significant. For example, in the conventional learning, for the image IM102, the label of the cat (class CL1) is set to “1” and the label of the dog (class CL2) is set to “0”, and the images IM101 and IM103 are used for learning. There wasn't. However, even when there is no difference in the proportion of each class, it is important as learning data. However, when the number of classes increases, there is a problem that data that is not used increases.

  Therefore, the generation apparatus 100 can use data that has not been used in the past and classify the image itself by using a target (correct answer information) to learn the ratio of the number of workers who have selected each class. It is possible to appropriately estimate even a difficult image. Further, for example, even if the threshold value is used as a normal classifier, the generalization performance of the learning device can be improved by increasing the number of learning data.

[2. (Estimation process)
An example of the estimation process according to the embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of an estimation process according to the embodiment. In FIG. 2, when new target information is acquired, the generation apparatus 100 estimates a rate of classification into each class with respect to the target information, and provides information based on the estimation.

  First, the generation device 100 acquires an image IM10 that is target information (step S21). In the example of FIG. 2, the generation device 100 acquires the image IM10 to be classified from the terminal device 10 used by the user U101.

  The generation apparatus 100 that has acquired the image IM10 inputs the image IM10 into the model. For example, the generation apparatus 100 inputs data of the image IM10 (hereinafter referred to as “data IM10”) to the model M1.

  In the example of FIG. 2, the generation apparatus 100 calculates a score indicating the ratio at which the image IM10 is classified into each class by the processing as shown in the processing group PS21. The generation device 100 inputs the data IM10 to the model M1 (Step S22). The model M1 to which the data IM10 is input outputs a score (step S23). The model M1 outputs a score corresponding to each class. In the example of FIG. 2, the model M1 to which the data IM10 is input outputs the score “0.55” of the cat (class CL1) and the score “0.45” of the dog (class CL2) as shown in the score SC11. Is output. Accordingly, the generation apparatus 100 uses the model M1 and the ratio (probability) that the image IM10 is classified as a cat (class CL1) is 55%, and the ratio (probability) that is classified as a dog (class CL2). Estimated to be 45%.

  And the production | generation apparatus 100 produces | generates an estimation result (step S24). In the example of FIG. 2, the generation apparatus 100 generates the estimation result information ER21 that the ratio that the image IM10 that is the estimation target is classified as a cat is 55% and the ratio that is classified as a dog is 45%. .

  Thereafter, the generation device 100 provides information based on the estimation result (step S25). In the example of FIG. 2, the generation device 100 provides the generated estimation result information ER21 to the terminal device 10.

  As described above, the generation device 100 estimates the ratio of classification into each class for the target information by using the model. In the example of FIG. 2, the generation apparatus 100 inputs the image IM10 to the model M1, and causes the model M1 to output a score indicating the ratio at which the image IM10 is classified into each class. Then, the generation device 100 estimates that the higher the score output by the model M1, the higher the possibility that the target information is classified into that class. In the example of FIG. 2, the generation apparatus 100 estimates that an area with a higher score of a cat (class CL1) output from the model M1 is more likely to be classified as a cat. Then, the generation device 100 provides the estimated result to the terminal device 10. As a result, the user who receives the information provided from the generation apparatus 100 can grasp which class the target information is classified into. In addition, the user who has received the information on the ratio information from the generation apparatus 100 can grasp how difficult the target information is to be classified.

[2-1. (Target of estimation)
In the example of FIG. 1 and FIG. 2, a case is shown in which a model for estimating whether an image is classified into a cat (class CL1) or a dog (class CL2) is generated, or estimation is performed using the model. The generation apparatus 100 may generate a model for performing any classification. For example, the generation apparatus 100 may generate a plurality of models by associating various pieces of correct answer information with each target information. For example, the generating apparatus 100 may generate a plurality of models by associating various pieces of correct answer information with each target information.

  Further, the generation apparatus 100 may generate a model that targets not only images but also various pieces of target information such as character information and article content that combines images and character information.

[2-2. About answer)
In the example of FIG. 1, in order to simplify the explanation, a case where each user (worker) selects either a dog or a cat and answers either one is shown. You may get answers by percentage of each class. For example, the generation apparatus 100 determines from the user U1 the ratio “0.6 (60%)” that the image IM101 is considered to be a cat (class CL1) and the ratio “0.4 (40 that is considered to be a dog (class CL2)). %) ”Or the like may be obtained. For example, the generation apparatus 100 determines from the user U2 the ratio “0.7 (70%)” that the image IM101 is considered to be a cat (class CL1) and the ratio “0.3 (30) that is considered to be a dog (class CL2). %) ”Or the like may be obtained.

  Further, the generation apparatus 100 may generate correct answer information based on answers from the ratios of the classes from the users. For example, when generating the correct answer information based on the answers of the user U1 and the user U2, the generating apparatus 100 sets the ratio of the cat (class CL1) of the image IM101 to “0.65 (= (0.6 + 0.7) / 2) (65%) ". For example, when generating the correct answer information based on the answers of the user U1 and the user U2, the generating apparatus 100 sets the ratio of the dog (class CL2) of the image IM101 to “0.35 (= (0.4 + 0.3)”. / 2) (35%) ". Note that the above is an example, and the generation apparatus 100 may generate correct answer information by acquiring various answers from each user.

[2-3. (Weighting according to responding users (workers))
Further, the generation apparatus 100 may weight the answers according to the users who answered. For example, the generation apparatus 100 may weight each user's answer based on information indicating the evaluation and reliability of each user in crowdsourcing, such as the career and skill level of each user.

  For example, the generation apparatus 100 may set the weight to be greater than “1” for an answer from a user whose career history is a predetermined number of years or more. For example, the generation apparatus 100 may be worth answering two users by setting the weight to “2” for the answers of users who have a career history of a predetermined number of years or more. Further, for example, the generation apparatus 100 may set the weight to be greater than “1” for a user's answer whose skill level as a worker is equal to or higher than a predetermined threshold. For example, the generation apparatus 100 may set the weight of “1.5” for the answer of a user whose career as a worker is a predetermined number of years or more, and may be worth the answer of 1.5 people. Note that the above is an example, and the generation apparatus 100 may weight each user's answer using various information.

[3. Configuration of the generator
Next, the configuration of the generation apparatus 100 according to the embodiment will be described with reference to FIG. FIG. 4 is a diagram illustrating a configuration example of the generation apparatus according to the embodiment. As illustrated in FIG. 4, the generation apparatus 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The generation device 100 includes an input unit (for example, a keyboard and a mouse) that receives various operations from an administrator of the generation device 100 and a display unit (for example, a liquid crystal display) for displaying various types of information. May be.

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card). The communication unit 110 is connected to the network by wire or wireless, and transmits / receives information to / from the terminal device 10.

(Storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. As illustrated in FIG. 4, the storage unit 120 according to the embodiment includes a learning data storage unit 121, a model information storage unit 122, a user information storage unit 123, and an estimated information storage unit 124.

(Learning data storage unit 121)
The learning data storage unit 121 according to the embodiment stores various types of information related to learning data. FIG. 5 is a diagram illustrating an example of a learning data storage unit according to the embodiment. For example, the learning data storage unit 121 stores teacher data used for generating a model. The learning data storage unit 121 illustrated in FIG. 5 includes items such as “data ID”, “target information”, and “total information”. “Total information” includes items such as “correct answer information (ratio information)”, “number of users”, “cat (CL1)”, and “dog (CL2)”.

  “Data ID” indicates identification information for identifying data. For example, the data identified by the data ID “DT101” corresponds to the data DT101 shown in the example of FIG. “Target information” indicates target information included in the data identified by the data ID. For example, “target information” indicates target information to be classified. “Aggregated information” indicates information obtained by aggregating answers to tasks performed by a worker by crowdsourcing.

  “Correct answer information (ratio information)” in “aggregated information” indicates correct answer information (ratio information) corresponding to data identified by the data ID. For example, “correct answer information (ratio information)” indicates a ratio of answers of each class in all answers of workers.

  The “number of users” in the “aggregated information” indicates the number of users who answered about the corresponding target information. “Cat (CL1)” in the “aggregated information” indicates the number of users who answered as cats for the corresponding target information. In the example of FIG. 5, “cat (CL1)” in “aggregated information” also includes information for identifying a user who made a reply with a cat regarding the corresponding target information. Further, “dog (CL2)” in the “aggregated information” indicates the number of users who have made a reply with the dog regarding the corresponding target information. In the example of FIG. 5, “dog (CL2)” in the “aggregated information” includes information for identifying a user who has made a reply with the dog regarding the corresponding target information.

  For example, in the example illustrated in FIG. 5, the data (data DT101) identified by the data ID “DT101” indicates that the target information to be classified is the image IM101. In addition, in the data DT101, the ratio of correct information classified as “cat” is “0.58 (58%)”, and the ratio classified as “dog” is “0.42 (42%)”. It shows that. Data DT101 indicates that the number of users is 1000. The data DT101 indicates that the number of users who answered “cat” is 580, and the user includes the user U1, the user U2, and the like. The data DT101 indicates that the number of users who answered “dog” is 420, and the user includes the user U3 and the user U4. That is, the data DT101 indicates that, for the image IM101, out of 1000 users (workers), 580 responded as cats and 420 responded as dogs. Therefore, in the data DT101, the ratio of the cat in the correct answer information is “580/1000 = 0.58”, and the ratio of the dog in the correct answer information is “420/1000 = 0.42”.

  Note that the learning data storage unit 121 is not limited to the above, and may store various information according to the purpose. For example, the learning data storage unit 121 may store information related to the date and time when learning data was added. Further, for example, the learning data storage unit 121 may store information indicating what kind of determination processing each learning data has been added. For example, the learning data storage unit 121 may store information indicating whether each learning data has been determined by an administrator's selection.

(Model information storage unit 122)
The model information storage unit 122 according to the embodiment stores information about the model. For example, the model information storage unit 122 stores model information (model data) generated by the generation process. FIG. 6 is a diagram illustrating an example of the model information storage unit according to the embodiment. The model information storage unit 122 illustrated in FIG. 6 includes items such as “model ID”, “use”, and “model data”. In FIG. 6, only the models M1 to M3 are illustrated, but a large number of model information such as M4 and M5 may be stored according to each application (estimation target).

  “Model ID” indicates identification information for identifying a model. For example, the model identified by the model ID “M1” corresponds to the model M1 illustrated in the example of FIG. “Use” indicates the use of the corresponding model. “Model data” indicates data of a corresponding model associated with the model data. For example, “model data” includes information including nodes in each layer, functions adopted by the nodes, connection relationships between the nodes, and connection coefficients set for connections between the nodes.

  For example, in the example shown in FIG. 6, the model (model M1) identified by the model ID “M1” has an application of “image (dog cat classification)”, and the input image has two classes of dogs and cats. Indicates that it is used to estimate whether it is classified as either. The model data of the model M1 is model data MDT1.

  For example, in the example illustrated in FIG. 6, the model (model M3) identified by the model ID “M3” has a usage of “character information (category classification)” and the input character information includes a plurality of categories (classes). Among them, it is used for estimating which class is classified. For example, the category (class) to be classified may include three or more categories such as politics, sports, and entertainment. The model data of the model M3 is model data MDT3.

  For example, the model M1 (model data MDT1) belongs to an input layer to which target information to be classified is input, an output layer, and any layer from the input layer to the output layer and other than the output layer. A first element, and a second element whose value is calculated based on the weight of the first element and the first element, and each of the target information input to the input layer belongs to each layer other than the output layer By using the element as the first element and performing an operation based on the first element and the weight of the first element, the score value used for estimating the ratio classified into each class of the target information is output from the output layer It is a model for functioning a computer.

Here, it is assumed that the models M1 to M3 and the like are realized by a regression model represented by “y = a ₁ * x ₁ + a ₂ * x ₂ +... + A _i * x _i ”. In this case, for example, the first element included in the model M1 corresponds to input data (xi) such as x1 or x2. The weight of the first element corresponds to the coefficient ai corresponding to xi. Here, the regression model can be regarded as a simple perceptron having an input layer and an output layer. When each model is regarded as a simple perceptron, the first element can correspond to any node of the input layer, and the second element can be regarded as a node of the output layer.

  Further, it is assumed that the models M1 to M3 and the like are realized by a neural network having one or more intermediate layers such as DNN. In this case, for example, the first element included in the model M1 corresponds to any node of the input layer or the intermediate layer. The second element corresponds to the next node, which is a node to which a value is transmitted from the node corresponding to the first element. The weight of the first element corresponds to a connection coefficient that is a weight considered for a value transmitted from a node corresponding to the first element to a node corresponding to the second element.

  The model information storage unit 122 is not limited to the above, and may store various model information according to the purpose. For example, the model information storage unit 122 may store a model used for estimating the rate at which article content combining an image and character information is classified into each class.

(User information storage unit 123)
The user information storage unit 123 according to the embodiment stores various types of information regarding the user. For example, the user information storage unit 123 stores information on a plurality of workers (users) who perform tasks such as crowdsourcing. FIG. 7 is a diagram illustrating an example of a user information storage unit according to the embodiment. The user information storage unit 123 illustrated in FIG. 7 includes items such as “user ID”, “age”, “sex”, “home”, “work location”, and “interest”.

  “User ID” indicates identification information for identifying a user. For example, the user identified by the user ID “U1” corresponds to the user U1 illustrated in the example of FIG. “Age” indicates the age of the user identified by the user ID. The “age” may be a specific age of the user identified by the user ID, such as 35 years old. “Gender” indicates the gender of the user identified by the user ID.

  “Home” indicates location information of the user's home identified by the user ID. In the example illustrated in FIG. 7, “home” illustrates an abstract code such as “LC11”, but may be information indicating latitude and longitude. For example, “home” may be a region name or an address.

  “Work location” indicates position information of the user's work location identified by the user ID. In the example illustrated in FIG. 7, “work location” illustrates an abstract code such as “LC12”, but may be information indicating latitude and longitude. Further, for example, the “work location” may be an area name or an address.

  “Interest” indicates the interest of the user identified by the user ID. That is, “interest” indicates an object that is highly interested by the user identified by the user ID. In the example illustrated in FIG. 7, one “interest” is illustrated for each user, but may be plural.

  For example, in the example illustrated in FIG. 7, the age of the user identified by the user ID “U1” is “20s”, and the gender is “male”. For example, the user identified by the user ID “U1” indicates that the home is “LC11”. For example, the user identified by the user ID “U1” indicates that the work location is “LC12”. For example, the user identified by the user ID “U1” indicates that he / she is interested in “sports”.

  Note that the user information storage unit 123 is not limited to the above, and may store various types of information according to the purpose. For example, the user information storage unit 123 may store information regarding the demographic attribute of the user and information regarding the psychographic attribute. For example, the user information storage unit 123 may store information such as name, family structure, income, interest, and lifestyle. Further, for example, the user information storage unit 123 may store information indicating the evaluation and reliability of each user in crowdsourcing such as the career and skill level of each user.

(Estimated information storage unit 124)
The estimation information storage unit 124 according to the embodiment stores various types of information related to estimation such as an estimation target and an estimation result. FIG. 8 shows an example of the estimated information storage unit 124 according to the embodiment. The estimation information storage unit 124 illustrated in FIG. 8 includes items such as “estimation target”, “class”, and “score”. The estimated information storage unit 124 illustrated in FIG. 8 indicates information related to the class classification (dog cat classification) of the image IM10 estimated in FIG.

  The “estimation target” indicates a classification target (target) to be estimated. “Class” indicates a class for classifying the estimation target. “Score” indicates a score that is an evaluation value of the corresponding class. For example, “score” indicates a ratio estimated to be classified into a corresponding class.

  For example, in the example illustrated in FIG. 8, the estimation target is the image IM10. Further, the image IM10 indicates that the score of the cat (class CL1) is “0.55”. For example, the ratio that the image IM10 is classified as a cat (class CL1) is 55%. Further, the image IM10 indicates that the score of the dog (class CL2) is “0.45”. For example, the ratio that the image IM10 is classified as a dog (class CL2) is 45%.

  Note that the estimated information storage unit 124 is not limited to the above, and may store various information according to the purpose.

(Control unit 130)
Returning to the description of FIG. 4, the control unit 130 is a controller and is stored in a storage device inside the generation apparatus 100 by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. Various programs (corresponding to an example of a generation program) are implemented by using the RAM as a work area. The control unit 130 is a controller, and is realized by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The control unit 130 includes an input layer to which target information to be classified is input, an output layer, and an input layer to an output layer by information processing according to the model M1 stored in the model information storage unit 122. A first element belonging to any layer other than the output layer, and a second element whose value is calculated based on the first element and the weight of the first element, and is input to the input layer For each target information, the elements belonging to each layer other than the output layer are set as the first element, and the calculation based on the first element and the weight of the first element is performed. The score value used for estimation is output from the output layer.

  As shown in FIG. 4, the control unit 130 includes an acquisition unit 131, a generation unit 132, an estimation unit 133, and a provision unit 134, and implements or executes the information processing functions and operations described below. . Note that the internal configuration of the control unit 130 is not limited to the configuration illustrated in FIG. 4, and may be another configuration as long as information processing described later is performed. The control unit 130, by information processing according to the model M1 (model data MDT1) stored in the storage unit 120, a second element whose value is calculated based on the first element and the weight of the first element, And performing calculation based on the first element and the weight of the first element with each element belonging to each layer other than the output layer as the first element for the target information input to the input layer The score value used for estimating the ratio classified into each class is output from the output layer.

(Acquisition part 131)
The acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 acquires various types of information from the learning data storage unit 121, the model information storage unit 122, the user information storage unit 123, the estimated information storage unit 124, and the like. The acquisition unit 131 may acquire various types of information from an external information processing apparatus. The acquisition unit 131 may acquire various types of information from the terminal device 10 or the like. For example, the acquisition unit 131 acquires an answer to the target information from the worker.

  For example, the acquisition unit 131 acquires target information to be classified and ratio information indicating a ratio classified into each class of target information by each of a plurality of users. For example, the acquisition unit 131 acquires target information to be classified and ratio information indicating a ratio classified into each class of target information selected by each of a plurality of users. For example, the acquisition unit 131 acquires ratio information indicating a ratio classified into each class of target information by each of a plurality of users in crowdsourcing. For example, the acquisition unit 131 acquires ratio information based on the number of users and the number of users who have selected each class of target information. For example, the acquisition unit 131 acquires target information including image information. For example, the acquisition unit 131 acquires target information including character information.

  In the example of FIG. 1, the acquisition unit 131 collects information for generating correct answer information. For example, the acquisition unit 131 classifies the image group IML1 such as the images IM101 to IM103 into users (workers) by crowdsourcing, and acquires the answer.

  In the example of FIG. 1, the acquisition unit 131 acquires a response to the image IM101 from the user U1. For example, the acquisition unit 131 acquires a reply from the user U1 that the image IM101 is “cat”. For example, the acquisition unit 131 acquires a reply from the user U2 that the image IM101 is “cat”. For example, the acquisition unit 131 acquires a reply from the user U3 that the image IM101 is “dog”. For example, the acquisition unit 131 acquires a reply from the user U4 that the image IM101 is “dog”. For example, the acquisition unit 131 acquires a reply from the user U5 that the image IM101 is “cat”.

  For example, the acquisition unit 131 acquires new target information. In the example of FIG. 2, the acquisition unit 131 acquires the image IM10 to be classified from the terminal device 10 used by the user U101.

(Generator 132)
The generation unit 132 generates various types of information. For example, the generation unit 132 generates a model as shown in the model information storage unit 122 using the learning data stored in the learning data storage unit 121. For example, the generation unit 132 generates a model used for estimating the ratio classified into each class of the target information based on the learning data acquired by the acquisition unit 131. For example, the generation unit 132 is a model used for estimating a ratio classified into each class of target information based on learning data including target information and correct answer information indicating a ratio at which the target information is classified into each class. Is generated.

  For example, based on the target information and the ratio information acquired by the acquisition unit 131, the generation unit 132 estimates the ratio at which one target information is classified into each class when the one target information is input. Generate a model. For example, when one piece of target information is input, the generation unit 132 generates a model that is a neural network that estimates the rate at which the one piece of target information is classified into each class. For example, the generation unit 132 generates a model that is a neural network that performs convolution processing and pooling processing. For example, the generation unit 132 generates a model based on target information in which a difference between ratios corresponding to each class in the ratio information is within a predetermined range.

  For example, the generation unit 132 generates models M1 to M3 and stores the generated models M1 to M3 and the like in the model information storage unit 122. Note that the generation unit 132 may generate the models M1 to M3 and the like using any learning algorithm. For example, the generation unit 132 generates models M1 to M3 and the like using a learning algorithm such as a neural network, a support vector machine (SVM), clustering, and reinforcement learning. As an example, when the generation unit 132 generates models M1 to M3 and the like using a neural network, the models M1 to M3 and the like include an input layer including one or more neurons, an intermediate layer including one or more neurons, and one And an output layer including the above neurons.

  The generation unit 132 generates a model and stores the generated model in the model information storage unit 122. Specifically, the generation unit 132 includes an input layer to which target information to be classified is input, an output layer, and any layer from the input layer to the output layer and belonging to layers other than the output layer. Each element belonging to each layer other than the output layer with respect to the target information input to the input layer, including one element and a second element whose value is calculated based on the first element and the weight of the first element As a first element, a model that outputs a score value used for estimating a ratio classified into each class of target information from the output layer by performing an operation based on the first element and the weight of the first element Generate.

  In the example of FIG. 1, the generation unit 132 generates ratio information indicating a ratio classified into each class of target information. For example, the generation unit 132 adds a combination of target information and total information including ratio information as correct answer information as learning data. For example, the generation unit 132 adds data DT101 to DT103 and the like corresponding to each of the images IM101 to IM103 that are target information to the learning data storage unit 121.

  In the example of FIG. 1, the generation unit 132 generates a model based on learning data including the images IM101 to IM103 as described above. For example, the generation unit 132 generates a model by performing learning using the data DT101 to DT103 in the learning data storage unit 121 as learning data (teacher data). As illustrated in the model information storage unit 122, the generation unit 132 generates a model (model M1) identified by the model ID “M1” by the process illustrated in FIG.

  For example, the generation unit 132 generates an estimation result to be provided to the user based on the information estimated by the estimation unit 133. In the example of FIG. 2, the generation unit 132 generates the estimation result information ER21 that the ratio that the image IM10 that is the estimation target is classified as a cat is 55% and the ratio that is classified as a dog is 45%. .

(Estimation unit 133)
The estimation unit 133 estimates various information. The estimation unit 133 estimates various information using information stored in the learning data storage unit 121, the model information storage unit 122, the user information storage unit 123, the estimation information storage unit 124, and the like. For example, the estimation unit 133 estimates various information based on various information acquired by the acquisition unit 131.

  For example, the estimation unit 133 estimates the ratio classified into each class with respect to the target information using the model. The estimation part 133 estimates the ratio classified into each class with respect to new object information by inputting new object information into a model.

  In the example of FIG. 2, the estimation unit 133 calculates a score indicating the ratio at which the image IM10 is classified into each class by processing as shown in the processing group PS21. The estimation unit 133 inputs the data IM10 to the model M1. The model M1 to which the data IM10 is input by the estimation unit 133 outputs a score. The model M1 outputs a score corresponding to each class. In the example of FIG. 2, the model M1 to which the data IM10 is input outputs the score “0.55” of the cat (class CL1) and the score “0.45” of the dog (class CL2) as shown in the score SC11. Is output. As a result, the estimation unit 133 uses the model M1 and the ratio (probability) that the image IM10 is classified as a cat (class CL1) is 55%, and the ratio (probability) that the image IM10 is classified as a dog (class CL2). Estimated to be 45%.

  For example, the estimation unit 133 calculates a score using a model having an arbitrary structure such as the above-described regression model or neural network. Specifically, the model M1 is a value obtained by quantifying the estimation of a predetermined target (that is, one item) when target information to be classified (that is, each element used for calculating the score described above) is input. The coefficient is set so as to output a score indicating whether or not there is a high possibility that the target information is classified into each class. The estimation unit 133 calculates a score related to each exhibition target using such a model M1.

  In the above example, an example in which the model M1 is a model that outputs a value obtained by quantifying the estimation of the ratio classified into each class of the target information when the target information to be classified is input. . However, the model (model X) according to the embodiment may be a model generated based on a result obtained by repeatedly inputting / outputting data to / from the model M1. For example, the model X may be a model (model Y) that is learned so that target information to be classified is input and a score output by the model M1 is output. Alternatively, the model M1 may be a model that has been learned so that target information to be classified is input and the output value of the model Y is output. When the estimation unit 133 performs an estimation process using GAN (Generative Adversarial Networks), the model M1 may be a model that constitutes a part of the GAN.

(Providing unit 134)
The providing unit 134 provides various information. For example, the providing unit 134 provides various information to the terminal device 10. The providing unit 134 provides a service based on a ratio in which one piece of target information estimated by the estimating unit 133 is classified into each class. For example, the providing unit 134 provides information indicating which class the one target information is based on the ratio of the one target information estimated by the estimation unit 133 into each class. Further, for example, the providing unit 134 may provide information regarding the model generated by the generating unit 132 to an external information processing apparatus. For example, the providing unit 134 may provide information output from the model to an external information processing apparatus.

  In the example of FIG. 1, the providing unit 134 provides target information to the terminal device 10-1 used by the user U1 who is a worker. For example, the providing unit 134 provides the image IM101, which is target information, to the terminal devices 10-1 to 10-5. For example, the providing unit 134 provides the terminal device 10-1 to 10-5 with the image group IML1 that is the target information.

  For example, the providing unit 134 provides information based on the estimation result. In the example of FIG. 2, the providing unit 134 provides the estimation result information ER <b> 21 generated by the generating unit 132 to the terminal device 10.

[4. Generation process flow)
Next, the procedure of the generation process by the generation system 1 according to the embodiment will be described with reference to FIG. FIG. 9 is a flowchart illustrating an example of the generation process according to the embodiment.

  As illustrated in FIG. 9, the generation device 100 acquires learning data (step S101). For example, the generation device 100 acquires learning data from the learning data storage unit 121.

  Thereafter, the generation device 100 generates a model based on the learning data (step S102). In the example of FIG. 1, the generation apparatus 100 generates a model M1 using learning data from the learning data storage unit 121.

[5. (Estimation process flow)
Next, the procedure of the estimation process by the generation system 1 according to the embodiment will be described with reference to FIG. FIG. 10 is a flowchart illustrating an example of the estimation process according to the embodiment.

  As illustrated in FIG. 10, the generation device 100 acquires target information (step S201). In the example of FIG. 2, the generation device 100 acquires the image IM10 as target information from the terminal device 10 used by the user U101.

  Further, the generation apparatus 100 estimates the ratio of classification into each class for the target information using the target information and the model (Step S202). In the example of FIG. 2, the generation apparatus 100 uses the model M1, and the ratio (probability) that the image IM10 is classified as a cat (class CL1) is 55%, and the ratio (class CL2) that is classified as a dog (class CL2) ( (Probability) is estimated to be 45%.

  In addition, the generation device 100 provides information related to the ratio classified into each class with respect to the estimated target information (step S203). In the example of FIG. 2, the generation apparatus 100 uses the estimation result information ER21 that the image IM10 to be estimated is classified as a cat by 55% and the ratio classified as a dog by 45% as user U101. Is provided to the terminal device 10 used.

[6. effect〕
As described above, the generation device 100 according to the embodiment includes the acquisition unit 131 and the generation unit 132. The acquisition unit 131 acquires target information to be classified and ratio information indicating a ratio classified into each class of target information by each of a plurality of users. Further, the generation unit 132 estimates the ratio of classification of one target information into each class when the one target information is input based on the target information and the ratio information acquired by the acquisition unit 131. Generate a model.

  As described above, in the generation apparatus 100 according to the embodiment, one target information is input based on the target information and the ratio information indicating the ratio classified into each class of the target information by each of a plurality of users. In this case, by generating a model that estimates the rate at which one target information is classified into each class, the rate at which the target information is classified into each class can be appropriately estimated.

  In the generation device 100 according to the embodiment, the acquisition unit 131 acquires ratio information indicating a ratio classified into each class of target information by each of a plurality of users in crowdsourcing.

  As described above, the generation apparatus 100 according to the embodiment is classified into each class of the target information by acquiring the ratio information indicating the ratio classified into each class of the target information by each of a plurality of users in crowdsourcing. Can be appropriately estimated.

  In the generation device 100 according to the embodiment, the acquisition unit 131 acquires ratio information based on the number of a plurality of users and the number of users who have selected each class of target information.

  As described above, the generation apparatus 100 according to the embodiment is classified into each class of the target information by acquiring the ratio information based on the number of the plurality of users and the number of users who have selected each class of the target information. Can be appropriately estimated.

  In the generation device 100 according to the embodiment, the acquisition unit 131 acquires target information including image information.

  As described above, the generation apparatus 100 according to the embodiment can appropriately estimate the ratio of classification into each class of the target information by acquiring the target information including the image information.

  In the generation device 100 according to the embodiment, the acquisition unit 131 acquires target information including character information.

  As described above, the generation apparatus 100 according to the embodiment can appropriately estimate the ratio of classification into each class of the target information by acquiring the target information including the character information.

  Further, in the generation device 100 according to the embodiment, the generation unit 132 generates a model that is a neural network that estimates a ratio of one target information classified into each class when the one target information is input. .

  As described above, the generation apparatus 100 according to the embodiment generates a model that is a neural network that estimates a ratio of one target information classified into each class when one target information is input. It is possible to appropriately estimate the ratio of the target information classified into each class.

  In the generation device 100 according to the embodiment, the generation unit 132 generates a model that is a neural network that performs convolution processing and pooling processing.

  As described above, the generation apparatus 100 according to the embodiment can appropriately estimate the ratio classified into each class of target information by generating a model that is a neural network that performs convolution processing and pooling processing. it can.

  In addition, the generation device 100 according to the embodiment includes an estimation unit 133. The estimation unit 133 estimates the ratio classified into each class for the target information using the model.

  As described above, the generation apparatus 100 according to the embodiment can appropriately estimate the ratio classified into each class of the target information by estimating the ratio classified into each class with respect to the target information using the model. can do.

  In the generation device 100 according to the embodiment, the acquisition unit 131 acquires new target information. The estimation part 133 estimates the ratio classified into each class with respect to new object information by inputting new object information into a model.

  As described above, the generation apparatus 100 according to the embodiment classifies the target information into each class by estimating the ratio of the new target information that is classified into each class by inputting the new target information into the model. It is possible to appropriately estimate the ratio to be performed.

  In the generation device 100 according to the embodiment, the generation unit 132 generates a model based on target information in which the difference between the ratios corresponding to each class in the ratio information is within a predetermined range.

  As described above, the generation apparatus 100 according to the embodiment generates each model of the target information by generating the model based on the target information in which the difference between the ratios corresponding to the classes in the ratio information is within a predetermined range. It is possible to appropriately estimate the ratio classified as “1”.

[7. Hardware configuration)
The generation apparatus 100 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in FIG. 11, for example. FIG. 11 is a hardware configuration diagram illustrating an example of a computer that realizes the function of the generation device. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F). ) 1700.

  The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each unit. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program depending on the hardware of the computer 1000, and the like.

  The HDD 1400 stores programs executed by the CPU 1100, data used by the programs, and the like. The communication interface 1500 receives data from other devices via the network N and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the network N.

  The CPU 1100 controls an output device such as a display and a printer and an input device such as a keyboard and a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. In addition, the CPU 1100 outputs the generated data to the output device via the input / output interface 1600.

  The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads such a program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. Etc.

  For example, when the computer 1000 functions as the generation apparatus 100 according to the embodiment, the CPU 1100 of the computer 1000 performs control by executing a program or data (for example, model M1 (model data MDT1)) loaded on the RAM 1200. The function of the unit 130 is realized. The CPU 1100 of the computer 1000 reads and executes these programs or data (for example, model M1 (model data MDT1)) from the recording medium 1800. As another example, these programs or data are transmitted from other devices via the network N. May be obtained.

  As described above, some of the embodiments and modifications of the present application have been described in detail with reference to the drawings. It is possible to carry out the present invention in other forms that have been modified and improved.

[8. Others]
In addition, among the processes described in the above-described embodiments and modifications, all or a part of the processes described as being automatically performed can be manually performed, or are described as being performed manually. All or part of the processing can be automatically performed by a known method. In addition, the processing procedures, specific names, and information including various data and parameters shown in the document and drawings can be arbitrarily changed unless otherwise specified. For example, the various types of information illustrated in each drawing is not limited to the illustrated information.

  Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured.

  In addition, the above-described embodiments and modifications can be combined as appropriate within a range that does not contradict processing contents.

  In addition, the “section (module, unit)” described above can be read as “means” or “circuit”. For example, the acquisition unit can be read as acquisition means or an acquisition circuit.

DESCRIPTION OF SYMBOLS 1 Generation system 100 Generation apparatus 121 Learning data storage part 122 Model information storage part 123 User information storage part 124 Estimated information storage part 130 Control part 131 Acquisition part 132 Generation part 133 Estimation part 134 Provision part 10 Terminal apparatus N network

Claims (14)

An acquisition unit that acquires target information to be classified and ratio information indicating a ratio classified into each class of the target information by each of a plurality of users;
Based on the target information and the ratio information acquired by the acquisition unit, a model is generated that estimates a ratio at which the one target information is classified into each class when the one target information is input. A generator to
A generating apparatus comprising: The acquisition unit
Obtaining ratio information indicating a ratio classified into each class of the target information by each of the plurality of users in crowdsourcing;
The generating apparatus according to claim 1, wherein: The acquisition unit
Obtaining the ratio information based on the number of the plurality of users and the number of users who have selected each class of the target information;
The generating apparatus according to claim 1 or claim 2, wherein The acquisition unit
The generation apparatus according to claim 1, wherein the target information including image information is acquired. The acquisition unit
The generation apparatus according to claim 1, wherein the target information including character information is acquired. The generator is
The said model which is a neural network which estimates the ratio by which said one target information is classified into each said class when said one target information is input is generated. The generating device according to claim 1. The generator is
The generation apparatus according to claim 6, wherein the model that is the neural network that performs the convolution process and the pooling process is generated. Using the model, an estimation unit that estimates a ratio classified into each class with respect to target information,
The generator according to claim 1, further comprising: The acquisition unit
Obtain new target information,
The estimation unit includes
The generation apparatus according to claim 8, wherein the new target information is input to the model to estimate a ratio of the new target information that is classified into each class. The generator is
The said model is produced | generated based on the said target information from which the difference between the ratios corresponding to each class in the said ratio information becomes in a predetermined | prescribed range. The Claim 1 characterized by the above-mentioned. Generator. A generation method executed by a computer,
An acquisition step of acquiring target information to be classified and ratio information indicating a ratio classified into each class of the target information by each of a plurality of users;
Based on the target information and the ratio information acquired in the acquisition step, a model is generated that estimates a ratio at which the one target information is classified into each class when one target information is input. Generating process to
A generation method comprising: An acquisition procedure for acquiring target information to be classified and ratio information indicating a ratio classified into each class of the target information by each of a plurality of users;
Based on the target information acquired by the acquisition procedure and the ratio information, when one target information is input, a model for estimating a ratio at which the one target information is classified into each class is generated Generation procedure to
A program for causing a computer to execute. Learning data including target information to be classified and correct information indicating a ratio classified into each class of the target information by each of a plurality of users,
A first element having an input layer and an output layer, the layer belonging to any layer from the input layer to the output layer other than the output layer, and weights of the first element and the first element A second element whose value is calculated based on the above and the target information input to the input layer, the elements belonging to each layer other than the output layer as the first element, the first element and the The output value indicating the calculation result is output from the output layer of the model by being input to the input layer of the model that performs the calculation based on the weight of the first element, and the correct information corresponding to the target information To perform learning based on the comparison with the output value,
Learning data to make the computer function. An input layer for inputting target information to be classified;
The output layer,
A first element belonging to any layer from the input layer to the output layer other than the output layer;
A second element whose value is calculated based on the first element and a weight of the first element;
By performing an operation based on the first element and the weight of the first element with respect to the target information input to the input layer, each element belonging to each layer other than the output layer as the first element, In order to output a score value indicating the ratio classified into each class of target information from the output layer,
A model that allows computers to function.

Images