JP7179705B2 - Information processing device, information processing method and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Conventionally, techniques related to image feature extraction by neural networks have been provided. For example, a technique for generating a saliency map of an image is provided by a convolutional neural network.

Karen Simonyan, Andrea Vedaldi, Andrew Zisserman, "Deep Inside Convolutional Networks: Visualizing Image Classification Models and Saliency Maps", International Conference on Machine Learning (ICLR), Apr14-16, 2014, Banff, Canada. Misha Denil, Alban Demiraj, Nando de Freitas, "Extraction of Salient Sentences from Labeled Documents", International Conference on Machine Learning (ICLR), Apr14-16, 2015, San Diego, USA.

However, with the above-described conventional technology, it is not always possible to extract low-level features according to the object corresponding to the classification result with high accuracy.

The present application has been made in view of the above, and provides an information processing device, an information processing method, and an information processing program capable of highly accurately extracting low-order features according to an object corresponding to a classification result. for the purpose.

An information processing apparatus according to the present application includes a specifying unit that specifies a useful region useful for class classification in an image to be processed, and an image included in the useful region specified by the specifying unit in the image to be processed. and an extracting unit for extracting low-order features from.

According to one aspect of the embodiment, there is an effect that it is possible to highly accurately extract low-order features according to the target object corresponding to the classification result.

FIG. 1 is a diagram illustrating an overview of information processing according to an embodiment; FIG. 2 is a diagram illustrating an example of information processing according to the embodiment; FIG. 3 is a diagram illustrating a configuration example of an information processing system according to the embodiment; FIG. 4 is a diagram illustrating a configuration example of an information processing apparatus according to the embodiment; 5 is a diagram illustrating an example of an image information storage unit according to the embodiment; FIG. 6 is a diagram illustrating an example of a feature information storage unit according to the embodiment; FIG. FIG. 7 is a flowchart illustrating a learning processing procedure according to the embodiment; FIG. 8 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing apparatus.

Hereinafter, modes for implementing an information processing apparatus, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described with reference to the drawings. The information processing apparatus, information processing method, and information processing program according to the present application are not limited to this embodiment. In addition, in the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

[1. Overview of information processing]
First, an overview of information processing according to the embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating an overview of information processing according to an embodiment; Information processing according to the embodiment is performed by the information processing apparatus 100 .

Prior to the description of FIG. 1, the information processing system according to the embodiment will be described with reference to FIG. FIG. 3 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. The information processing system 1 according to the embodiment includes a terminal device 10, a provider device 20, and an information processing device 100, as shown in FIG. The terminal device 10, the provider device 20, and the information processing device 100 are communicably connected via a network N by wire or wirelessly. The information processing system 1 shown in FIG. 3 may include multiple terminal devices 10 , multiple provider devices 20 , and multiple information processing devices 100 .

The terminal device 10 is an information processing terminal used by a user. The terminal device 10 is, for example, an information processing terminal used by a user who wants to search for products. The terminal device 10 is, for example, a smart phone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like.

For example, the terminal device 10 transmits a search query to the information processing device 100 according to a user's operation. For example, a user may wish to search for items that are similar or match an object depicted in a given image. In such a case, the user inputs this predetermined image into the terminal device 10 and presses the search button. Then, the terminal device 10 transmits this predetermined image to the information processing device 100 as a search query. In such a case, the predetermined image can be called a search query image or the like.

The provider device 20 is an information processing terminal used by a provider who provides an image that can be a search result corresponding to a search query (search query image). In this embodiment, the provider is assumed to be a store that sells products to a shopping service (hereinafter sometimes referred to as "service SH") operated by the business owner of the information processing device 100. do. In such a case, an image that can be a search result will actually show (introduce) the product of the provider, and thus such an image can be called a product image.

Here, premises for performing information processing according to the embodiment will be described. For example, it is assumed that the user wants to purchase the object OBx1 shown in the image IMGx1 and tries to search whether the object OBx1 is sold as a product. In such a case, by extracting the low-level features of the product included in the product image from the product image submitted by the provider (store), the product having the low-level features corresponding to the object OBx1 can be obtained. It is conceivable to search for the projected product image.

However, it is not always possible to extract low-order features with high accuracy. For example, if the product image is an abstract and simple image in which the product has a single background color, it may be possible to extract the low-order features of the product with high accuracy. However, in most cases, many product images are not in such a convenient mode that low-order features can be extracted with high accuracy as described above. For example, product images often consist of not only the product but also various other objects, people, complex backgrounds, and complex colors. cannot be extracted to

In addition, in order to extract low-level features from product images, extraneous information other than products (for example, objects and backgrounds other than products) can be removed to separate products. can be extracted. As such a method, by learning a model that can perform pixel-level segmentation and rectangle-level object detection, there is a method of removing unnecessary information from the product image to be processed using such a model. mentioned. However, there is a problem that the cost required for model learning and calculation using the model is high.

In other words, due to the structure of the product image, it is not possible to extract the low-order features of the product with high accuracy. is not realistic. Therefore, based on such assumptions and problems, the present embodiment uses an existing learning device (model) that classifies objects (for example, products) in an image to reduce costs. Another object of the present invention is to extract low-order features of an object (for example, a product) in an image with high accuracy.

Specifically, in the embodiment, the information processing apparatus 100 uses an existing learning device (model) to reduce costs and accurately detect low-level features of an object (for example, a product) in an image. The following measures are performed as information processing for extraction. Specifically, the information processing apparatus 100 identifies a useful region useful for class classification in the image to be processed, and extracts low-level features from the image included in the identified useful region in the image to be processed. do. Specifically, the information processing apparatus 100 inputs an image to be processed into a model, which is a neural network that outputs classification results obtained by classifying an input image, and obtains predetermined information in the intermediate layer of the model. do. Then, the information processing device 100 identifies the useful area based on the predetermined information.

For example, the information processing apparatus 100 identifies, as a useful area, an area determined to contribute to the classification result higher than a predetermined value in the image to be processed based on predetermined information. For example, the information processing apparatus 100 acquires, as predetermined information, a feature map obtained in an intermediate layer and corresponding to the degree of contribution to the classification result for the image to be processed. Therefore, the information processing apparatus 100 maps the feature map obtained in the intermediate layer to the classification result of the image to be processed in a pooled state, and compares the feature map after mapping with the image to be processed. Based on the result, a region determined to contribute to the classification result higher than a predetermined value in the image to be processed is identified as a useful region. Further, for example, the information processing apparatus 100 determines that the image to be processed is an image to be processed that is divided into grids based on the feature map, and the contribution of the image to be processed is determined based on the contribution calculated for each grid. A region determined to be higher than a predetermined value is specified as a useful region.

An example of information processing according to the above embodiment is shown. For example, if an existing learning device determines that a predetermined object included in an image to be processed belongs to class CL, the information processing apparatus 100 according to the embodiment becomes a trigger for determining that it belongs to class CL. Information is obtained from the middle layers of the model, ie information useful in determining that it belongs to class CL. Then, the information processing apparatus 100 applies the acquired information to the image to be processed, thereby identifying useful useful regions determined to be class CL in the image to be processed.

First, an example of the overall flow of information processing according to the embodiment will be described below with reference to FIG. After that, FIG. 2 is used to show an example of information processing according to the embodiment.

In the example of FIG. 1, provider device 20 is used by provider T1, and terminal device 10 is used by user U1. Further, in the example of FIG. 1, the provider device 20 submits the product image IM11 to the information processing device 100 according to the operation of the provider T1 (step S11). The product image IM11 shows the product handled by the provider T1, that is, the object OB11 to be traded. According to the example of FIG. 1, object OB11 is a T-shirt.

Here, as described above, in the example of FIG. 1, in order to show an example in which the information processing apparatus 100 receives product images only from the provider T1, the information processing according to the embodiment focuses on the product image IM11. Indicates information processing. However, the information processing apparatus 100 actually receives product images from many providers other than the provider T1. processing.

Returning to the description of FIG. Upon receiving the product image IM11, the information processing apparatus 100 performs a series of identification processes for identifying useful regions useful for class classification in the product image IM11 (step S12). Here, as learning information about the existing learner LE, a model M, which is a neural network that outputs a result of classifying a predetermined object included in an input image, is taken as an example. Also, when the product image IM11 is applied to the existing learning device LE having the model M as learning information, it is assumed that the class "T-shirt" is output as the class (category) to which the object OB11 to be traded belongs. Then, in step S12, the information processing apparatus 100 performs a series of specifying processes for specifying useful regions that have become useful for classifying the object OB11 into the class "T-shirt" in the product image IM11.

Next, the information processing apparatus 100 extracts low-order features related to a predetermined object from the image included in the useful area specified in step S12 in the product image IM11 (step S13). For example, the information processing apparatus 100 extracts low-order features related to the object OB11 to be traded. Details of the specifying process in step S12 and the extraction process in step S13 will be described later with reference to FIG. In addition, through the processing up to this point, the information processing apparatus 100 can obtain low-order features related to a predetermined object (object to be traded) included in the product image for each product image.

In such a state, it is assumed that the information processing apparatus 100 receives a search query image QE21 from the terminal device 10 of the user U1 (step S14). Then, the information processing apparatus 100 performs a similar image search to search for an image similar to the search query image QE21 from the product images submitted by each provider (step S15). Specifically, the information processing apparatus 100 searches for product images in which an object included in the search query image QE21 and similar to the specified object, which is an object specified by the user U1, is photographed as a transaction object. do. For example, the information processing apparatus 100 searches for product images whose trade targets are objects similar to the specified object by matching the specified object with low-order features extracted from each product image.

Then, the information processing device 100 transmits the product image (search result image) obtained by the search in step S15 to the terminal device 10 (step S16).

[2. Example of information processing]

From now on, an example of information processing according to the embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of information processing according to the embodiment; In FIG. 2, as the information processing according to the embodiment, details of the specifying process in step S12 and the extraction process in step S13 shown in FIG. 1 will be described.

According to the example of FIG. 1, the information processing apparatus 100 receives the submission of the product image IM11, and therefore inputs the product image IM11 (an example of the image to be processed) to the learning device LE (step S121).

Here, the learning device used by the information processing apparatus 100 will be described. The learning device LE used by the information processing apparatus 100 is, for example, a learning device in which a plurality of nodes that output operation results for input data are connected in multiple layers, and learns features of an abstracted image by supervised learning. is a learned learner. For example, the learner LE is a neural network in which layers having a plurality of nodes are connected in multiple stages, and may be a DNN (Deep Neural Network) realized by so-called deep learning technology. In addition, the features of an image include not only specific features that appear in the image, such as the presence or absence of text contained in the image, color, and composition, but also what the object being imaged is, and what kind of user the image is. It is a concept that includes features of abstracted (meta) images, such as whether they are liked by people or the atmosphere of the image.

For example, the learner LE is generated by the following learning method using deep learning technology. For example, in the learner LE, connection coefficients between nodes are initialized and images having various features are input. Then, the learning device LE is generated by processing such as back propagation (error backpropagation method) that corrects parameters (connection coefficients) so that the error between the output of the learning device LE and the input image is reduced. For example, the learner LE is generated by performing processing such as back propagation so as to minimize a predetermined loss function. By repeating the processing as described above, the learning device LE can output an output capable of better reproducing the input image, that is, output the features of the input image.

Note that the learning method of the learner LE is not limited to the method described above, and any known technique can be applied. Images used for learning by the learner may be data sets of various images such as images containing T-shirts and images not containing T-shirts. Any method can be applied to the method of inputting an image to the learning device LE, the format of the data output from the learning device LE, the contents of features that are explicitly learned by the learning device LE, and the like. In other words, the information processing apparatus 100 can use any learning device as long as it can calculate a feature quantity representing a feature abstracted from an image.

1 and 2, the information processing apparatus 100 uses a learner LE based on a so-called convolutional neural network that repeats convolution and pooling of local regions of an input image. Below, a convolutional neural network may be described as CNN. For example, the CNN-based learner LE has a function of extracting and outputting features from an image, and in addition, has invariance of output with respect to positional variations of characters, imaging targets, etc. included in the image. Therefore, the learning device LE can accurately calculate the abstracted features of the image. For example, the learning device LE calculates, as a classification result, a class to which a predetermined object included in the input image belongs as an abstract feature of the image.

Specifically, in FIG. 2, the information processing apparatus 100 classifies each of the plurality of objects included in the image to be processed into a class "T-shirt" among the plurality of objects. Let us use a learner LE that identifies objects that

For this reason, in FIG. 2, the learning device LE to which the product image IM11 is input identifies whether the product image IM11 includes a T-shirt (step S122). In FIG. 2, the product image IM11 includes an object OB11 that is a T-shirt. Therefore, the learning device LE classifies each of the plurality of objects included in the product image IM11, thereby identifying the object OB11 classified into the class “T-shirt” from among the plurality of objects. As a result, in step S122, the learning device LE generates the identification information IR indicating that the object OB11 classified into the class "T-shirt" is shown in the product image IM11 as an object to be traded. Note that step S122 is a process for explaining the operation of the learning device LE, and does not have to be performed.

Here, the learner LE is a learner generated by CNN and includes a plurality of intermediate layers A to C and the like. Therefore, the information processing apparatus 100 acquires information in a predetermined intermediate layer (hereinafter referred to as "intermediate image") when the product image IM11 is input to the learning device LE. In FIG. 2, the information processing apparatus 100 acquires an intermediate image in the intermediate layer B when the product image IM11 is input to the learning device LE (step S123). Specifically, information processing apparatus 100 acquires intermediate image group MG10 including intermediate images MM11 to MM19. Note that, in FIG. 2, regions showing characteristics in the intermediate images MM11 to MM19 are shown in dark colors. For example, the intermediate image MM12 indicates that the central portion includes a characteristic area. Further, for example, the intermediate image MM16 indicates that it does not include an area representing a substantial feature.

Intermediate images will be described in more detail. For example, the learning device LE obtains feature strength represented by a two-dimensional matrix from the product image IM11. Therefore, the intermediate image is a so-called feature map in which the matrix is heat-mapped. Therefore, in this embodiment, the intermediate image can be called a feature map.

Then, the information processing apparatus 100 extracts (obtains) an intermediate image that contributes to an improvement in the recognition rate of the predetermined target from the intermediate image group MG10 (step S124). In FIG. 2, the information processing apparatus 100 extracts intermediate images that contribute to improving the recognition rate of the T-shirt corresponding to the transaction object OB11 from the intermediate image group MG10. That is, the information processing apparatus 100 extracts an intermediate image whose degree of contribution is higher than a predetermined value and which has led to the determination that the object OB11 belongs to the class "T-shirt". In other words, the information processing apparatus 100 extracts an intermediate image in which the probability that the object OB11 to be traded is determined to be of the class "T-shirt" is higher than a predetermined value. Each intermediate image has a reduced size of the product image IM11, which is the original image, and has a grid shape with rougher image quality than the product image IM11. Here, for example, since the probability of being determined to be the class “T-shirt” is calculated for each grid, the information processing apparatus 100 determines that the probability of being determined to be the class “T-shirt” from the intermediate image group MG10 is higher than the predetermined value. Extract the intermediate image corresponding to the region composed of high grids.

In the example of FIG. 2, the information processing apparatus 100 selects the intermediate images MM12, MM14, MM17, and MM18 as the intermediate images whose degree of contribution is higher than a predetermined value and which has led to the determination that the object OB11 belongs to the class "T-shirt". is extracted. Note that the class “T-shirt” here is an example of the classification result for the product image IM11, which is the image to be processed.

Information processing apparatus 100 may extract an intermediate image based on a change in the T-shirt recognition rate according to processing of each of intermediate images MM11 to MM19 included in intermediate image group MG10. Processing of the intermediate image referred to here may be performed by various means depending on the purpose, such as increasing the brightness of the intermediate image by a predetermined value. It is presumed that an intermediate image that causes a change in the recognition rate of the T-shirt due to processing is an intermediate image that affects the recognition of the T-shirt. Therefore, the information processing apparatus 100 extracts an intermediate image that causes a change in the T-shirt recognition rate by processing.

Then, the information processing apparatus 100 synthesizes the intermediate images MM12, MM14, MM17, and MM18 extracted in step S124 (step S125). In FIG. 2, the information processing apparatus 100 generates a synthetic image CM11 by synthesizing intermediate images MM12, MM14, MM17, and MM18. Specifically, the information processing apparatus 100 pools the intermediate images MM12, MM14, MM17, and MM18 among the intermediate images MM11 to MM19 acquired from the intermediate layer B. FIG. Then, the information processing apparatus 100 maps the pooled intermediate images MM12, MM14, MM17, and MM18 to the class "T-shirt", which is the classification result of classifying the object OB11, thereby generating the composite image CM11. Note that the synthesized image CM11 is also a feature map. The composite image CM11 also has a reduced size of the product image IM11, which is the original image, and has a grid shape with rougher image quality than the product image IM11.

Then, as shown in FIG. 2, the information processing apparatus 100 generates a composite image CM11 including an area representing the characteristics of the object OB11, which is an object classified into the class "T-shirt".

Next, the information processing apparatus 100 identifies a useful area in the product image IM11 that is useful for classifying the object OB11 into the class “T-shirt” (step S126). For example, the information processing apparatus 100 enlarges the synthetic image CM11 to the same size as the product image IM11, compares the enlarged synthetic image with the product image IM11, and identifies an area including the object OB11. The area where the This point will be described in more detail. As described above, the composite image CM11 is an intermediate image corresponding to an area formed by grids with a higher probability than a predetermined value of being determined as the class "T-shirt". Therefore, the information processing apparatus 100 returns the composite image CM11 to the product image IM11 to obtain the product image IM11 divided into grids, and then, based on the probability (contribution) calculated for each grid, Then, an area determined to have a probability higher than a predetermined value is determined as a useful area.

In the example of FIG. 2, the information processing device 100 identifies the region R11 as the useful region. The product image IM11 shown in FIG. 2 includes a vase, a chair, and a complicated background in addition to the object OB11. Only the object OB11 to be detected is accurately extracted (detected).

Next, the information processing apparatus 100 extracts low-order features of the object extracted from the region R11 (that is, the object OB11) (step S127). The low-level features referred to here are the hue, edge, flow, etc. of the object extracted in the region R11. For example, the information processing apparatus 100 extracts the low-order features of the object extracted in the region R11 by analyzing the image of the portion extracted in the region R11 of the product image IM11. The example of FIG. 2 shows an example in which the information processing apparatus 100 extracts a low-level feature FA11 as a low-level feature. Also, the extracted low-order features are stored in the feature information storage unit 122 .

As described with reference to FIGS. 1 and 2, the information processing apparatus 100 according to the embodiment identifies useful regions useful for class classification in an image to be processed, , to extract low-level features (low-level features) from the image included in the useful region. For example, as described above, the information processing apparatus 100 generates information used for appropriately recognizing objects classified into a predetermined class from information in a neural network (for example, CNN). In the example of FIGS. 1 and 2, the information processing apparatus 100 extracts intermediate images that contributed to the classification of the object OB11 as a T-shirt from the intermediate image group MG10 in the intermediate layer of the CNN. Then, the information processing apparatus 100 generates a synthetic image by synthesizing the extracted intermediate images. That is, the information processing apparatus 100 generates a composite image using only intermediate images that contributed to the classification of the object OB11 as a T-shirt. Therefore, the information processing apparatus 100 can generate a composite image that accurately shows the characteristic region of the object OB11 from the product image IM11. can be extracted with high precision. As a result, the information processing apparatus 100 can highly accurately extract the low-order features corresponding to the object corresponding to the classification result of the class "T-shirt", that is, the object OB11.

Further, since the information processing apparatus 100 according to the embodiment can use the low-level features extracted as described above in the similar image search described with reference to the drawings, for example, the information processing apparatus 100 can be more sensitive to the features of the search query image received from the user. Product images having similar features can be searched with high accuracy. As a result, the information processing apparatus 100 can improve user satisfaction in similar image retrieval.

Further, according to the information processing described with reference to FIGS. 1 and 2, the information processing apparatus 100 according to the embodiment can obtain low-order features with high accuracy while utilizing existing learning devices without using costly models. Therefore, low-order features can be extracted with high accuracy while suppressing cost.

[3. Configuration of Information Processing Device]
Next, the information processing apparatus 100 according to the embodiment will be described using FIG. FIG. 4 is a diagram illustrating a configuration example of the information processing apparatus 100 according to the embodiment. As shown in FIG. 4, the information processing apparatus 100 has a communication section 110, a storage section 120, and a control section . For example, the information processing device 100 is a server device that performs the information processing described with reference to FIGS. 1 and 2 .

(Regarding communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 110 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the terminal device 10 and the provider device 20, for example.

(Regarding storage unit 120)
The storage unit 120 is realized by, for example, a RAM (Random Access Memory), a semiconductor memory device such as a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 has an image information storage unit 121 and a feature information storage unit 122 .

(Regarding the image information storage unit 121)
The image information storage unit 121 stores various information about images. In the examples of FIGS. 1 and 2, the image information storage unit 121 stores information about product images submitted by the provider. Here, FIG. 5 shows an example of the image information storage unit 121 according to the embodiment. In the example of FIG. 5, the image information storage unit 121 has items such as "image ID" and "image data".

"Image ID" indicates identification information for identifying the product image submitted by the provider. "Image data" indicates the product image itself identified by the "image ID", that is, image information.

That is, the example of FIG. 5 shows an example in which the information processing apparatus 100 receives submission of a product image (product image IM11) identified by the image ID “IM11”. Although not shown in FIG. 5, a provider ID that identifies the provider may be further linked.

(Regarding the feature information storage unit 122)
The feature information storage unit 122 stores information on low-order features. Here, FIG. 6 shows an example of the feature information storage unit 122 according to the embodiment. In the example of FIG. 6, the feature information storage unit 122 has items such as "image ID", "area information", and "lower-level feature".

"Image ID" indicates identification information for identifying the product image submitted by the provider, as in the example of FIG. "Region information" is information indicating a useful region useful for class classification. The “area information” may be, for example, a product image (original image) identified by the “image ID” and stored as a product image indicating the useful area. "Low-order features" indicate low-order features extracted from the object surrounded by the useful region indicated by the "region information".

That is, in the example of FIG. 6, in the product image (product image IM11) identified by the image ID “IM11”, since the region R11 is specified as the useful region, the low-order feature of the object in the region R11 is An example in which feature FA11 is extracted is shown. Note that this example corresponds to the example shown in FIG. Further, although not shown in FIG. 5, information indicating an object extracted in the useful area may be further linked.

(Regarding the control unit 130)
Returning to FIG. 4, the control unit 130 is realized by executing various programs stored in the storage device inside the information processing apparatus 100 using the RAM as a work area by the CPU, MPU, or the like. Also, the control unit 130 is realized by an integrated circuit such as an ASIC or FPGA, for example.

As shown in FIG. 3, the control unit 130 includes an acquisition unit 131, an input unit 132, an extraction unit 133, a generation unit 134, an identification unit 135, a feature extraction unit 136, a search processing unit 137, a presentation 138, and implements or executes the information processing functions and actions described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 3, and may be another configuration as long as it performs information processing described later. Moreover, the connection relationship between the processing units of the control unit 130 is not limited to the connection relationship shown in FIG. 3, and may be another connection relationship.

(Regarding the acquisition unit 131)
Acquisition unit 131 acquires an image. For example, the acquisition unit 131 acquires product images from the provider. That is, the acquisition unit 131 receives a product image submission from the provider. In the example of FIGS. 1 and 2, the acquisition unit 131 acquires the product image IM11 from the provider device 20 of the provider T1. That is, the acquisition unit 131 receives the submission of the product image IM11 from the provider device 20 of the provider T1. The acquisition unit 131 also stores the acquired product image in the image information storage unit 1221 .

In addition, the acquisition unit 131 can acquire information necessary for the processing from the storage unit and output the information to the corresponding processing unit when the processing is performed by going to each processing unit. For example, when an image to be processed is input to the learning device by the input unit 132, the acquisition unit 131 acquires the image to be processed from the image information storage unit 121, and sends the acquired image to be processed to the input unit 132. Send.

(Regarding the input unit 132)
The input unit 132 inputs an image to be processed to a model, which is a neural network that outputs a classification result of classifying a predetermined object included in the input image. In the example of FIG. 2, the input unit 132 inputs the product image IM11 (an example of the image to be processed) to the learning device LE.

(Regarding the extraction unit 133)
The extraction unit 133 extracts (obtains) predetermined information in the intermediate layer of the neural network. For example, the extraction unit 133 extracts (obtains), as the predetermined information, a feature map obtained in the intermediate layer and corresponding to the degree of contribution to the classification result for the image to be processed. For example, the extraction unit 133 extracts an intermediate image (an example of a feature map) that contributes to improving the recognition rate of a given object from a group of intermediate images in the intermediate layer of the neural network that recognizes the given object in the image. For example, the extraction unit 133 classifies each of a plurality of objects included in the image to be processed, thereby identifying (recognizing) an object classified into the class “T-shirt” from among the plurality of objects. An intermediate image that contributes to an improvement in the T-shirt recognition rate is extracted from the group of intermediate images in the intermediate layer of the neural network. In other words, the extraction unit 133 extracts from the group of intermediate images an intermediate image whose degree of contribution is higher than a predetermined value and which has caused the determination that the image belongs to the class “T-shirt”.

Also, for example, the extraction unit 133 extracts an intermediate image that contributes to an improvement in the recognition rate of a predetermined object from an intermediate image group in an intermediate layer of a neural network that performs convolution processing and pooling processing. For example, the extraction unit 133 extracts an intermediate image that contributes to an improvement in the recognition rate of a predetermined object from intermediate images in the CNN intermediate layer. In FIG. 1, the extraction unit 133 extracts intermediate images MM12, MM14, MM17, and MM18 that contribute to improving the T-shirt recognition rate from the intermediate image group MG10 in the intermediate layer of the CNN.

(Regarding the generation unit 134)
The generation unit 134 generates a composite image by combining the intermediate images extracted by the extraction unit 133 . In FIG. 2, the generation unit 134 generates a synthetic image CM11 by synthesizing the intermediate images MM12, MM14, MM17, and MM18. Specifically, the generator 134 pools the intermediate images MM12, MM14, MM17, and MM18 out of the intermediate images MM11 to MM19 acquired from the intermediate layer B. FIG. Then, the generating unit 134 maps the pooled intermediate images MM12, MM14, MM17, and MM18 to the class “T-shirt”, which is the classification result of classifying the object OB11 (predetermined target), thereby generating the composite image CM11. to generate

(Regarding the identification unit 135)
The specifying unit 135 specifies a useful region useful for class classification in the image to be processed. For example, the specifying unit 135 determines whether the degree of contribution to the classification result in the image to be processed is determined based on the feature map obtained in the intermediate layer and based on the degree of contribution to the classification result of the image to be processed. A region higher than a predetermined value is determined (identified). Then, the specifying unit 135 specifies, as a useful region, a region determined to contribute to the classification result higher than a predetermined value in the image to be processed.

For example, the identification unit 135 maps the feature maps obtained in the intermediate layers to the classification result of the image to be processed in a pooled state, and compares the feature map after mapping with the image to be processed. Based on , an area determined to contribute to the classification result higher than a predetermined value in the image to be processed is specified as a useful area. As an example, the identification unit 135 determines that the image to be processed is an image to be processed divided into grids based on the feature map. A region determined to be higher than a predetermined value is specified as a useful region.

In the example of FIG. 2, the identifying unit 135 enlarges the synthetic image CM11 to the same size as the product image IM11, and compares the enlarged synthetic image with the product image IM11 to specify the area including the object OB11. do. Then, the specifying unit 135 defines the specified area as a useful area. Here, the composite image CM11 is an intermediate image corresponding to an area configured by grids with a higher probability than a predetermined value of being determined as the class “T-shirt”. Therefore, the specifying unit 135 returns the composite image CM11 to the product image IM11 to obtain the product image IM11 divided into grids, and based on the probability (contribution) calculated for each grid, , to determine (identify) regions where the probability is higher than a predetermined value. Then, the feature extracting unit 136 determines the region determined to have a probability higher than a predetermined value as a useful region. In the example of FIG. 2, the feature extraction unit 136 identifies the region R11 as the useful region.

(Regarding the feature extraction unit 136)
The feature extraction unit 136 extracts low-level features from the image included in the useful region identified by the identification unit 135, among the images to be processed. For example, the feature extraction unit 136 extracts low-order features of the object extracted in the region R11 by analyzing the image of the portion extracted in the region R11 of the product image IM11. The feature extraction unit 136 also stores the extracted low-order features in the feature information storage unit 122 .

(Regarding the search processing unit 137)
The search processing unit 137 performs search processing for searching for images corresponding to a search query. For example, the search processing unit 137 performs a similar image search for searching for similar images that are related (similar or matching) to the search query image. More specifically, the search processing unit 137 searches for a product image in which an object included in the search query image, for example, an object similar to a specified object, which is an object specified by the user, is photographed as a transaction target. . For example, the search processing unit 137 searches for product images whose transaction targets are objects similar to the specified object by matching the specified object with low-order features extracted from each product image.

(Regarding presentation unit 138)
The presentation unit 138 presents predetermined information to the access source user. For example, the presentation unit 138 presents (transmits) search results obtained by the search processing performed by the search processing unit 137 to the user who sent the search query. For example, the presentation unit 138 presents (transmits) a product image as a search result obtained by the search processing by the search processing unit 137 to the user who sent the search query image.

[4. Processing procedure]
Next, the procedure of information processing according to the embodiment will be described with reference to FIG. FIG. 7 is a flowchart illustrating a learning processing procedure according to the embodiment;

First, the acquisition unit 131 acquires an image to be processed (step S201). For example, the acquisition unit 131 acquires a product image submitted by a provider. Next, the input unit 132 inputs the processing target image acquired by the acquisition unit 131 to the learning device (step S202). For example, the input unit 132 inputs the image to be processed to a model, which is a neural network that outputs a result of classifying a predetermined object included in the input image.

Next, the extraction unit 133 acquires an intermediate image group in the intermediate layer of the neural network (step S203). Then, the extraction unit 133 extracts an intermediate image that contributes to an improvement in the recognition rate of a predetermined object (for example, an object to be traded) from the group of intermediate images acquired in step S203 (step S204). For example, the extraction unit 133 extracts, as an intermediate image that contributes to an improvement in the recognition rate (contribution), a feature map obtained in an intermediate layer that corresponds to the contribution to the classification result for the image to be processed. Extract (obtain).

Next, the generation unit 134 generates a composite image by combining the intermediate images extracted by the extraction unit 133 (step S205). For example, the generator 134 pools the feature maps extracted in step S205. Then, the generation unit 134 generates a composite image by mapping the pooled feature map to the classification result of classifying a predetermined object.

Next, the specifying unit 135 specifies a useful region useful for class classification in the image to be processed (step S206). For example, the identification unit 135 identifies an area including a predetermined object by comparing the synthesized image generated in step S206 and the image to be processed, and defines the identified area as the useful area. The synthesized image is an intermediate image (feature map) corresponding to an area formed by grids in which the probability that a given object has been determined to belong to the class indicated by the classification result is higher than a given value. For this reason, the identifying unit 135 obtains the product image to be processed divided into grids by returning the composite image to the image to be processed, and then the probability (contribution) calculated for each grid. Based on this, a region determined to have a probability higher than a predetermined value is defined as a useful region.

Then, the feature extraction unit 136 extracts low-level features from the image included in the useful region specified by the specification unit 135, among the images to be processed. In the useful area, the predetermined object is separated from other objects, the background, etc., with high accuracy, so the feature extraction unit 136 extracts low-level features indicating the predetermined object from the image that substantially includes only the predetermined object. do.

[5. Modification]
The information processing apparatus 100 according to the above embodiment may be implemented in various different forms other than the above embodiment. Therefore, other embodiments of the information processing apparatus 100 will be described below.

[5-1. feature vector]

In the above embodiment, an example was given in which the feature extraction unit 136 extracts low-order features from the image included in the useful region specified by the specification unit 135 . However, the feature extracting unit 136 extracts low-order features from the image corresponding to each grid in the grid-divided product image to be processed. Then, the feature extraction unit 136 converts the feature amount of the extracted low-level feature into a feature vector. Further, the feature extraction unit 136 performs control so that the converted feature vector becomes information used in similar image search using the image to be processed.

For example, each grid contains a portion of a given object (eg, a traded object). Therefore, the feature extraction unit 136 extracts low-order features for each part of a predetermined object and converts each extracted low-order feature into a feature vector. Then, the feature extraction unit 136 stores the image ID indicating the image to be processed and the feature vector obtained from the image to be processed in the feature information storage unit 122, for example, so that the feature information is stored in the feature information storage unit 122. Control so that vectors are used for similar image retrieval instead of low-order features.

As described above, the information processing apparatus 100 according to the embodiment extracts a low-order feature for each grid image obtained by dividing a product image to be processed into a grid, and converts each extracted low-order feature into a feature vector. Further, the information processing apparatus 100 controls so that each feature vector is used for targeting in similar image retrieval. As a result, the information processing apparatus 100 can accurately search for an image in which an object closer to the specified object specified in the search query image is shown as a transaction target, for example.

[5-2. Processing for increasing resolution]
In addition, the identifying unit 135 identifies useful regions useful for class classification in each of a plurality of consecutive images to be processed (for example, a plurality of identical images to be processed) as the images to be processed. Then, the feature extraction unit 136 generates one superimposed image in which the images included in the useful region specified by the specifying unit 135 and corresponding to each of the plurality of consecutive images to be processed are superimposed. Extract low-order features from As described above, the information processing apparatus 100 according to the embodiment superimposes images included in the useful region and corresponding to each of a plurality of consecutive images to be processed, thereby achieving higher resolution. image can be obtained. As a result, the information processing apparatus 100 can more accurately extract low-order features corresponding to the target object.

[6. Hardware configuration]
Further, the information processing apparatus 100 according to the above embodiment is implemented by a computer 1000 configured as shown in FIG. 8, for example. FIG. 8 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of the information processing apparatus 100. As shown in FIG. Computer 1000 has CPU 1100 , RAM 1200 , ROM 1300 , HDD 1400 , communication interface (I/F) 1500 , input/output interface (I/F) 1600 and media interface (I/F) 1700 .

The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400 and controls each section. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program depending on the hardware of the computer 1000, and the like.

HDD 1400 stores programs executed by CPU 1100 and data used by these programs. Communication interface 1500 receives data from other devices via communication network 50 and sends the data to CPU 1100 , and transmits data generated by CPU 1100 to other devices via communication network 50 .

The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, through an input/output interface 1600 . CPU 1100 acquires data from an input device via input/output interface 1600 . CPU 1100 also outputs the generated data to an output device via input/output interface 1600 .

Media interface 1700 reads programs or data stored in recording medium 1800 and provides them to CPU 1100 via RAM 1200 . CPU 1100 loads such a program from recording medium 1800 onto RAM 1200 via media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. etc.

For example, when the computer 1000 functions as the information processing apparatus 100 according to the embodiment, the CPU 1100 of the computer 1000 implements the functions of the control unit 130 by executing programs loaded on the RAM 1200 . In addition, data in storage unit 120 is stored in HDD 1400 . CPU 1100 of computer 1000 reads these programs from recording medium 1800 and executes them, but as another example, these programs may be obtained from another device via communication network 50 .

[7. others〕
Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

As described above, the embodiments of the present application have been described in detail based on several drawings, but these are examples, and various modifications and It is possible to carry out the invention in other forms with modifications.

Also, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the acquisition unit can be read as acquisition means or an acquisition circuit.

1 information processing system 10 terminal device 20 provider device 100 information processing device 120 storage unit 121 image information storage unit 122 feature information storage unit 130 control unit 131 acquisition unit 132 input unit 133 extraction unit 134 generation unit 135 identification unit 136 feature extraction unit 137 search processing unit 138 presentation unit

Claims (9)

A learner trained as a neural network for recognizing a predetermined object included in an image to be processed, wherein the predetermined an acquisition unit that acquires an intermediate image that contributes to an improvement in object recognition rate;
The degree of contribution of the image to be processed to the classification result is a predetermined value based on a comparison result of comparing the intermediate image mapped to the classification result for the image to be processed and the image to be processed. an identification unit that identifies the region determined to be higher as a useful region;
and an extracting unit that extracts low-order features from an image included in the useful region specified by the specifying unit, among the images to be processed. 2. The information processing apparatus according to claim 1, wherein the acquisition unit acquires, as the intermediate image, a feature map corresponding to a degree of contribution to a classification result for the image to be processed. The specifying unit maps the pooled feature maps to a classification result of the image to be processed, and compares the feature map after mapping with the image to be processed. 3. The information processing apparatus according to claim 2, wherein, in the image to be processed, an area determined to contribute to a classification result higher than a predetermined value is specified as the useful area. The specifying unit determines the degree of contribution in the image to be processed based on the degree of contribution calculated for each grid in the image to be processed divided into grids based on the feature map. 4. The information processing apparatus according to claim 3, wherein an area determined to be higher than a predetermined value is specified as the useful area. The extraction unit extracts low-order features from the grid for each grid, and the feature vector converted from the feature amount of the extracted low-order features is information used in similar image search using the image to be processed. 5. The information processing apparatus according to claim 4, wherein the control is performed such that The identifying unit identifies useful regions useful for class classification in each of a plurality of consecutive images to be processed, as the images to be processed,
The extracting unit extracts low-level features from one superimposed image obtained by superimposing a plurality of consecutive images corresponding to each of a plurality of images to be processed, which are included in the useful region specified by the specifying unit. The information processing apparatus according to any one of claims 1 to 5, characterized in that: 7. The extracting unit extracts, as the low-order features, low-order features that serve as information used in similar image retrieval using the image to be processed. The information processing device according to . An information processing method executed by an information processing device,
A learner trained as a neural network for recognizing a predetermined object included in an image to be processed, wherein the predetermined an acquisition step of acquiring an intermediate image that contributes to an improvement in object recognition rate;
The degree of contribution of the image to be processed to the classification result is a predetermined value based on a comparison result of comparing the intermediate image mapped to the classification result for the image to be processed and the image to be processed. An identifying step of identifying the region determined to be higher as a useful region;
an extracting step of extracting low-order features from an image included in the useful region specified in the specifying step, among the images to be processed. A learner trained as a neural network for recognizing a predetermined object included in an image to be processed, wherein the predetermined an acquisition procedure for acquiring an intermediate image that contributes to improving an object recognition rate;
The degree of contribution of the image to be processed to the classification result is a predetermined value based on a comparison result of comparing the intermediate image mapped to the classification result for the image to be processed and the image to be processed. An identification procedure for identifying a region determined to be higher as a useful region;
An information processing program for causing a computer to execute an extraction procedure for extracting low-order features from an image included in the useful region identified by the identification procedure, among the images to be processed.

Images