JP5018147B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

  The present invention relates to an information processing device, an information processing method, and a program, and in particular, information processing that makes it possible to easily search for desired information even when it is difficult to verbalize the desired information. The present invention relates to an apparatus, an information processing method, and a program.

  2. Description of the Related Art In recent years, there is a recording / reproducing apparatus that records and reproduces a television broadcast program (hereinafter simply referred to as a program) and automatically records the program based on the preference of a user who views the program. In such a recording / reproducing apparatus, for example, a program to be recorded is searched using a technique called content-based filtering (hereinafter referred to as CBF technique) (see, for example, Patent Document 1).

  The CBF method calculates the similarity between the feature vector of a program that is pre-assigned to the program by the distribution side or the sales side and the feature vector indicating the user's preference information based on the cosine distance, etc. This is a technique of searching and recording a program as a program that matches the user's preference. The feature vector indicating the user preference information is generated, for example, by having the user input a word representing the content of the user's favorite program.

  Searching for predetermined information based on an input word is performed in various situations. For example, when a user tries to search for desired information from the Internet or the like, it is generally performed to input a plurality of words that are strongly related to the desired information and to search based on the input words. It has been broken. Therefore, in searching for such information, it is necessary to verbalize the desired information to the word level in order to derive the desired information.

JP 2004-194107 A

  However, there are cases where it is difficult to verbalize desired information to the word level. In this case, the desired information cannot be searched or it may take a long time to search.

  The present invention has been made in view of such a situation, and when searching for desired information, even if it is difficult to verbalize the desired information, it can be easily searched. It is to make.

The information processing apparatus according to the first aspect of the present invention is the same as the image feature amount storage means for storing feature amounts representing image features for a plurality of registered images registered in advance, and the registered image and the predetermined item. As the information mapped on the space, an item correspondence storage means for storing the correspondence between the registered image and a predetermined item, and obtaining a query image that is an image designated by the user to search for a desired item Means, feature amount extraction means for extracting the feature amount of the query image, and similarity calculation means for calculating a first similarity between the query image and the registered image using the extracted feature amount. , item a short distance on the said first high similarity the registration image space, from the item correspondence storage unit, a presentation item is the item to be presented to the user And a presentation item determination means for determining Te.

  The similarity calculation means also calculates a second similarity that is a similarity between the registered image and the item, and the presentation item determination means causes the first similarity and the second similarity to be calculated. The present item can be determined based on the value obtained by adding the products of the similarities.

The presenting item determining means may calculate the frequency of an item whose distance in the space is close to the plurality of registered images having a high first similarity, and determine an item with a high frequency as the presenting item. it can.

The information processing method according to the first aspect of the present invention is the same as the image feature amount storage means for storing the feature amount representing the feature of the image for a plurality of registered images registered in advance, and the registered image and the predetermined item. An information processing method for an information processing apparatus comprising: item correspondence storage means for storing correspondence between the registered image and a predetermined item as information mapped on the space of the information processing apparatus, and determining a presentation item that is the item to be presented to the user A query image, which is an image designated by the user to search for a desired item, extracts a feature amount of the query image, and uses the extracted feature amount to extract the query image and the query image the similarity between the registered image is calculated and the high degree of similarity the registration image item distance is short on the space, from the item correspondence storage means, said Hisage And determining as an item.

The program according to the first aspect of the present invention includes an image feature amount storage unit that stores feature amounts representing image features for a plurality of registered images registered in advance, and the registered image and a predetermined item in the same space. A program for causing a computer to execute a process of determining a presentation item, which is the item to be presented to a user, using item correspondence storage means for storing correspondence between the registered image and a predetermined item as information mapped above A query image, which is an image designated by the user for searching for a desired item, extracts a feature amount of the query image, and uses the extracted feature amount to extract the query image and the query image the similarity between the registered image is calculated and the high degree of similarity the registration image item distance is short on the space, the item corresponding memory From stage, comprising determining as the presentation item.

In the first aspect of the present invention, a query image, which is an image designated by a user for searching for a desired item, is acquired, and its feature value is extracted. Further, using the extracted feature amount, the similarity between the query image and the registered image is calculated, and an item having a spatial distance close to the registered image having a high similarity is determined as the presentation item from the item correspondence storage unit. Is done.

  According to the first aspect of the present invention, the user can search for desired information.

  Further, according to the first aspect of the present invention, even when it is difficult to verbalize desired information, the user can easily search for desired information.

  Embodiments of the present invention will be described below. Correspondences between the constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

The information processing apparatus (for example, the server 3 in FIG. 3) according to the first aspect of the present invention stores image feature amount storage means (for example, a plurality of pre-registered registered images). The item correspondence storage unit (for example, FIG. 3) stores the correspondence between the registered image and the predetermined item as information obtained by mapping the feature amount DB 11 in FIG. 3 and the registered image and the predetermined item on the same space . Image item correspondence DB 12), an acquisition unit (for example, the communication unit 31 in FIG. 3) that acquires a query image that is an image designated by the user in order to search for a desired item, and a feature amount of the query image Similarity calculation for calculating a first similarity between the query image and the registered image, using feature amount extraction means (for example, the feature amount extraction unit 32 in FIG. 3) to be extracted and the extracted feature amount. The item (for example, the similarity calculation unit 33 in FIG. 3) and the item that presents to the user an item whose distance in the space is close to the registered image having the first similarity from the item correspondence storage unit It comprises presenting items determining means for determining as the presentation item is (e.g., presentation item determining unit 34 of FIG. 3) and a.

An information processing method according to the first aspect of the present invention includes an image feature amount storage unit (for example, a feature amount DB 11 in FIG. 3) that stores feature amounts representing image features for a plurality of registered images registered in advance. As information obtained by mapping the registered image and the predetermined item on the same space, an item correspondence storage unit (for example, the image item correspondence DB 12 in FIG. 3) that stores the correspondence between the registered image and the predetermined item is provided. In the information processing method of the information processing apparatus (for example, the server 3 in FIG. 3) that determines the presented item that is the item presented to the user, a query that is an image designated by the user to search for a desired item An image is acquired (for example, step S13 in FIG. 6), a feature amount of the query image is extracted (for example, step S14 in FIG. 6), and the extracted feature amount Is used to calculate the similarity between the query image and the registered image (for example, step S15 in FIG. 6), and the registered image having a high similarity is close to the item in the space. It includes a step of determining as the presentation item from the storage means (for example, step S18 in FIG. 6).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration example of an embodiment of a search system to which the present invention is applied.

  The search system of FIG. 1 is a system that can easily search for an item desired by the user. The mobile phone 1 that the user owns and inputs predetermined information, the Internet or LAN (Local Area network) and the information transmitted from the mobile phone 1 to search for an item desired by the user and information for identifying the item as the search result, for example, the name of the item, Alternatively, the server (information processing device) 3 that returns a place (item storage place) where the user can obtain the item to the mobile phone 1 is configured. In the following, transmitting information for identifying an item is simply referred to as transmitting an item.

  Here, the item refers to software or hardware (article) used by the user. For example, software (data) such as television broadcast programs, sentences, movies, photographs, music, etc. (moving images, still images, audio, or combinations thereof), articles such as clothes, shoes, food (hardware) ) Is an item in this specification.

  In a conventional search system, when a user searches for a predetermined item, a plurality of pieces of information that are strongly related to the desired item are input as words, and items related to any of the plurality of input words are searched. The process of presenting was performed.

  Accordingly, when it is difficult to express information strongly related to a desired item as a word, in other words, when there is no accurate word to express the desired item, the desired item may not be searched well.

  For example, when trying to search for clothes (items) that match the scenery that the user sees at a certain place, it is difficult to express the scenery that the user sees in several words, and it is difficult to search for a desired item. there were.

  The search system of FIG. 1 is a system that enables a desired item to be searched even in such a scene.

  The mobile phone 1 has at least a photographing function for photographing (capturing) an image and a communication function for transmitting and receiving character data and image data in addition to a normal call function. The mobile phone 1 is an image provided to the server 3 to search for a desired item based on a user's operation, and an image related to the desired item (hereinafter referred to as a query image) is displayed on the network 2. To the server 3. It is assumed that a search command indicating that a search process is executed using the query image is transmitted from the mobile phone 1 together with the query image.

  The server 3 receives the query image transmitted from the mobile phone 1 via the network 2 together with the search command. Then, the server 3 searches for a predetermined item based on the query image, and transmits (replies) the searched item to the mobile phone 1 via the network 2.

  Therefore, in the search system of FIG. 1, it is possible to search for a desired item without inputting a word.

  With reference to FIG. 2, an outline of the first item search process by the search system of FIG. 1 will be described.

  The mobile phone 1 transmits a query image designated by the user to the server 3. The query image can be, for example, an image taken by the mobile phone 1 or an image received by the mobile phone 1 by e-mail.

  A feature amount DB (Database) 11 of the server 3 stores feature amounts representing features of various images. The server 3 receives the query image transmitted from the mobile phone 1 and extracts the feature amount of the received query image. Then, the server 3 detects N (N ≧ 1) images (image groups) whose image feature amounts are similar to those of the query image from among a large number of images stored in the feature amount DB 11. An image detected from the feature amount DB 11 and having a similar feature amount to the query image is referred to as a similar image.

  In the image item correspondence DB 12 of the server 3, one or more items are associated with each image stored in the feature amount DB 11 based on the similarity of the feature amounts. The same item may be associated with images having similar feature amounts.

  The server 3 acquires an item associated with each of the detected N similar images from the image item correspondence DB 12. Then, the server 3 calculates the item frequency for each of the items associated with the N similar images. Here, the item frequency represents the number of similar images associated with a predetermined item in the image item correspondence DB 12. For example, the item f is a similar image A, a similar image B, and a similar image D. If they are associated with each other, the frequency of the item f is calculated as 3.

  The server 3 uses the calculated item frequency as a score, determines an item with a high score as a presentation item to be presented to the user as a result of the search process, and transmits it to the mobile phone 1. Note that the number of items to be presented is not limited to one, and some items with higher scores can be determined as the items to be presented.

  3 is a block diagram illustrating a functional configuration example of the search system of FIG. 1 that executes the first item search process described with reference to FIG. In addition, illustration of the network 2 is abbreviate | omitted in FIG.

  The cellular phone 1 includes an imaging unit 21, an operation input unit 22, a display unit 23, a communication unit 24, and a control unit 25. On the other hand, the server 3 includes a feature amount DB 11, an image item correspondence DB 12, a communication unit 31, a feature amount extraction unit 32, a similarity calculation unit 33, and a presentation item determination unit 34.

  The imaging unit 21 captures a predetermined subject based on a user operation on the operation input unit 22. The captured image is stored in a memory (not shown).

  The operation input unit 22 includes a button for starting or ending a call, a button for inputting characters or numbers, and the like. The operation input unit 22 receives a user operation and supplies operation information corresponding to the operated button to the control unit 25.

  The display unit 23 is configured by a display such as an LCD (Liquid Crystal Display), and displays predetermined characters and images. The communication unit 24 transmits / receives predetermined information to / from the communication unit 31 of the server 3. For example, the communication unit 24 transmits an image captured by the imaging unit 21 and temporarily stored in a memory (not shown) to the server 3 as a query image, and receives an item retrieved from the query image. The control unit 25 controls each unit of the mobile phone 1.

  The feature amount DB 11 of the server 3 stores feature amounts representing image features for each of a plurality of registered images registered in advance. The feature amount DB 11 also stores feature amounts representing the features of the items for many items to be searched.

  The image item correspondence DB 12 stores the registered image and the search target item in association with each other based on the similarity of the feature amount.

  FIG. 4 shows an example of the feature amount DB 11, and FIG. 5 shows an example of the image item correspondence DB 12. However, the data shown in FIGS. 4 and 5 is a part of the data stored in the feature amount DB 11 and the image item correspondence DB 12.

  For example, as shown in FIG. 4, the feature amount DB 11 stores feature amounts for images A and B as registered images and items a and b as items to be searched.

  More specifically, the ratios of the four types of moods of “cool”, “throat”, “simple”, and “warm” are set in the image or item as feature amounts and stored in the feature amount DB 11. Here, the mood is a concept or word representing sensitivity (emotion). In addition, the ratio indicates whether the image or item has a predetermined mood by “0” or “1.0”.

  Image A has a ratio of “1.0” for the mood of “cool”, a ratio of “1.0” for the mood of “throat”, and “ The ratio of “0” is set to “0” for the mood of “warm”. Similarly, image B has “0”, “1.0”, “0”, and “1.0” for the “cool”, “throat”, “rustic”, and “warm” moods, respectively. ”.

  Item a is “0”, “1.0”, “1.0”, and “0” for the “cool”, “throat”, “rustic”, and “warm” moods, respectively. The item b has a ratio of “1.0”, “1.0”, “0” for “cool”, “throat”, “rustic”, and “warm” moods, respectively. And a ratio of “0”.

  The total feature amount of each registered image or item can be expressed as a vector having each mood as a component, and this vector is referred to as a mood vector.

  On the other hand, in the image item correspondence DB 12, as shown in FIG. 5, items having similar feature quantities (mood vectors) are stored in association with the images A to D, respectively.

  Specifically, it is stored that the feature amount of the image A is similar to the item b, the item f, and the item h, and the feature amount of the image B is similar to the item e and the item f. Is remembered. Further, it is stored that the feature amount of the image C is similar to the item c and the item x, and the feature amount of the image D is similar to the item f, the item h, the item y, and the item z.

  Returning to FIG. 3, the communication unit 31 of the server 3 transmits and receives predetermined information to and from the communication unit 24 of the mobile phone 1. For example, the communication unit 31 receives a query image transmitted from the mobile phone 1. The query image acquired by receiving is supplied to the feature amount extraction unit 32. In addition, the communication unit 31 transmits the presentation item as a search result supplied from the presentation item determination unit 34 to the communication unit 24 of the mobile phone 1.

  The feature amount extraction unit 32 extracts the feature amount of the query image supplied from the communication unit 31 and supplies the mood vector obtained as a result to the similarity calculation unit 33.

  Here, as the feature amount of the image, for example, a vector or histogram representing color information, shape information, frequency component, etc. by RGB or CIELAB of the image can be adopted. 4 is learned in advance as to whether it corresponds to the mood such as “cool”, “throat”, “simple”, and “warm” shown in FIG. A mood vector can be generated based on the feature amount of the image.

  The learning of correspondence between the feature quantity of the image and the mood, for example, by taking a questionnaire on what emotion (mood) the user feels when viewing a predetermined image to a large number of users through a questionnaire, etc. The feature amount is extracted, and what kind of emotion the user has when the feature amount of the image, that is, what kind of mood should be given to the image, correspondence analysis and principal component It is determined by multivariate analysis such as analysis. As a result, it is possible to add a mood to an unknown image that has not been learned if the feature amount of the image is known.

  Further, the feature quantity extraction unit 32 extracts the feature quantity of the registered image supplied from the communication unit 31 when storing data in the feature quantity DB 11, and stores the mood vector obtained as a result in the feature quantity DB 11, The mood vector of the item supplied from the communication unit 31 is also stored in the feature amount DB 11. The mood vector of this item is given by hand, for example.

  The similarity calculation unit 33 calculates the similarity between the query image supplied from the feature amount extraction unit 32 and each registered image stored in the feature amount DB 11, and represents N registered images with high similarity. Information is supplied to the presentation item determination unit 34. For the calculation of the similarity, for example, the cosine distance or inner product of the mood vector of the query image and the mood vector of the registered image can be adopted.

  The similarity calculation unit 33 also associates the registered image with the item in the image item correspondence DB 12 based on the registered image and the mood vector of each item stored in the feature amount DB 11. More specifically, the similarity calculation unit 33 calculates the similarity between the registered image stored in the feature amount DB 11 and each item stored in the feature amount DB 11, and the similarity is equal to or greater than a predetermined value. Is stored in the image item correspondence DB 12 as an item corresponding to the registered image for all the registered images stored in the feature amount DB 11.

  The presented item determination unit 34 refers to the image item correspondence DB 12 for each of the N registered images supplied from the similarity calculation unit 33 and searches for the item corresponding to the registered image. And the presentation item determination part 34 calculates the frequency of an item as a score about each searched item. Furthermore, the presentation item determination unit 34 determines an item with a high score as a presentation item to be presented to the user, and supplies it to the communication unit 31.

  Next, the first item search process will be described with reference to the flowchart of FIG.

  If the user now tries to search for an item that matches the scenery he / she sees, first, in step S <b> 11, the imaging unit 21 of the mobile phone 1 performs a predetermined operation based on the user's operation on the operation input unit 22. Shoot the subject (landscape). It is assumed that the image captured in step S11 is a query image.

  In step S <b> 12, the communication unit 24 transmits the query image captured by the imaging unit 21 to the server 3 under the control of the control unit 25.

  In step S <b> 13, the communication unit 31 of the server 3 receives the query image transmitted from the communication unit 24 of the mobile phone 1 via the network 2. The received image is supplied to the feature amount extraction unit 32.

  In step S <b> 14, the feature amount extraction unit 32 generates a mood vector of the query image supplied from the communication unit 31 and supplies the mood vector to the similarity calculation unit 33.

  In step S <b> 15, the similarity calculation unit 33 calculates the similarity between the mood vector of the query image supplied from the feature amount extraction unit 32 and the mood vector of each registered image stored in the feature amount DB 11. Then, the similarity calculation unit 33 supplies information representing N registered images with high similarity to the presentation item determination unit 34.

  In step S <b> 16, the presentation item determination unit 34 extracts items associated with N registered images (registered images with high similarity) represented by the information supplied from the similarity calculation unit 33 from the image item correspondence DB 12. get.

  In step S17, the presentation item determination unit 34 calculates the item frequency as a score, and in step S18, determines an item with a high score as a presentation item. The determined presentation item is supplied from the presentation item determination unit 34 to the communication unit 31.

  In step S <b> 19, the communication unit 31 transmits the presentation item to the mobile phone 1 via the network 2.

  In step S <b> 20, the mobile phone 1 receives the transmitted presentation item and displays it on the display unit 23. That is, the communication unit 24 receives the presentation item transmitted from the server 3 and supplies it to the control unit 25. The control part 25 controls the display part 23, displays the item on the display part 23, and complete | finishes a process.

  As described above, according to the first item search process, the user's desired item is searched using the query image. Therefore, the user can input the desired item to the user without inputting a predetermined character (word). Can be presented. Therefore, even if it is difficult to verbalize desired information, the user can easily search.

  In the above-described example, the item frequency is used as a score as it is, but it is also possible to obtain the item score by other calculations. For example, 10 items are assigned to an item linked to a registered image with the first rank of similarity, and 6 points are assigned to an item linked to a registered image with the second rank of similarity. The item associated with the registered image of the third place may be determined by a rank-weighted score such that the item score is given a weight according to the similarity rank, such as 4 points. Further, the presentation item may be determined by calculating a weighted score of the item with the similarity value itself as a weight.

  Next, an outline of the second item search process by the search system of FIG. 1 will be described with reference to FIG.

  The mobile phone 1 transmits the query image to the server 3. The server 3 receives the query image transmitted from the mobile phone 1 and extracts the feature amount of the received query image. Further, the server 3 detects N similar images (image groups) whose image feature amounts are similar to those of the query image from among a large number of registered images stored in the feature amount DB 11. The process so far is the same as the first item search process described above.

  As shown in FIG. 7, the mapping information DB 51 of the server 3 stores mapping information in which registered images and items stored in the feature amount DB 11 are mapped on the same space R.

  The server 3 determines a presentation item from items whose distances in the space R of the detected N similar images are close. More specifically, the server 3 detects, for each of the detected N similar images, a plurality of items that are close in space (the similar image and the distance in space are within a predetermined range). Then, the server 3 calculates the item frequency for all detected items as a score, determines an item with a high score as a presentation item, and transmits the item to the mobile phone 1.

  FIG. 8 is a block diagram illustrating a functional configuration example of the search system in FIG. 1 that executes the second item search process described with reference to FIG. 7. In FIG. 8, parts corresponding to those in FIG. 3 are denoted by the same reference numerals, and redundant description is omitted as appropriate. Further, the illustration of the network 2 is omitted as in FIG.

  8 is different from FIG. 3 in that a mapping information DB 51 is provided instead of the image item correspondence DB 12 and a presentation item determination unit 52 is provided instead of the presentation item determination unit 34.

  As described above, the mapping information DB 51 stores mapping information in which registered images and items stored in the feature amount DB 11 are mapped on the same space R. This mapping information can be created manually, or can be created by the similarity calculation unit 33 using correspondence analysis or the like.

  When creating mapping information manually, a space and a dimension are defined in advance, and the registered image of the feature amount DB 71 and the position (coordinates) of the item determined by hand are stored as mapping information in the space. The

  On the other hand, when using correspondence analysis, a set of items (a plurality of items) similar to a registered image is manually created in advance, and the similarity calculation unit 33 performs correspondence analysis using the cross tabulation result. Thus, the registered image and the item can be mapped on the same multidimensional space.

  The similarity calculation unit 33 calculates the similarity between the query image supplied from the feature amount extraction unit 32 and each registered image stored in the feature amount DB 11, and represents N registered images with high similarity. Information is supplied to the presentation item determination unit 52.

  The presentation item determination unit 52 refers to the mapping information DB 51 for each of the N registered images represented by the information supplied from the similarity calculation unit 33, and selects a plurality of items whose distances in the space R are close to the registered image. To detect. And the presentation item determination part 52 calculates the frequency of the item similar to a 1st item search process as a score about all the detected items, determines an item with a high score as a presentation item, The communication part 31 Supply.

  In the present embodiment, information about a plurality of similar images is supplied from the similarity calculation unit 33 to the presentation item determination unit 52. However, the number of similar images supplied from the similarity calculation unit 33 If the number is one, the item that is closest in space to the one similar image may be the presentation item.

  The second item search process will be described with reference to the flowchart of FIG.

  The processing in steps S31 to S35 in FIG. 9 is the same as the processing in steps S11 to S15 in FIG.

  In step S36, the presentation item determination unit 52 refers to the mapping information DB 51, and the N registration images (registered images with high similarity) represented by the information supplied from the similarity calculation unit 33 and the spatial distance are determined. Detect multiple items near you.

  In step S <b> 37, the presented item determination unit 52 calculates the item frequency as a score for all detected items, and in step S <b> 38, determines an item with a high score as a presented item.

  In steps S39 and S40, as in steps S19 and S20 described above, the presentation item is transmitted to the mobile phone 1 by the communication unit 31, and the presentation item is displayed on the display unit 23 in the mobile phone 1 that has received the presentation item. Then, the process ends.

  As described above, also in the second item search process, the user can present a desired item to the user without inputting a predetermined character (word). Therefore, even if it is difficult to verbalize desired information, the user can easily search.

  In the second item search process, the score of each item can be obtained by a value other than the item frequency described above.

  For example, until each item on the space R is determined, the similarity between the query image and the similar image (referred to as similarity I), and the similar image and the item represented by the distance on the space R Two similarities of similarity (referred to as similarity II) are calculated, and the score of each item is calculated as the sum of the products of similarity I and similarity II (sum of items) You can also.

  Next, with reference to FIG. 10, an outline of the third item search process by the search system of FIG. 1 will be described.

  The third item search process is a search process in the case where characters (words) are additionally input in addition to images.

  That is, it is assumed that the user specifies (inputs) “clothes” as a word related to a desired item (hereinafter referred to as a query word) together with the query image in the mobile phone 1. The mobile phone 1 transmits the designated query image and the query word “clothes” to the server 3 via the network 2.

  As shown in FIG. 10, the feature vector DB 71 of the server 3 stores a mood vector and a keyword describing the item in advance for a predetermined item.

  First, similarly to the two embodiments described above, the server 3 receives the query image transmitted from the mobile phone 1 and extracts the feature amount of the received query image. Further, the server 3 searches for an item stored in the feature amount DB 71 based on the query word transmitted from the mobile phone 1. Thereby, a plurality of items (item group) having the same keyword as the query word are detected from the feature amount DB 71.

  Next, the server 3 calculates the similarity with the query image for each item having the query word as a keyword. More specifically, the similarity between the mood vector set for the item and the mood vector generated from the feature amount extracted from the query image is calculated. As a result, an item with a high degree of similarity is determined as a presentation item and transmitted to the mobile phone 1.

  FIG. 11 is a block diagram illustrating a functional configuration example of the search system of FIG. 1 that executes the third item search process described with reference to FIG. In FIG. 11, parts corresponding to those in FIG. 3 are denoted by the same reference numerals, and redundant description will be omitted as appropriate. Further, the illustration of the network 2 is omitted as in FIG.

  In FIG. 11, the feature amount DB 71, the search unit 72, and the presentation item determination unit 73 are provided in place of the feature amount DB 11 and the presentation item determination unit 34, and the point that the image item correspondence DB 12 is omitted. 3 is different from FIG.

  The communication unit 31 receives the query image and the query word transmitted from the communication unit 24 of the mobile phone 1, and supplies the query image to the feature amount extraction unit 32 and the query word to the search unit 72.

  As described above, the feature amount extraction unit 32 extracts the feature amount of the query image, and supplies the mood vector obtained as a result to the similarity calculation unit 33.

  The feature DB 71 stores mood vectors for various registered images and items. As shown in FIG. 12, the feature amount DB 71 is different from the feature amount DB 11 of FIG. 3 in that not only the mood vector but also the keyword is stored in association with each item.

  In the example of FIG. 12, the item a includes mood vectors of “cool”, “throat”, “rustic”, and “warm” (“0”, “1.0”, “1.0”, and “0”). )) And keywords “clothes” and “for women” are stored. Item b includes “cool”, “throat”, “rustic”, and “warm” mood vectors (“1.0”, “1.0”, “0”, and “0”), The keywords “clothes” and “for men” are stored.

  The search unit 72 in FIG. 11 checks the keyword for each item in the feature amount DB 71 (FIG. 12), detects all items having the query word supplied from the communication unit 31 as a keyword, and the mood vector of the detected item Is supplied to the similarity calculator 33.

  The similarity calculation unit 33 calculates the similarity between the mood vector of the item supplied from the search unit 72 and the mood vector of the query image supplied from the feature amount extraction unit 32, and presents the calculation result as a presentation item determination unit. 73.

  The presentation item determination unit 73 determines a presentation item based on the similarity between each item (item having a query word) supplied from the similarity calculation unit 33 and the query image, and supplies the determined presentation item to the communication unit 31. That is, a predetermined number of items having a high degree of similarity are supplied to the communication unit 31 as presentation items.

  FIG. 13 is a flowchart for explaining the third item search process.

  First, in step S51, the operation input unit 22 of the mobile phone 1 accepts an input of a query word based on a user operation. In step S52, the imaging unit 21 captures a predetermined subject.

  In step S <b> 53, the communication unit 24 transmits the query word and the query image to the server 3 under the control of the control unit 25.

  In step S <b> 54, the communication unit 31 of the server 3 receives the query word and the query image transmitted from the communication unit 24 of the mobile phone 1 via the network 2, and sends the query image to the feature amount extraction unit 32. Are respectively supplied to the search unit 72.

  In step S <b> 55, the feature amount extraction unit 32 generates a mood vector of the query image supplied from the communication unit 31 and supplies the mood vector to the similarity calculation unit 33.

  In step S <b> 56, the search unit 72 searches the feature amount DB 71 for items having the query word supplied from the communication unit 31. The search unit 72 supplies the mood vector of the item obtained as a result of the search to the similarity calculation unit 33.

  In step S <b> 57, the similarity calculation unit 33 calculates the similarity between the mood vector of the item supplied from the search unit 72 and the mood vector of the query image supplied from the feature amount extraction unit 32.

  In step S <b> 58, the presentation item determination unit 73 determines a presentation item based on the similarity between each item supplied from the similarity calculation unit 33 and the query image. That is, the presentation item determination unit 73 supplies a predetermined number of items with high similarity to the communication unit 31 as presentation items.

  In step S <b> 59, the communication unit 31 transmits the presentation item to the mobile phone 1 via the network 2.

  In step S <b> 60, the mobile phone 1 receives the transmitted presentation item and displays it on the display unit 23. That is, the communication unit 24 receives the presentation item transmitted from the server 3 and supplies it to the control unit 25. The control part 25 controls the display part 23, displays the item on the display part 23, and complete | finishes a process.

  As described above, according to the third item search process, the search target items can be narrowed down using the query word, and a desired item corresponding to the query image can be presented to the user. Therefore, even if it is difficult to verbalize desired information, the user can easily search.

  Next, another third item search process will be described with reference to the flowchart of FIG. In the above-mentioned example, the feature amount (mood vector) obtained from the entire query image is compared with the feature amount (mood vector) of the item stored in the feature amount DB 71. Is used to designate a point of interest (region of interest) of particular interest, and to search for items based on the feature amount of the point of interest.

  First, in step S71, the operation input unit 22 of the mobile phone 1 receives an input of a query word based on a user operation, and in step S72, the imaging unit 21 captures a predetermined subject.

  In step S <b> 73, the display unit 23 displays the captured query image under the control of the control unit 25. And the operation input part 22 receives the input of the attention point with respect to the displayed query image based on a user's operation. Information about the point of interest received by the operation input unit 22 (hereinafter referred to as point-of-interest information) is supplied to the control unit 25.

  In step S <b> 74, the communication unit 24 transmits the query word, the query image, and the point of interest information to the server 3 under the control of the control unit 25.

  In step S75, the communication unit 31 of the server 3 receives the query word, the query image, and the point of interest information transmitted from the communication unit 24 of the mobile phone 1 via the network 2, and obtains the query image and the point of interest information. The query terms are supplied to the feature amount extraction unit 32 and the search unit 72, respectively.

  In step S76, the feature amount extraction unit 32 generates a mood vector of a predetermined region (hereinafter, appropriately referred to as a query partial image) specified by the point-of-interest information of the query image supplied from the communication unit 31, It supplies to the similarity calculation part 33. The predetermined area specified by the point-of-interest information can be, for example, a predetermined square area centered on the point of interest specified by the user.

  In step S <b> 77, the search unit 72 searches the feature amount DB 71 for an item having the query word supplied from the communication unit 31. The search unit 72 supplies the mood vector of the item obtained as a result of the search to the similarity calculation unit 33.

  In step S78, the similarity calculation unit 33 calculates the similarity between the mood vector of the item supplied from the search unit 72 and the mood vector of the query partial image supplied from the feature amount extraction unit 32.

  The processing in steps S79 to S81 is the same as the processing in steps S58 to S60 in FIG.

  According to the third item search process, for example, among the “field” and “river” captured in the query image shown in FIG. 10, the “field” region represents the mood of “Nodoka”, Assuming that the “river” region represents the “cool” mood, when the user sets a point of interest in the “river” portion, only the “cool” mood is extracted by the feature amount extraction unit 32. The Therefore, it is possible to generate a mood vector that emphasizes the mood of a predetermined region in the query image. As a result, it is possible to more faithfully specify the part that the user cannot express well in the language for item search.

  In the example described above, the query image is described as an example of a landscape image taken by the user on the spot, but may be an image acquired from the Internet or the like.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 15 is a block diagram illustrating a configuration example of hardware of a computer that executes the above-described series of processing by a program.

  In a computer, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 are connected to each other via a bus 104.

  An input / output interface 105 is further connected to the bus 104. The input / output interface 105 includes an input unit 106 including a keyboard, a mouse, and a microphone, an output unit 107 including a display and a speaker, a storage unit 108 including a hard disk and nonvolatile memory, and a communication unit 109 including a network interface. A drive 110 for driving a removable medium 121 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is connected.

  In the computer configured as described above, the CPU 101 loads, for example, the program stored in the storage unit 108 to the RAM 103 via the input / output interface 105 and the bus 104 and executes the program. Is performed.

  The program executed by the computer (CPU 101) is, for example, a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, or a semiconductor. The program is recorded on a removable medium 121 that is a package medium including a memory or the like, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  The program can be installed in the storage unit 108 via the input / output interface 105 by attaching the removable medium 121 to the drive 110. Further, the program can be received by the communication unit 109 via a wired or wireless transmission medium and installed in the storage unit 108. In addition, the program can be installed in the ROM 102 or the storage unit 108 in advance.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  In the above-described embodiment, the process of specifying the query image and the process of searching for the user's desired information based on the query image are performed by different devices, that is, the mobile phone 1 and the server 3. These two processes may be performed only by either the mobile phone 1 or the server 3 (device). For example, the mobile phone 1 can also perform first to third item search processing for searching for information desired by the user based on a designated query image. In this case, the feature amount DB 11 and the image item correspondence DB 12 correspond to, for example, a recording medium such as a semiconductor memory that is built in the mobile phone 1 or that can be attached to and detached from a drive (not shown).

  In this specification, the steps described in the flowcharts include processes that are executed in parallel or individually even if they are not necessarily processed in time series, as well as processes that are executed in time series in the described order. Is also included.

  In the present specification, the term “system” represents the entire apparatus constituted by a plurality of apparatuses.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

It is a figure which shows the structural example of one Embodiment of the search system to which this invention is applied. It is a figure explaining the outline | summary of a 1st item search process. It is a block diagram which shows the functional structural example of 1st Embodiment of the search system of FIG. It is a figure which shows the example of feature-value DB of 1st Embodiment. It is a figure which shows the example of image item corresponding | compatible DB of 1st Embodiment. It is a flowchart explaining a 1st item search process. It is a figure explaining the outline | summary of a 2nd item search process. It is a block diagram which shows the functional structural example of 2nd Embodiment of the search system of FIG. It is a flowchart explaining a 2nd item search process. It is a figure explaining the outline | summary of a 3rd item search process. It is a block diagram which shows the functional structural example of 3rd Embodiment of the search system of FIG. It is a figure which shows the example of feature-value DB of 3rd Embodiment. It is a flowchart explaining a 3rd item search process. It is a flowchart explaining the other 3rd item search process. It is a block diagram which shows the structural example of one Embodiment of the computer to which this invention is applied.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Mobile phone, 3 Server, 11 Feature-value DB, 12 Image item corresponding | compatible DB, 31 Communication part, 32 Feature-value extraction part, 33 Similarity degree calculation part, 34 Presentation item determination part, 51 Mapping information DB, 52 Presentation item determination part , 71 feature DB, 72 search unit, 73 presentation item determination unit

Claims (5)

Image feature amount storage means for storing feature amounts representing image features for a plurality of registered images registered in advance;
Item correspondence storage means for storing correspondence between the registered image and the predetermined item as information obtained by mapping the registered image and the predetermined item on the same space;
Obtaining means for obtaining a query image, which is an image designated by a user in order to search for a desired item;
Feature quantity extraction means for extracting the feature quantity of the query image;
Similarity calculation means for calculating a first similarity between the query image and the registered image using the extracted feature quantity;
Information comprising: a presentation item determining unit that determines, from the item correspondence storage unit, an item that is close to the first registered image with the first similarity as the presentation item that is the item to be presented to the user. Processing equipment. The similarity calculation means also calculates a second similarity that is a similarity of the feature quantity between the registered image and the item,
The presentation item determination means, wherein the first degree of similarity based on the second value obtained by adding the product of similarity, the information processing apparatus according to claim 1 for determining the presentation item. The presentation item determination means according to claim 1, the distance on the said first similarity highly plurality of the registration image space calculates the frequency of the near item, the frequent items is determined as the presentation item The information processing apparatus described in 1. Image feature amount storage means for storing feature amounts representing image features for a plurality of registered images registered in advance, and information obtained by mapping the registered image and a predetermined item on the same space , In an information processing method of an information processing apparatus that includes an item correspondence storage unit that stores correspondence with a predetermined item, and determines a presentation item that is the item presented to a user
Obtaining a query image which is an image specified by the user to search for a desired item;
Extracting feature quantities of the query image;
Using the extracted feature amount, the similarity between the query image and the registered image is calculated,
An information processing method including a step of determining, as the presentation item, an item whose distance in the space is close to the registered image having a high similarity from the item correspondence storage unit. Image feature amount storage means for storing feature amounts representing image features for a plurality of registered images registered in advance, and information obtained by mapping the registered image and a predetermined item on the same space , In a program for causing a computer to execute a process of determining a presentation item that is the item to be presented to a user using an item correspondence storage unit that stores a correspondence with a predetermined item.
Obtaining a query image which is an image specified by the user to search for a desired item;
Extracting feature quantities of the query image;
Using the extracted feature amount, the similarity between the query image and the registered image is calculated,
The program which makes a computer perform the process including the step which determines the item with the near distance in the said registration image with the said high similarity as the said presentation item from the said item corresponding | compatible storage means.

Images