JP4054787B2 - Data retrieval device - Google Patents

Description

  The present invention relates to an apparatus for searching for data using annotation information associated with data such as images and documents.

  With the spread of digital cameras and the like, the amount of images owned by individuals is increasing. As the number of images increases, it becomes difficult to find a desired image. Therefore, an apparatus for efficiently retrieving a desired image from a large number of images is required.

  Conventionally, a search method using voice recognition has been proposed. In Patent Document 1, a user first adds audio data to an image. For example, voice data “Fujisan” is added to an image taken at Mt. Fuji. This can be achieved with a voice recording function such as a digital camera. Next, the voice data is voice-recognized, and the result is associated with the image as a voice recognition result annotation. In the previous example, a character string such as “Mt. Fuji (Kana Kanji)”, “Fujisan (Reading)”, “Fujisan (phoneme string)”, or the like is associated with the image taken at Mt. Fuji as a speech recognition result annotation. The search is performed by inputting a character string as a search condition and comparing the speech recognition result annotation with the character string of the search condition.

  A method of searching using time information has been proposed. In Patent Document 2, time information is used as one of the conditions for searching for an image similar to a key image. This utilizes the fact that similar images have similar time information. For example, images taken at a certain athletic meet all have the same date.

In addition, a method of searching by devising the folder display has been proposed. In Patent Document 3, a plurality of images are selected from images stored in a folder and displayed as a representative image of the folder. The user can guess the image stored in the folder to some extent by looking at the representative image of the folder.
JP 2003-219327 A JP 2000-276484 A JP 2000-76281 A

  In Patent Document 1, it is not possible to search for an image that is not associated with a speech recognition result annotation.

  Further, Patent Document 2 has a problem that images that are separated in time cannot be searched simultaneously. For example, when searching for images of athletic meet held every year, images of athletic events held in the same year as the selected key image (time information is close) are searched, and images of athletic events of other years are searched. Not.

  Patent Document 3 has a problem that the representative image of a folder is always constant. When searching for photos of Mt. Fuji and searching for family photos, the representative images are the same, and the search efficiency is reduced.

In order to solve the above problems, the data search method executed by the data search apparatus for searching for data according to the present invention has a search unit storing a plurality of folders, and annotation information is associated with each folder. A search process for searching for data using a received search condition and a similarity calculated based on the annotation information from a storage unit capable of storing the data that is not associated with the annotation data, and display control means Comprises a display control step for controlling a plurality of data searched in the search step to be displayed on a display device based on the similarity , and the display control step includes the data included in each folder. The rows on the display of each folder are determined in the order of similarity of the data with the highest similarity, and the data searched in the search step, A plurality of data stored to one folder, a row of the determined the folder controls to display in the order of the similarity, wherein the display control step further wherein the annotation information is not associated with the search Data that was not searched in the process and stored in the same folder as the data searched in the search process was searched in the search process on the same line as the data searched in the search process Control is performed so that the data is identified and displayed on the display device.

  As described above, the data retrieval apparatus of the present invention can retrieve data that is not associated with annotation information.

  Hereinafter, a preferred embodiment of a data search method according to the present invention will be described with reference to the drawings.

(Embodiment 1)
FIG. 1 is a block diagram illustrating the configuration of a data search apparatus according to an embodiment of the present invention. In this embodiment, a case where an image is handled as an example of data and a speech recognition result annotation is used as an example of annotation information will be described. Reference numeral 101 denotes an image database that stores images to be searched. The image database may be placed in the data retrieval apparatus, or may be used in another apparatus connected by a network. Reference numeral 102 denotes search means for searching for an image based on the speech recognition result annotation and the input search condition. Reference numeral 103 denotes display control means for identifying and displaying an image that is not associated with a speech recognition result annotation and that is related to the searched image, from the searched image.

  FIG. 4 is a diagram for explaining the image database 101. 401 is an image. Reference numeral 402 denotes audio data associated with an image. Reference numeral 403 denotes a speech recognition result annotation associated with an image.

  As shown in the figure, some images are not associated with images associated with speech recognition result annotations. Each image holds time information such as the date when the image was created. The management structure may be anything, a structure in which all images are stored in one folder, or a folder hierarchical structure.

  FIG. 2 is a flowchart for explaining the processing procedure of the data search apparatus according to the embodiment of the present invention.

  In S201, a search character string serving as a search condition is input. This search character string may be read or converted into a phoneme string using language processing technology. Whether the input search character string is used as it is or whether it is converted into a reading or phoneme string is determined according to the character string specification of the speech recognition result annotation. That is, if the speech recognition result annotation uses reading, it is converted to reading, and if it uses a phoneme string, it is converted to a phoneme string.

  In S202, one image associated with the speech recognition result annotation is extracted from the image database. In S203, the similarity between the speech recognition result annotation associated with the image extracted in S202 and the search character string is calculated. This similarity calculation uses a technique such as DP matching. In S204, it is determined whether or not all the images associated with the speech recognition result annotation have been processed. If all the images have been processed, the process proceeds to S205, and if any unprocessed images remain, the process returns to S202.

  In S205, the top N images from the ones with high similarity calculated in S203 are set as searched images.

  In S206, one image that is not associated with the speech recognition result annotation is extracted from the image database. In S207, the image extracted in S206 is associated with all the searched images having time information within ± T hours of the time information of the image extracted in S206. In S208, it is determined whether or not all images not associated with the speech recognition result annotation have been processed. If all the images have been processed, the process proceeds to S209, and if any unprocessed images remain, the process returns to S206.

  In S209, the searched images are displayed in order of similarity. An example is shown in FIG. In S210, it is determined whether there is an instruction to end the search. If there is an instruction, the process ends. If there is no instruction, the process proceeds to S211. For example, an operation for closing a window displaying the searched image is set as an instruction to end the search.

  In S211, it is determined whether there is an instruction to display an image related to the searched image. If there is an instruction, the process proceeds to S212, and if there is no instruction, the process returns to S210. For example, an operation for selecting the searched image is set as a display instruction. In S212, an image related to the searched image is displayed. For example, when an image with the third similarity is selected, an image associated with the image with the third similarity is displayed in another window in S207. An example is shown in FIG.

(Embodiment 2)
FIG. 3 is a flowchart for explaining the processing procedure of the data search apparatus according to the embodiment of the present invention. S201 to S205 perform the same processing as S201 to S205 in FIG.

  In S301, 0 is substituted into the variable i. In S302, 1 is added to the value of the variable i.

  In S303, it is determined whether the variable i is N or less. If N or less, the process proceeds to S304, and if greater than N, the process ends.

  In S304, it is determined whether or not an image with the i-th similarity is already displayed. If already displayed, the process returns to S302, and if not displayed, the process proceeds to S305. In S305, the image with the i-th similarity is displayed at the left end of the new line, and the displayed mark is added to the image with the i-th similarity. In S306, the value of variable i is substituted into variable j. In S307, 1 is added to the value of the variable j. In S308, it is determined whether the variable j is N or less. If N or less, the process proceeds to S309, and if greater than N, the process proceeds to S312.

  In S309, it is determined whether or not the image with the j-th similarity is stored in the same folder as the image with the i-th similarity. If it is stored in the same folder, the process proceeds to S310, and if it is stored in a different folder, the process returns to S307.

  In S310, it is determined whether or not the image with the j-th similarity has been displayed. If already displayed, the process returns to S307, and if not displayed, the process proceeds to S311. In step S311, the image with the j-th similarity is displayed on the same line as the image with the i-th left-justification, and the displayed mark is added to the image with the j-th similarity.

  In S312, an image that is not associated with a speech recognition result annotation and that has a similarity in the same folder as the i-th image is appropriately selected, and the similarity is in the same row as the i-th image. Display left-justified.

  Here, the selection of an image that is not associated with the voice recognition result annotation may be selected randomly from a folder, or may be selected so that time information is equally spaced.

  In addition, the number of images that are not associated with the voice recognition result annotation may be a fixed number, the sum of the searched images may be a fixed number, or in proportion to the number of images included in the folder. Also good.

  Also, an image that is not associated with a speech recognition result annotation, such as changing the size of the image to be displayed or changing the color of the image frame, is displayed so that it can be identified from the searched image.

  FIG. 7 is a diagram illustrating an example of an image displayed by the process of FIG. Images displayed in the same row are images stored in the same folder. The numbers in the figure indicate the rank of similarity, and images with the same folder are displayed on the left side as the similarity is higher. Further, the sum of the number of retrieved images and the number of images not associated with the speech recognition result annotation is displayed so as to be constant.

(Embodiment 3)
In S207, the image extracted in S206 may be associated with the searched image having the time information closest to the time information of the image extracted in S206.

(Embodiment 4)
In S207, the image extracted in S206 may be associated with the searched image having time information within ± T hours of the time information of the image extracted in S206 and having the earliest time information.

(Embodiment 5)
In S207, a shooting condition may be added to the image, and the image extracted in S206 may be associated with the searched image having the time information within ± T hours of the time information of the image extracted in S206 and having the closest shooting condition. .

(Embodiment 6)
In S206 to S208, the image that is not associated with the speech recognition result annotation is associated with the searched image. In advance, using the time information, the image that is not associated with the speech recognition result annotation is associated with the speech recognition result annotation. The image may be associated with the obtained image.

  In this case, the processing of S206 to S208 can be omitted during the search. In addition, the same method as S207 can be used as a method of associating using time information.

(Embodiment 7)
In the above embodiment, the image has been described. However, the present invention is not limited to this. For example, it is also possible to add audio data to document data read by a copying machine or a scanner.

(Other embodiments)
The present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. A recording medium recording software program codes for realizing the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus stores program codes stored in the recording medium. Needless to say, this can also be achieved by executing read.

  In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  As a recording medium for supplying the program code, for example, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like is used. be able to.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS running on the computer performs actual processing based on an instruction of the program code. Needless to say, a case where the function of the above-described embodiment is realized by performing part or all of the processing, is also included.

  Further, after the program code read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

It is a block diagram explaining the structure of the data search device concerning one Example of this invention. It is a flowchart explaining the process sequence of the data search device concerning one Example of this invention. It is a flowchart explaining the process sequence of the data search device concerning one Example of this invention. It is a figure explaining the image database. It is an example of a display of the searched image. This is a display example of data related to a searched image, which is data not associated with a speech recognition result annotation. This is a display example of data that is not associated with a voice recognition result annotation and is stored in the same folder as the searched image.

Explanation of symbols

101 Image database 102 Search means 103 Display control means

Claims (4)

A data search method executed by a data search device for searching data,
Based on the search condition and the annotation information received from the storage means capable of storing the data in which the annotation information is associated with each folder and the data in which the annotation information is not associated with each folder. A search process for searching for data using the calculated similarity ,
A display control step, wherein the display control means controls to display a plurality of data searched in the search step on a display device based on the similarity ,
The display control step is the data searched in the search step by determining a row on the display of each folder in the order of similarity of the data having the highest similarity among the data included in each folder. A plurality of data stored in the same folder are controlled to be displayed in the order of the similarity in the determined row of the folder;
The display control step further includes the search for data stored in the same folder as the data searched in the search step that is not associated with the annotation information and was not searched in the search step . A data search method characterized by controlling to display on the display device by identifying the data searched in the search step in the same line as the data searched in the step. A control program for causing a data search apparatus for searching data to execute the data search method according to claim 1. A computer-readable storage medium storing the control program according to claim 2 . A data retrieval device for retrieving data,
Similarities calculated based on the received search conditions and the annotation information from storage means capable of storing data in which a plurality of folders are stored and annotation information is associated with each folder and data in which annotation information is not associated A search means for searching for data using
Display control means for controlling to display a plurality of data searched by the search means on a display device based on the similarity ,
The display control means determines the rows on the display of each folder in the order of similarity of the data having the highest similarity among the data included in each folder, and is the data searched by the search means. A plurality of data stored in the same folder are controlled to be displayed in the order of the similarity in the determined row of the folder;
The display control means is further configured to search the data stored in the same folder as the data searched by the search means that is not associated with the annotation information and searched by the search means . A data search method characterized in that control is performed so that the data searched by the search means is identified and displayed on the display device in the same line as the data searched by the means.

Images