JP6752122B2 - Image processing device, image processing method - Google Patents

Description

The present invention relates to an image retrieval technique.

A search device has been proposed in which an object is detected from images taken by a plurality of surveillance cameras installed in a surveillance area, an image of the detected object is registered in a database, and an image of a specific object is searched from the database. In Patent Document 1, a face image is extracted from an image taken by a surveillance camera to generate a feature amount of the face image, a search process is executed using the feature amount of the face image specified by the user from the image, and similar features are obtained. A surveillance video search device that searches for a large amount of facial images has been proposed.

Japanese Patent No. 5227911

However, in the surveillance video search device of Patent Document 1, in order to execute the search, it is necessary to specify the face image of the target to be searched from the video. Therefore, it is troublesome for the user to search the video for the face of the target to be searched.

The present invention has been made in view of such a problem, and provides a technique capable of specifying a query used for an image search by a simpler method than before.

The uniformity of the present invention is a determination means for determining whether or not an object image belongs to the same class as a plurality of reference object images, and when it is determined that the determination means belongs, the object image is used as a query candidate image. When the object image is registered and it is determined that the determination means does not belong, the registration means for registering the object image as a non-query candidate image is provided.

According to the configuration of the present invention, the query used for the image search can be specified by a simpler method than before.

The block diagram which shows the functional configuration example of an image processing apparatus. The figure which shows an example of the input image. The figure explaining the model. A flowchart showing the operation of the image processing device. The figure which shows an example of the input image. The figure explaining the model. The block diagram which shows the functional configuration example of an image processing apparatus. The figure explaining the operation of the position estimation part 721. The flowchart of the object image and the registration process of the local feature amount of the object image. The block diagram which shows the functional configuration example of an image processing apparatus. A flowchart showing the operation of the image processing device. A block diagram showing a hardware configuration example of a computer device.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In addition, the embodiment described below shows an example when the present invention is concretely implemented, and is one of the specific examples of the configuration described in the claims.

[First Embodiment]
First, an example of the functional configuration of the image processing apparatus according to the present embodiment will be described with reference to the block diagram of FIG. In each subsequent embodiment including this embodiment, the configuration for searching for a person will be described, but if the search target is an object other than a person, this embodiment will be described instead of the person in the following description. You can adopt the object.

The image acquisition unit 101 acquires an input image and sends the acquired input image to the extraction unit 102 in the subsequent stage. The acquisition form of the input image by the image acquisition unit 101 is not limited to a specific acquisition form. For example, the image acquisition unit 101 may acquire an image of each frame captured by the surveillance camera and sequentially output from the surveillance camera as an input image, or sequentially input a plurality of images stored in the storage device. It may be obtained from the storage device.

The extraction unit 102 extracts an area in which an object such as a person is reflected from the input image received from the image acquisition unit 101 as an object area, and uses the image in the extracted object area as an object image for the evaluation unit 103 in the subsequent stage. And output. That is, what is shown in the object image may be an object other than a person, such as a car or a dog. It should be noted that even if only a person is targeted, an area in which the person appears is extracted from the input image as an object area, and the image in the extracted object area is output as an object image to the evaluation unit 103 in the subsequent stage. good.

An example of the input image is shown in FIG. When the input image 201 shown in FIG. 2 is input from the image acquisition unit 101 to the extraction unit 102, the extraction unit 102 outputs the object image 202 in the input image 201 to the evaluation unit 103 in the subsequent stage. There are various methods for acquiring an object image from an input image, and any method may be adopted in the present embodiment. For example, among each pixel in the input image, a method is adopted in which a pixel having a pixel value different from that of the corresponding pixel position pixel in the background image prepared in advance is specified as a target pixel, and an area composed of the target pixels is set as an object area. Is also good. As another method, an object detection method can also be used. For example, the object detection process can be executed by using the method described in Document 1 below, and the image in the detected region can be used as an object image.

(Reference 1) Histograms of orientationed gradients for human decitation, N. et al. Dalal, CVPR2005
The evaluation unit 103 performs the processing as described below for each object image input from the extraction unit 102, and calculates the evaluation value of the object image. First, the evaluation unit 103 extracts a local feature amount from the object image. The local feature amount extracted from the object image is not limited to a specific feature amount, and for example, a HOG feature amount as disclosed in Document 1 above and a SIFT feature amount as disclosed in Document 2 below are used. It may be extracted from the object image.

(Reference 2) David G Lowe, "Distinctive Image Features from Sclae-Invariant Keypoints", International Journal of Computer Vision 60 (2): 91-110.
Next, the evaluation unit 103 applies the model created in advance to the local features of the object image, and the local features of the object image are in the same class (classification) as the local features of the human images of various persons prepared in advance. ), And the evaluation value of the object image is obtained according to the judgment result. This model will be described with reference to FIG. The model used here is created through the following process. First, the result of clustering the local feature amounts of the person image 303 of various people (for example, the person to be searched) prepared in advance by GMM (Gaussian Mixture Model) is created as a histogram 304. Then, the histogram 304 is learned by SVM (Support Vector Machine), and the result of learning the boundary 306 of the class is used as the model 305. In the case of FIG. 3, the boundary 306 of the class represents the boundary between the class to which the histogram 304 of the local feature amount of the person image 303 of various persons belongs and the class other than the class. In FIG. 3, the histograms of the local features of each of the plurality of person images 303 are indicated by ◯, and the histograms of the local features belonging to the class to which ◯ does not belong are indicated by ×.

When calculating the evaluation value of the object image 202 using such a model 305, first, the local feature amount of the object image 202 is clustered by GMM, and the histogram 302 is created from the clustering result. Then, the model 305 is applied to the histogram 302 (indicated by Δ in FIG. 3), and the histogram 302 is a “class to which the histogram of the local features of the human image 303 of various people belongs” with reference to the boundary 306 ( Judge whether or not it belongs to the class to which ○ belongs. When the histogram 302 belongs to the "class to which the histogram of the local features of the human image 303 of various people belongs" (the class to which ○ belongs) with reference to the boundary 306, the evaluation value of the object image 202 is "1". "Is set. On the other hand, when the histogram 302 does not belong to the "class to which the histogram of the local features of the human image 303 of various people belongs" with respect to the boundary 306 (belongs to the class to which x belongs), the evaluation value of the object image 202 Is set to "-1". In FIG. 3, since the histogram 302 belongs to the “class to which the histogram of the local feature amount of the human image 303 of various persons belongs” (the class to which ○ belongs), “1” is set as the evaluation value of the object image 202. Will be done.

Then, the evaluation unit 103 registers the object image in which "1" is set as the evaluation value and the local feature amount of the object image in the registration DB (database) 111 (first storage unit) and the search DB 112 (second storage unit). ) And register. On the other hand, the evaluation unit 103 registers the object image in which "-1" is set as the evaluation value and the local feature amount of the object image in the search DB 112 without registering in the registration DB 111.

It should be noted that the object image of the class to which ○ belongs and the local feature amount of the object image are registered in the registration DB 111 and the search DB 112, and the object image of the class to which × belongs and the local feature amount of the object image are registered in the search DB 112. For example, the configuration of the evaluation unit 103 is not limited to the above configuration.

In this way, for each input image, the object image in the input image and the local feature amount of the object image are registered in the registration DB 111 and the search DB 112 or registered only in the search DB 112 according to the evaluation value of the object image. To do.

The selection unit 104 selects an object image to be used as a query from the object images registered in the registration DB 111 as a query image. Various configurations can be considered as the configuration for selecting the query image from the object images registered in the registration DB 111. For example, the selection unit 104 displays a list of each object image registered in the registration DB 111 on the display unit 113 (display control), and receives a selection instruction from the user. Then, when the user operates an operation unit (not shown) to perform a selection operation of selecting an object image as a query image from the displayed object images, the selection unit 104 sets the object image selected by the user and the object image selected by the user. The local feature amount of the object image and the local feature amount are acquired from the registration DB 111. The number of query images to be selected (acquired) may be one or a plurality.

The search unit 105 searches the search DB 112 for an object image having a local feature amount similar to the local feature amount of the query image for each query image acquired by the selection unit 104 from the registration DB 111. Since the search processing performed for each query image is the same processing, the search processing for the query image A will be described below as an example.

First, the search unit 105 obtains the degree of similarity between the local feature amount A of the query image A and each local feature amount B registered in the search DB 112. The similarity between the local feature amount A (q) and the local feature amount B (d) can be obtained as the cosine similarity according to the following equation (n is the number of dimensions of q and d).

The similarity between the local feature amount A and the local feature amount B is not limited to the cosine similarity, and a similarity such as SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference) may be adopted. Then, the search unit 105 acquires an object image of the local feature amount B whose similarity with the local feature amount A is equal to or higher than the threshold value from the search DB 112, and displays the acquired object image on the display unit 113. It should be noted that what is displayed is not limited to the object image of the local feature amount B whose similarity with the local feature amount A is equal to or greater than the threshold value, and for example, the similarity may be further displayed. Further, the object images may be displayed in descending order of similarity. Further, instead of displaying the object image of the local feature amount B whose similarity with the local feature amount A is equal to or higher than the threshold value, the object image of the local feature amount B having the highest similarity with the local feature amount A is displayed. Is also good.

The operation of the image processing apparatus described above will be described with reference to the flowchart of FIG. First, the object image and the registration process of the local feature amount of the object image will be described with reference to the flowchart of FIG. 4A. The process according to the flowchart of FIG. 4A is for the input image for one frame acquired by the image acquisition unit 101, and the process according to the flowchart of FIG. 4A is This is performed on the input image of each frame acquired by the image acquisition unit 101.

In step S401, the image acquisition unit 101 acquires an input image and sends the acquired input image to the extraction unit 102 in the subsequent stage. In step S402, the extraction unit 102 extracts an area in which an object such as a person appears from the input image received from the image acquisition unit 101 as an object area, and evaluates the image in the extracted object area as an object image in the subsequent stage. Output to unit 103.

In step S403, the evaluation unit 103 calculates the evaluation value of the object image input from the extraction unit 102. In step S404, the evaluation unit 103 determines whether or not the evaluation value calculated in step S403 is “1”. As a result of this determination, if the evaluation value calculated in step S403 is "1", the process proceeds to step S405, and if the evaluation value calculated in step S403 is not "1", the process proceeds to step S406.

In step S405, the evaluation unit 103 registers the object image and the local feature amount of the object image in both the registration DB 111 and the search DB 112. In step S406, the evaluation unit 103 registers the object image and the local feature amount of the object image in the search DB 112 without registering them in the registration DB 111.

Next, a search process for searching for an object image from the search DB 112 by using an object image selected from the object images registered in the registration DB 111 as a query will be described with reference to the flowchart of FIG. 4B.

In step S411, the selection unit 104 selects an object image to be used as a query from the object images registered in the registration DB 111 as a query image. In step S412, the search unit 105 obtains the similarity between the local feature amount of the query image acquired by the selection unit 104 from the registration DB 111 and each local feature amount registered in the search DB 112.

In step S413, the search unit 105 acquires an object image of the local feature amount whose similarity is equal to or greater than the threshold value from the search DB 112, and in step S414, the search unit 105 displays the acquired object image on the display unit 113. To display.

That is, the configuration of the image processing device described above is only an example of the following configuration. That is, it is determined whether or not the object image belongs to the same class as the plurality of reference object images, and if it is determined that the object image belongs, the object image is registered as a query candidate image, and if it is determined that the object image does not belong to the object. Register the image as a non-query candidate image. For example, it is not always necessary to provide the registration DB 111 and the search DB 112 as in the first embodiment, and the query candidate image and the non-query candidate image may be registered in the common memory. In that case, the query candidate image may be registered in the memory together with the identifier (representing the query candidate image), for example. As a result, for example, when a query candidate image is extracted from the object image group and displayed in a list in order to allow the user to select a query from the object image group registered in the memory, the query candidate image is registered together with an identifier representing the query candidate image. The object image may be read from the memory and displayed as a list.

<Modification example>
In the above configuration, the evaluation value is set according to whether or not the histogram of the local feature amount belongs to the specified class, but the evaluation value may be set by another method. For example, the evaluation value may be set by the method described in Document 3 below.

(Reference 3) Geoffrey E Hinton, "Reducing the Dimensionality of Data with Neural Networks", Science 313 (5786): 504-507
FIG. 5 shows an example of the input image acquired by the image acquisition unit 101. It is assumed that two object images 502 and 503 are extracted from the input image 501. In the following, a case where the evaluation values of the two object images 502 and 503 are obtained will be described.

As the model used here, the model 611 shown in FIG. 6 is used. The model 611 is a hierarchical neural network composed of three layers, an input layer, an intermediate layer, and an output layer. When a person image is input to the input layer, the person image is converted from the output layer to the person image after passing through the intermediate layer. It is output. This hierarchical neural network is such that the person image input to the input layer and the converted person image output from the output layer when the person image is input to the input are the same. I learned about portrait images.

The input layer is represented by y = f (Wx + b). W and b are weight vectors (weight vectors between the input layer and the intermediate layer) and bias vectors obtained as learning parameters by the above learning, respectively, and x is an input vector to the input layer. As shown on the upper side of FIG. 6, by inputting the object image 502 into the input layer as the input vector x, the object image 502 is projected onto a lower dimensional vector y. Further, as shown on the lower side of FIG. 6, by inputting the object image 503 into the input layer as the input vector x, the object image 503 is projected onto a lower-dimensional vector y.

The intermediate layer is represented by z = f (W'y + b'). W'and b'are weight vectors (weight vectors between the intermediate layer and the output layer) and bias vectors obtained as learning parameters by the above learning, respectively, and y is the output vector from the intermediate layer. As shown on the upper side of FIG. 6, by inputting the object image 502 into the input layer as the input vector x, the object image 602 is output as the vector z from the output layer. Further, as shown on the lower side of FIG. 6, by inputting the object image 503 into the input layer as the input vector x, the object image 603 is output from the output layer as the vector z.

Then, for each of the object images 502 and 503, the error Err is obtained according to the following equation.

That is, for the object image 502, the error between the object image 502 (x) and the object image 602 (z) is obtained, and for the object image 503, the error between the object image 503 (x) and the object image 603 (z) is obtained. Ask. The error between images used here is the sum of squared differences in pixel values at corresponding pixel positions between images for all pixels. The formula for calculating the error between images is not limited to this formula.

Then, the error obtained for the object image 502 is used as the evaluation value of the object image 502, and the error obtained for the object image 503 is used as the evaluation value of the object image 503. Then, the object image whose evaluated value is less than the threshold value and the local feature amount of the object image are registered in the registration DB 111 and the search DB 112, and the object image whose evaluated evaluation value is equal to or more than the threshold value and the local feature amount of the object image are It is registered in the search DB 112 without being registered in the registration DB 111. Here, as the local feature amount of the object image, the vector y output from the intermediate layer when the object image is input to the input layer is used.

In the example of FIG. 6, since the error between the object image 502 and the object image 602 is less than the threshold value, the local feature amounts of the object image 502 and the object image 502 are registered in the registration DB 111 and the search DB 112. On the other hand, in the object image 603, since the corresponding area of the object image 503 cannot be restored in the area 604, the error between the object image 503 and the object image 603 is equal to or larger than the threshold value. In such a case, the local feature amounts of the object image 503 and the object image 503 are registered in the search DB 112 without being registered in the registration DB 111. According to this method, for example, when two people overlap, it is possible to determine that the model 611 is not suitable as a selection candidate as a query.

The evaluation unit 103 performs the processing of obtaining the evaluation value of the object image using the hierarchical neural network described above and changing the registration destination of the object image and the local feature amount of the object image according to the evaluation value. .. Since the hierarchical neural network has already been learned as described above, it is created in advance and registered in the image processing device, and the evaluation unit 103 acquires and uses this registered hierarchical neural network to obtain an object image. Find the evaluation value of.

That is, the configuration of the image processing device described above is only an example of the following configuration. That is, the hierarchical type trained so that the image input to the hierarchical neural network and the image output from the hierarchical neural network when the image is input to the hierarchical neural network become the same image. Get a neural network. Then, it is determined whether or not the error between the object image and the image output from the hierarchical neural network when the object image is input to the hierarchical neural network is less than the threshold value. Then, if the error is less than the threshold value, the object image is registered as a query candidate image, and if the error is more than the threshold value, the object image is registered as a non-query candidate image.

[Second Embodiment]
In the present embodiment, when images taken by each of a plurality of cameras are acquired as input images and a plurality of object images are obtained for one object, which one is used according to the positional relationship between the object and the camera. Determine whether the object image taken by the camera is the target for calculating the evaluation value. Including the present embodiment, the differences from the first embodiment will be mainly described below, and the same as the first embodiment will be described unless otherwise specified below.

First, an example of the functional configuration of the image processing apparatus according to the present embodiment will be described with reference to the block diagram of FIG. In FIG. 7, the same functional unit as the functional unit shown in FIG. 1 is assigned the same reference number, and the description relating to the functional unit will be omitted.

Each of the image acquisition units 101-1 to 101-N (N is an integer of 2 or more) acquires the captured image captured by the first to Nth cameras as an input image. In addition, each of the image acquisition units 101-1 to 101-N may directly acquire the captured image captured by the first to Nth cameras from the first to Nth cameras, or may acquire a memory or the like. It may be acquired indirectly through. In any case, it is assumed that the input images acquired by the image acquisition unit 101-1-101-N are all captured images at the same shooting time.

Each of the extraction units 102-1-102-N acquires an object image from the input image acquired from the image acquisition unit 101-1-101-N in the same manner as the extraction unit 102, and obtains the acquired object image in the subsequent stage. Output to the position estimation unit 721.

The position estimation unit 721 estimates the three-dimensional position of the object corresponding to the object image. The position estimation unit 721 obtains the local feature amount of each object image received from the extraction unit 102-1-102-N, and the local feature amount is similar to the object image group received from the extraction unit 102-1-102-N. It is divided into groups of object images (groups of objects that appear to be the same object). Then, the position estimation unit 721 obtains the three-dimensional position of the object corresponding to the group by performing the following processing for each group. That is, the position estimation unit 721 used the position in the input image of the extraction source of the object image belonging to the group and the three-dimensional position (known) of the camera that captured the input image of the extraction source of the object image belonging to the group. The three-dimensional position of the object corresponding to the group is obtained by triangulation. Then, the position estimation unit 721 uses the object image extracted from the input image captured by the camera at the position closest to the three-dimensional position of the object corresponding to the group as the object image of the object corresponding to the group. Adopt (specific) as.

The operation of the position estimation unit 721 will be described by taking FIG. 8 as an example. In FIG. 8, four cameras (cameras 801-1 to 801-4) are provided (that is, N = 4 above). Since the camera provided at the position closest to the three-dimensional position of the person 812 as an object is the camera 801-3, the object image of the person 812 is the person 812 extracted from the input image acquired from the camera 801-3. The object image of is adopted. Further, since the camera provided at the position closest to the three-dimensional position of the person 811 as an object is the camera 801-2, the object image of the person 811 is extracted from the input image acquired from the camera 801-2. The object image of the person 811 is adopted.

Further, the object image taken by the camera closest to the object is not necessarily adopted, and when the object is concealed by another object, the object image taken by another camera may be adopted. For example, suppose that the camera closest to the person 811 is 801-3. However, as seen from the camera 801-3, the person 811 is hidden by the person 812. In such a case, the object image taken by the camera closest to the person 811 is adopted next to the camera 801-3.

For example, in the case of the camera 801-3, the vector a from the position of the camera 801-3 to the position of the person 811 and the vector b from the position of the camera 801-3 to the position of the person 812 are obtained. Then, the following equation is calculated to obtain the angle θ formed by the vector a and the vector b.

When the angle θ is less than the threshold value (a small value close to 0 degrees) and the magnitude of the vector a is larger than the magnitude of the vector b as shown in the following equation, the person 811 as viewed from the camera 801-3. Can be determined to be concealed by the person 812.

By performing such processing for each camera, it is possible to determine which camera does not use the object image as the object image. The object image and the registration process of the local feature amount of the object image will be described with reference to the flowchart of FIG. The process according to the flowchart of FIG. 9 is for the input image for one frame acquired by the image acquisition unit 101-1-101-N. The process according to the flowchart of FIG. 9 is performed on the input image of each frame acquired by the image acquisition unit 101-1-101-N. Further, in FIG. 9, the same processing steps as in FIG. 4A are assigned the same step numbers, and the description of the processing steps will be omitted.

In step S401, the image acquisition unit 101-1-101-N acquired the input image, and in step S402, the extraction unit 102-1-102-N acquired the input image, respectively, and the image acquisition unit 101-1-101-N acquired the input image. Acquire an object image from the input image.

In step S921, the position estimation unit 721 estimates the three-dimensional position of the object corresponding to the object image, and sets the object image extracted from the input image captured by the camera at the position closest to the three-dimensional position of the object in the subsequent stage. Is output to the evaluation unit 103 of.

The search process for searching the object image from the search DB 112 by using the object image selected from the object images registered in the registration DB 111 as a query is the same as in the first embodiment.

[Third Embodiment]
In the present embodiment, when the object image is registered, the attribute information representing the attribute of the object image is further registered, and the attribute information is used at the time of image search. First, an example of the functional configuration of the image processing apparatus according to the present embodiment will be described with reference to the block diagram of FIG. The configuration of FIG. 10 is that the attribute estimation unit 1011 is added to the configuration shown in FIG. 7, and the selection unit 1012 is provided instead of the selection unit 104. In the following, the differences from the second embodiment will be mainly described.

The attribute estimation unit 1011 acquires the attribute of the object image, and registers the object image and the information representing the attribute together with the local feature amount of the object image. For example, the attribute estimation unit 1011 acquires local features corresponding to each attribute such as age, gender, clothes, luggage, and hair color from the local features of the object image, and compares them with a model learned in advance by SVM. Determine the class of attributes. Then, the attribute estimation unit 1011 registers the label of the class determined for each attribute in addition to the object image and the local feature amount of the object image. Regarding the registration destination, if the object image is a query candidate image, it is the registration DB 111 and the search DB 112, and if the object image is a non-query candidate image, it is the search DB 112. In addition, various methods can be applied to the method of acquiring the attribute of the object image.

The selection unit 1012 selects a class of attributes to be used as a query. For example, the selection unit 1012 displays a list of attribute classes such as age, gender, clothes, luggage, and hair color on the display unit 113, and receives a selection instruction from the user. Then, it is assumed that the user operates an operation unit (not shown) to perform a selection operation of selecting a class of attributes to be used as a query from the listed attributes. At this time, the selection unit 1012 uses the class of the attribute selected by the user as a query, searches for the object image registered in the registration DB 111 together with the label of the class, and displays the object image in a list on the display unit 113. The number of attribute classes to be selected may be one or may be plural. In the case of a plurality of objects, the object image corresponding to each selected attribute class will be searched. Then, the selection unit 104 receives an object image selection instruction from the user. Then, when the user operates an operation unit (not shown) to perform a selection operation of selecting an object image as a query image from the displayed object images, the selection unit 1012 displays the object image selected by the user and the object image. The local feature amount of the object image and the local feature amount are acquired from the registration DB 111. The number of query images to be selected (acquired) may be one or a plurality.

The object image, the local feature amount of the object image, and the registration process of the attributes of the object image will be described with reference to the flowchart of FIG. 11A. The process according to the flowchart of FIG. 11A is for the input image for one frame acquired by the image acquisition unit 101-1-101-N. The process according to the flowchart of FIG. 11A is performed on the input image of each frame acquired by the image acquisition unit 101-1-101-N. Further, in FIG. 11A, the same processing steps as those in FIG. 9 are assigned the same step numbers, and the description of the processing steps will be omitted.

In step S1121, the attribute estimation unit 1011 acquires the attribute of the object image. If the evaluation value calculated in step S403 is "1", the process proceeds to step S1122 via step S404, and if the evaluation value calculated in step S403 is not "1", the process proceeds through step S404. Proceed to S1123.

In step S1122, the attribute estimation unit 1011 registers the object image, the local feature amount of the object image, and the label corresponding to the attribute class of the object image in both the registration DB 111 and the search DB 112. In step S1123, the attribute estimation unit 1011 registers the object image, the local feature amount of the object image, and the label corresponding to the attribute class of the object image in the search DB 112 without registering in the registration DB 111.

Next, the search process for searching the object image from the search DB 112 will be described with reference to the flowchart of FIG. 11B. In FIG. 11B, the same processing steps as in FIG. 4B are assigned the same step numbers, and the description of the processing steps will be omitted.

In step S1131, the selection unit 1012 selects an attribute class to be used as a query, uses the selected attribute class as a query, and searches for an object image registered in the registration DB 111 together with the label of the attribute class. A list is displayed on the display unit 113.

In step S1132, the selection unit 104 acquires the object image selected from the object images listed in step S1131 and the local feature amount of the object image from the registration DB 111.

As described above, according to the present embodiment, the query can be selected while narrowing down by the attribute, and the search can be performed more efficiently. In addition, some or all of the embodiments and modifications described above may be used in combination as appropriate.

[Fourth Embodiment]
All the functional parts of the image processing apparatus shown in FIGS. 1, 7 and 10 may be configured by hardware, but each part except the registration DB 111, the search DB 112 and the display unit 113 is realized by software (computer program). Is also good. In such a case, if the computer device has the registration DB 111 and the search DB 112 as memories and the display unit 113 as a display device and has a processor that executes this software, the above image processing device may be used. Applicable. An example of a hardware configuration of such a computer device will be described with reference to the block diagram of FIG.

The CPU 1201 executes various processes using computer programs and data stored in the RAM 1202 and the ROM 1203. As a result, the CPU 1201 controls the operation of the entire computer device, and also executes or controls each of the above-described processes as performed by the above-mentioned image processing device.

The RAM 1202 has an area for storing computer programs and data loaded from the ROM 1203 and the external storage device 1206, and data received from the outside via the I / F (interface) 1207. Further, the RAM 1202 has a work area used by the CPU 1201 to execute various processes. As described above, the RAM 1202 can appropriately provide various areas. The ROM 1203 stores setting data, a boot program, and the like of the computer device that do not need to be rewritten.

The operation unit 1204 is composed of a user interface such as a mouse and a keyboard, and various instructions can be input to the CPU 1201 by the user operating the operation unit 1204. For example, the above selection operation by the user can be realized by the user operating the operation unit 1204.

The display device 1205 is composed of a CRT, a liquid crystal screen, a projector, and the like, and can display or project the processing result by the CPU 1201 with images, characters, and the like. The display device 1205 also functions as the display unit 113. The touch panel screen may be configured by integrating the operation unit 1204 and the display device 1205.

The external storage device 1206 is a large-capacity information storage device typified by a hard disk drive device. The external storage device 1206 stores an OS (operating system) and computer programs and data for causing the CPU 1201 to execute each of the above-mentioned processes as those performed by the above-mentioned image processing device. The computer program stored in the external storage device 1206 includes a computer program for realizing the functions of each unit except the registration DB 111, the search DB 112, and the display unit 113 in FIGS. 1, 7 and 10 in the CPU 1201. In addition, the data stored in the external storage device 1206 includes data (threshold values, learning parameters, etc.) treated as known information in the above description. The registration DB 111 and the search DB 112 may be provided in the external storage device 1206. The computer programs and data stored in the external storage device 1206 are appropriately loaded into the RAM 1202 according to the control by the CPU 1201, and are processed by the CPU 1201.

The I / F 1207 functions as an interface for performing data communication with an external device. For example, when the registration DB 111 and the search DB 112 are realized as an external device, reading and writing of information to the registration DB 111 and the search DB 112 is performed via the I / F 1207.

The CPU 1201, RAM 1202, ROM 1203, operation unit 1204, display device 1205, external storage device 1206, and I / F 1207 are all connected to the bus 1208. The hardware configuration of the computer device applicable to the above image processing device is not limited to the configuration shown in FIG.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101: Image acquisition unit 102: Extraction unit 103: Evaluation unit 104: Selection unit 105: Search unit 111: Registration DB 112: Search DB 113: Display unit

Claims (14)

A means of determining whether an object image belongs to the same class as a plurality of reference object images,
When it is determined that the determination means belongs, the object image is registered as a query candidate image, and when it is determined that the determination means does not belong, the object image is registered as a non-query candidate image. An image processing device characterized by comprising. The image processing apparatus according to claim 1, wherein the determination means determines whether or not the local feature amount of the object image belongs to the same class as the local feature amount of the plurality of reference object images. Claim 1 is characterized in that the determination means obtains an evaluation value having a value according to whether or not the local feature amount of the object image belongs to the same class as the local feature amount of the plurality of reference object images. Or the image processing apparatus according to 2. When the evaluation value is a value indicating that the local feature amount of the object image belongs to the same class as the local feature amount of the plurality of reference object images, the registration means queries the object image as a query candidate image. If the evaluation value is not a value indicating that the local feature amount of the object image belongs to the same class as the local feature amount of the plurality of reference object images, the object image is regarded as a non-query candidate image. The image processing apparatus according to claim 3, wherein the image processing apparatus is registered as. Furthermore,
Among the object images extracted from the images captured by a plurality of cameras, the position of the object is obtained using the object image of the same object, and the position is extracted from the image captured by the camera provided at the position closest to the position. Equipped with specific means to identify the image of the object
The image processing according to any one of claims 1 to 4, wherein the determination means determines whether or not the object image specified by the specific means belongs to the same class as a plurality of reference object images. apparatus. The hierarchical neural network that has been trained so that the image input to the hierarchical neural network and the image output from the hierarchical neural network when the image is input to the hierarchical neural network are the same image. And the means to get
A means for determining whether or not the error between the object image and the image output from the hierarchical neural network when the object image is input to the hierarchical neural network is less than the threshold value, and
If the error is less than the threshold value, the object image is registered as a query candidate image, and if the error is greater than or equal to the threshold value, the object image is registered as a non-query candidate image. Image processing device. The registration means further claims that the output from the intermediate layer of the hierarchical neural network when the object image is input to the hierarchical neural network is registered as a local feature amount of the object image. The image processing apparatus according to 6. Furthermore,
The image processing apparatus according to any one of claims 1 to 7, further comprising a display control means for displaying a list of object images registered as query candidate images on a display unit. 8. The display control means is characterized in that, among the object images registered as the query candidate images, the object images corresponding to the attributes selected according to the operation by the user are displayed in a list on the display unit. The image processing apparatus according to. Furthermore,
The object image selected from the object images listed on the display unit according to the operation by the user is used as a query, and the image registered as the query candidate image and the image registered as the non-query candidate image are searched. The image processing apparatus according to claim 8 or 9, further comprising a search means. When the object image is registered as a query candidate image , the registration means registers the object image as a query candidate image in the first storage unit and the second storage unit, and registers the object image as a non-query candidate image. In the case of registering as , the object image is registered as a non-query candidate image in the second storage unit without being registered in the first storage unit.
The search means queries the object images selected in response to an operation by the user from the object images listed in the display unit, and searches for the object images registered in the second storage unit. The image processing apparatus according to claim 10. This is an image processing method performed by an image processing device.
The determination means of the image processing device includes a determination step of determining whether or not the object image belongs to the same class as the plurality of reference object images.
When the registration means of the image processing device determines that it belongs in the determination step, the object image is registered as a query candidate image, and when it is determined that it does not belong in the determination step, the object image is not registered. An image processing method including a registration process for registering as a query candidate image. This is an image processing method performed by an image processing device.
An image input to the hierarchical neural network by the acquisition means of the image processing device and an image output from the hierarchical neural network when the image is input to the hierarchical neural network are the same image. The process of acquiring the hierarchical neural network that has been learned in
Whether or not the error between the object image and the image output from the hierarchical neural network when the object image is input to the hierarchical neural network is less than the threshold value by the determination means of the image processing device. Judgment process to judge and
If the error is less than the threshold value, the registration means of the image processing device registers the object image as a query candidate image, and if the error is greater than or equal to the threshold value, the object image is registered as a non-query candidate image. An image processing method characterized by including a process. A computer program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 11.