JP7346528B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to face recognition technology using images.

There is a face recognition technology that determines whether the face of a person in an image is the same as a person in another image. In face recognition, matching is generally difficult when the conditions of the photographing environment and the state of the object differ, such as the viewing angle of the object at the time of photographing, lighting, presence or absence of items worn such as masks and glasses, etc. Therefore, in Patent Document 1, when extracting features of a person from an image, it is determined whether a person is wearing a mask or glasses, and an image region from which feature amounts are extracted is dynamically changed according to the result.

Patent No. 4957056

However, in Patent Document 1, when registering a person, it is necessary to store characteristics of multiple patterns depending on the state of the worn items and the like.

The present invention has been made in view of the above problems, and an object of the present invention is to reduce the amount of information that should be registered when comparing objects in different states.

An image processing device according to the present invention that solves the above problems is characterized in that a first feature of a first object that meets a predetermined condition in a first image is obtained using a first trained model. and a second feature of a second object that does not meet the predetermined condition in the second image, the second acquired model being a second learned model. and a second acquisition means for acquiring a second feature amount obtained using the first object and the first object in the first image based on the first feature amount and the second feature amount. and a matching means for determining whether or not the second object in the second image is the same object , and the first and second trained models are connected to the first object and the second object. When the second objects are the same object, the first feature obtained using the first trained model as the feature of the first object that meets the predetermined condition, and the predetermined The feature quantity of the second object that does not meet the condition is trained so that the second feature quantity obtained using the second trained model becomes a similar feature quantity. .

According to the present invention, it is possible to further reduce the amount of information to be registered when comparing objects in different states.

Block diagram showing an example of functional configuration of an image processing device Block diagram showing an example of the hardware configuration of an image processing device Schematic diagram showing an example of the operation of the matching process Flowchart showing processing performed by the image processing device Flowchart showing processing performed by the image processing device Schematic diagram showing an example of the operation of learning processing Schematic diagram showing an example of the operation of learning processing Flowchart showing processing performed by the image processing device Schematic diagram showing an example of learning processing Block diagram showing an example of functional configuration of an image processing device Flowchart showing processing performed by the image processing device Flowchart showing processing performed by the image processing device Schematic diagram showing an example of the operation of learning processing Block diagram showing an example of functional configuration of an image processing device Block diagram showing an example of functional configuration of an image processing device Schematic diagram showing an example of the operation of the matching process Flowchart showing processing performed by the image processing device Flowchart showing processing performed by the image processing device Block diagram showing an example of functional configuration of an image processing device Flowchart showing processing performed by the image processing device Flowchart showing processing performed by the image processing device

<Embodiment 1>
An image processing device according to an embodiment of the present invention will be described with reference to the drawings. Components with the same reference numerals in the drawings operate in the same way, and redundant explanation will be omitted. Furthermore, the components described in this embodiment are merely examples, and the scope of the present invention is not intended to be limited thereto.

Conventional facial recognition technology has two major problems. The first is (1) when registering a person, it is necessary to store multiple patterns of characteristics depending on the state of things worn, etc. Or (2) it is necessary to perform feature amount conversion of the registered image after determining the state of the person's mask or the like. Therefore, when there are a large number of registered persons to be verified, method (1) requires a large storage area, and method (2) has the problem of poor verification speed. The image processing apparatus according to the present embodiment converts the object into a feature amount using different feature amount converting means depending on the state of the object in the image at the time of photographing, and then performs matching. This results in superior matching accuracy compared to conventional methods that do not change the feature value conversion means depending on the state. Further, according to the present invention, learning is adjusted so that the output feature amounts are similar to each other for the same object, while using different conversion means. Therefore, even if the conversion methods are different, they can be used for matching processing without distinction. Therefore, compared to the conventional method of extracting feature amounts of registered image patterns, the amount of memory required to store feature amounts is smaller. Alternatively, it is superior in calculation cost and speed of matching processing.

FIG. 1 is a diagram showing an example of the functional configuration of an image processing apparatus. The image processing device 1 includes a first image acquisition section 101, a second image acquisition section 102, an object parameter determination 103, a storage section 104, a first feature amount conversion section 105, a second feature amount conversion section 106, and a feature It has a quantity matching section 107. Details will be described later.

FIG. 2 is a hardware configuration diagram of the image processing device 1 in this embodiment. The CPU H101 controls the entire apparatus by executing a control program stored in the ROM H102. RAM H103 temporarily stores various data from each component. Additionally, the program is expanded and made executable by the CPU H101. The storage unit H104 stores conversion parameters for performing image conversion in this embodiment. As a medium for the storage unit H104, an HDD, flash memory, various optical media, etc. can be used. The acquisition unit H105 is configured with a keyboard, touch panel, dial, etc., and receives input from the user, and is used for setting an arbitrary viewpoint when reconstructing an image of a subject. The display unit H106 is configured with a liquid crystal display or the like, and displays the result of reconstructing the image of the subject. Further, this device can communicate with a photographing device and other devices via the communication unit H107.

<Image matching processing phase>
FIG. 3 is a schematic diagram of the matching process of this embodiment, and shows the difference between the method of the present invention and the conventional method. FIG. 3A shows a conventional method in which feature amounts are converted using the same parameters for an input image including a person to be subjected to authentication processing and a registered image including a registered person. At this time, if there is a large change in appearance, such as whether or not a mask or sunglasses are worn, accuracy is likely to deteriorate. On the other hand, there is a problem in that the size of the structure of the feature quantity converting section becomes large when trying to cope with all kinds of changes in appearance. FIG. 3(B) is a schematic diagram of the present invention. In the figure, when an input image is input, an object parameter determination 103 determines the state of the subject, such as whether or not a mask is worn. According to the determination result, the feature amount conversion unit 106 reads appropriate conversion parameters from the storage unit 104 and performs feature amount conversion. Here, a plurality of types of conversion parameters are learned depending on the state of the person and the shooting environment. Since the conversion parameters are learned individually based on the state of the subject, robust matching can be achieved even with large changes in appearance, such as whether or not a mask or sunglasses are worn.

In addition, in the method of this embodiment, the above-mentioned feature quantities are trained so that they have a high degree of similarity to each other if they are the same object, regardless of which transformation parameter they are transformed with (the learning method will be described later). ). Therefore, the feature amount matching unit 107 only needs to calculate the degree of similarity based on a basic method such as an inner product or an angle between feature amounts, and no special processing is required. In this way, one type of similarity can be used as a uniform matching measure regardless of the state of the object. For example, in the method of Patent Document 1, it is necessary to store the same number of feature quantities of registered persons as the number of feature extraction methods, whereas in the method of this embodiment, one transformation is required for registered persons. Since parameters are applied, it is possible to narrow down the features to be registered.

Next, the procedure of the verification process will be explained using FIG. 4. The purpose of this embodiment is to determine, when two person images are given, whether the same person or different people are shown based on image feature amounts. The processing shown in the flowchart of FIG. 4 is executed by the CPU 101 of FIG. 2, which is a computer, according to a computer program stored in the storage device 104. In the following description, each process (step) is indicated by adding S to the beginning thereof, thereby omitting the notation of the process (step).

First, in S101, the first image acquisition unit 101 acquires a first image (first image) including an object to be authenticated (here, a person). In S102, the determination unit 103 determines whether the first image satisfies a predetermined condition. If the predetermined conditions are met, the state of the object or the shooting environment is normal (close to the learned environment); otherwise, if a mask is worn or the illuminance of the environment has changed. etc., it is determined that the state is not normal. Here, specifically, it is determined whether the person in the first image is wearing a mask. Mask detection uses a method such as template matching. If the predetermined condition (not wearing a mask) is met, the process advances to S103. If the predetermined condition is not met (masked), the process advances to S104.

In S103, the first feature value conversion unit (first feature acquisition unit) 105 reads out parameters for feature value conversion for a normal person (first parameter set) and sets them in the learned model. The trained model is a neural network for acquiring object features from images. The trained model to which the first parameter set is applied is called a first trained model. In S104, the first feature amount conversion unit 105 reads the feature amount conversion parameters (second parameter set) for a person wearing a mask and sets them in the learned model. The trained model to which the second parameter set is applied is called a second trained model. Here, the feature quantity conversion unit 105 is configured with a convolutional neural network known in Non-Patent Document 1, for example. Alternatively, the feature value conversion unit 105 is configured with a deep neural network (hereinafter abbreviated as DNN) known as a Transformer network in Patent Document 2. In other words, the feature amount conversion unit 105 is a trained model for acquiring the features of the person included in the image, and acquires the feature amount using a parameter set learned according to the state of the person included in the image. do. (Non-Patent Document 1: Deng, et. Al., ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In CVPR, 2019). (Patent Document 2: US Pat. No. 1,095,6819). Here, the parameters for feature value conversion are various parameters such as the number of neuron layers, the number of neurons, and connection weights. Next, in S105, the first feature converter 105 converts the feature from the first image received from the first image acquisition unit 101 based on the first trained model or the second trained model. Convert.

Next, in S106 to S110, the same processing as in S101 to S105 described above is performed on the second image (second image). That is, if the person included in the second image is not wearing a mask, the feature amount is acquired from the trained model to which the first parameter set is applied. If the person included in the second image is wearing a mask, the feature amount is acquired from the second trained model to which the second parameter set is applied. However, the above processing is performed by the second image acquisition unit 102 and the second feature amount conversion unit (second feature acquisition unit) 106. As a result, the first image and the second image are each converted into feature amounts. These feature amounts are expressed as f ₁ and f ₂ . Let f ₁ and f ₂ be one-dimensional vectors as in Non-Patent Document 1. (It is converted into a one-dimensional vector through the fully connected layer processing of the DNN.) Furthermore, the DNN parameters received by the first feature converter 105 and the second feature converter 106 have the same configuration. Although it is not necessary, the number of output channels of neurons in the final layer should be the same. As a result, it is assumed that the dimension lengths of f ₁ and f ₂ are the same.

Next, in S111, the feature matching unit 107 calculates a similarity score between the two feature amounts. That is, based on the first feature amount and the second feature amount, it is determined whether the object included in the first image and the object included in the second image are the same. If the similarity score between the first feature amount and the second feature amount is greater than or equal to a predetermined threshold, the two images include the same object. If the similarity score between the first feature amount and the second feature amount is smaller than a predetermined threshold, the two images include different objects. Although a plurality of indices are known to measure the similarity between feature amounts, here, as in the method of Non-Patent Document 1, angles between feature vectors are used. Calculate the similarity score as follows.
(Formula 1)
Similarity score (f ₁ , f ₂ ) := cos(θ ₁₂ )
=<f ₁ , f ₂ >÷(|f ₁ |・|f ₂ |)
However, θ ₁₂ is the angle formed by the feature vectors f ₁ and f ₂ , <x, y> is the inner product of x and y, and |x| is the length of x. The feature matching unit 107 determines that the person is the same person if the above-mentioned similarity score is equal to or higher than a predetermined threshold, and otherwise determines that the person is a different person. This completes the operation of the matching process. Note that the first image and the second image may have a configuration in which feature amounts are acquired by a common image acquisition section and a feature amount conversion section.

<Learning processing phase>
The learning phase of this embodiment will be explained. Here, learning is performed using the <representative vector method> known in Non-Patent Document 1. The representative vector method is a face recognition learning method that sets feature vectors that represent each person and uses them together to increase learning efficiency. For details, please refer to Non-Patent Document 1. Note that the image processing device 2 in the learning processing phase is shown in FIG. The image conversion unit 200 transforms a first image group, which is a set of reference images of the target (for example, a face image of a person without wearing anything), into an image showing a predetermined state of the target (for example, a face image of a person wearing no mask). The second image group is a set of facial images of the person wearing the device. Specifically, an image showing something worn such as a mask is combined with a facial image, or the image is converted to have a certain brightness. The image acquisition unit 201 acquires a group of images used for learning. Here, in order to learn two or more types of parameter sets, two or more types of image groups are acquired. The feature amount conversion unit 202 obtains feature amounts from each image using a parameter set according to the state of the image and a learning model that extracts feature amounts from the images. The learning unit 203 learns a parameter set and a learning model for extracting feature amounts from images. In this embodiment, an example will be described in which the first learning model and the second learning model are learned alternately.

The processing flow procedure of this embodiment consists of FIGS. 5(A) and 5(B). Here, the process shown in FIG. 5(A) will be referred to as <first learning process>, and the process shown in FIG. 5(B) will be referred to as <second learning process>. In the <first learning process>, normal feature amount conversion learning is performed using a group of images of a person not wearing a mask (first group of images). In the <second learning process>, learning specialized for masked persons is performed using a group of images of persons wearing masks (second image group). Note that the solid line part in FIG. 14 is the configuration used in the process of <first learning process>, and the broken line part is the configuration used in the process of <second learning process>.

The contents of <first learning process> basically follow the method of Non-Patent Document 1. FIG. 5A shows processing in the learning phase performed by the image processing device. First, in S201, the feature quantity conversion unit 202 initializes the parameter set and representative vectors v ₁ to v _n of the first learning model with random numbers. Here, 1 to n are IDs of all persons included in the learning image. Each representative vector v is a d-dimensional vector (d is a predetermined value).

Next, in S202, the image acquisition unit 201 acquires images I ₁ to I _m randomly selected from the first image group. The first image group is a reference image group, and is a plurality of images of people not wearing masks, and includes one or more images for each person. Each image is attached with information about the person's ID.

Next, in S203, the feature amount conversion unit 202 obtains a first learning feature amount f _i by inputting each image I _i of the first image group to the first learning model. Here, the learning feature amount f _i is a d-dimensional vector. Next, in S204, the feature amount conversion unit 202 converts the similarity of the feature amounts between each person image and the representative vector (intra-class similarity) and the similarity of the feature amounts of the representative vectors of each person and another person (inter-class similarity). ), calculate the loss value.

(Formula 2)
Intra-class similarity score ( _fi ) = similarity score ( _fi , vy _(i) ),
Inter-class similarity score (f _i ) = Σ _j≠y(-i) similarity score (f _i , v _j )
However, here, y(i) is the ID number of the person in the image _Ii . The sum of these values for each image as shown below becomes the loss value used for learning.

(Formula 3)
Loss value = Σ _i inter-class similarity score (f _i ) - λ intra-class similarity score (f _i )
λ is a weight parameter for learning balance. Note that the above is an example of a loss value, and there are various known methods such as using a similarity score with a margin or cross entropy. For details, please refer to Non-Patent Document 1 etc.

Next, in S205 and S206, the learning unit 203 updates the first parameter set of the feature conversion unit (first learning model) so as to reduce the above loss value. In S205, the feature value conversion unit 203 updates the value of the representative vector, and in S206, updates the first parameter set. By using an error backpropagation method commonly used in DNNs, the loss value is slightly updated in the direction of decreasing it. As a result, the representative vector is improved so that it functions more as a value that represents the characteristics of each person, and the first learned model is improved so that it becomes similar to each other if it is a feature amount of the same person.

The above learning process is repeated until the learning converges or a predetermined number of times (S207). Next, in S208 and S209, the storage unit 104 stores and saves the first parameter set and the values of the representative vectors v ₁ to v _n .

FIG. 6 schematically shows an example of the results when the <first learning process> is completed. Representative vectors 601, 602, and 603 are obtained on the feature space 600 as feature vectors representing the persons with IDs 1 to 3. Furthermore, the first parameter set has been appropriately learned so that the features a, b, p, q, etc. of each person are located in the vicinity of these representative vectors (the image features of each person are indicated by black circles in the figure). ).

Next, <second learning process> is performed. In this process, a DNN (second learning model) for feature value conversion for a person wearing a mask is learned using a learning image group (second image group) of a person wearing a mask.

<Second learning process> will be explained using FIG. 5(B). As preparation, in S300, the image conversion unit 200 converts the first image group into a second image group that satisfies a predetermined condition. Specifically, an existing conversion method is used to generate a composite image of wearing items such as masks and sunglasses, or images with different illuminances. If the second image group is prepared in advance, S300 may be skipped. In S301, the feature value conversion unit 202 acquires a first parameter set and uses it as an initial value of the parameters of the second learning model. Next, learning of the second parameter of the second learning model is performed from S302 to S306 in the same manner as in the process flow of FIG. 5(A). The contents of the process, calculation of loss, etc. are the same as the processes of S202 to S207. However, the updating processing of the representative vectors v ₁ to v _n performed in S205 is not performed, and the values saved in the previous step S208 are used as fixed values. As a result, learning is performed such that the feature amount of a person wearing a mask approaches the representative vector of a person not wearing a mask. When the learning converges, in S307, the storage unit 104 stores the second parameter set and ends the learning. Note that the value of the representative vector is used only during learning, and is not used during the matching operation.

FIG. 7 is a diagram schematically showing the start point of the <second learning process>. The positions of representative vectors 601, 602, and 603 are fixed and will not be updated by learning thereafter. Images c and d of a person wearing a mask are located far from the representative vector 601 of that person. By performing the learning adjustment in the <second learning process>, the features of each person are adjusted in the second direction so that they approach the direction of their respective representative vectors, as shown by the arrow attached to feature c (numbered 702). parameter set is learned. As a result, when learning converges, the features using the first parameter set are calculated for images of a person not wearing a mask (a, b in FIG. 6), and images of a person wearing a mask (c, d) and the feature amount using the second parameter set become close to each other on the feature space.

<Derivative forms of learning methods>
Here are some other derivative forms of learning. For example, a learning form that does not use <representative vector> is also conceivable. A flow example of this learning operation process will be explained using FIG. 8 and a schematic diagram using FIG. 9. In this embodiment, a set of normal human images and a group of images obtained by superimposing and combining the same images with a mask image are used. FIG. 9A shows examples of images a, b, and p of normal people and images a', b', and p' on which masks are superimposed. In the example of this derivation, the second parameter set is learned so that the feature quantities of images a', b', and p' approach the feature quantities of images a, b, and p, respectively.

First, in the <first learning process>, a learning process based on the method described above is performed in S401 to S407 using a group of images of normal people. Note that, unlike the above-mentioned method, a loss value is calculated from the intra-class similarity and the inter-class similarity using the following formula without using the representative vector, and the first parameter set of the first learning model is updated.

(Formula 4)
Intraclass similarity score (f _i ) = Σ _{y(k) = y(i)} similarity score (f _i , f _k ),
Inter-class similarity score (f _i ) = Σ _{y (j)≠y (-i)} similarity score (f _i , f _j ),
Loss value = Σ _i inter-class similarity score (f _i ) - λ intra-class similarity score (f _i )
Here, f _i and f _k are a pair of feature quantities of the same person, and f _i and f _j are a pair of feature quantities of strangers. The results of <first learning process> are shown in FIG. 9(B).

Next, in <second learning process>, a second parameter set of the second learning model is learned. In S501, the feature value conversion unit 202 initializes the parameters of the DNN, and in S502, the image acquisition unit 201 converts the original image before superimposing the mask (first learning image) and the synthesized superimposed image ( second training image). In other words, the first learning image and the second learning image are images of the same object, but are a pair of images in which the state of the object and the shooting environment are different. In S503 and S504, the feature amount conversion unit 202 converts the first learning feature amount from the first learning model and the original image (first image) and the first learning feature amount from the second feature model and the composite image (second image). Obtain learning features for each. In S505, the learning unit 203 calculates loss values within and between classes of the person. At this time, in addition to the terms of similarity scores within and between classes of persons used so far, a term of similarity of image pairs is newly added as shown in the following equation.

(Formula 5)
Image pair similarity score (f _x ) = similarity score (f _x , f _x' )
(Formula 6)
Loss value = Σ _i inter-class similarity score (f _i ) - λ ₁ intra-class similarity score (f _i )
- λ ₂ image pair similarity score (f _i )
Note that in the above equation, f _x is a feature amount of image x, and f _x' is a feature amount of image x' obtained by superimposing and synthesizing a mask on image x. λ ₁ and λ ₂ are parameters for balancing each term.

The similarity term for image pairs is defined as the distance between the learning features of the original image before mask superimposition (first learning image) and the composite image after superposition (second learning image) is smaller than a predetermined value. Learn to become. A schematic diagram of the similarity terms of feature quantity pairs is also shown in FIG. 9C with numbers 900, 901, and 902 attached to arrows. In the figure, an arrow 903 indicates the conventional intra-class similarity, and an arrow 904 indicates the inter-class similarity. By defining a loss value by combining multiple degrees of similarity in this way, it is expected that the accuracy of matching will be improved. In S506, the second parameter set of the second learning model is trained to reduce the above loss value. Since the first learning model is not trained here, in this <second learning process>, the features of the original image without the mask are "fixed" and do not move, and the features of the synthesized image with the mask on are "fixed" and do not move. Learning is performed such that the amount changes in a direction closer to the feature amount of the mask not being worn. If the learning unit 203 determines that the learning has converged in S507, it saves the second parameter set of the second learning model and ends the learning in S508. The above are examples of derivative forms of learning methods.

Furthermore, other examples of learning methods are also possible. As one example, when learning the feature amount conversion unit for normal people in <first learning process>, it is conceivable to perform learning including some masked person images. In this way, even if the object parameter determination 103 fails during verification and an incorrect feature value conversion parameter is applied, it can be expected to prevent significant performance deterioration. Similarly, when learning the feature amount conversion unit for a person wearing a mask, it is also possible to mix and learn images of a normal person.

As described above, various types of learning processing are possible. It is also conceivable to apply the plurality of learning processing methods described here in stages according to the progress of learning. In this way, the processing for learning the image processing apparatus of the present invention is not limited to one example.

<Derivative form of the configuration of the feature amount conversion unit>
Next, an example of a derived form of the DNN configuration will be given. For example, it is conceivable to change the number of layers and the number of neurons between a DNN for feature value conversion for a normal person and a DNN for a person wearing a mask. In general, for objects that are difficult to match, such as a person wearing a mask or a person in profile, or for objects with a wide variety of appearances, performance is likely to be improved by using a large-scale DNN. Therefore, by adjusting the scale of each DNN depending on the target to be handled, it is possible to improve the cost-effectiveness of calculation cost and matching accuracy.

Another method is to use a DNN for feature value conversion for normal people and a DNN for people wearing masks, and share the first layer, and only partially change the second layer depending on the state of the person. is possible.

As another form, it is conceivable to use feature amount conversion means having completely different configurations, including a feature amount conversion section for a normal person and a feature amount conversion section for a person wearing a mask. For example, it is conceivable to use a convolutional neural network in the feature quantity transformation unit for a normal person, and to use a Transformer network known in Patent Document 2 for a person wearing a mask. Alternatively, a recursive neural network or the like may be used. As long as it is possible to adjust the parameters based on the loss value, a wide variety of feature value conversion means other than DNN can be applied to the feature value conversion unit.

As yet another form of derivation, the feature quantities f ₁ and f ₂ obtained by transforming the input image may be in the form of an N-dimensional matrix instead of a one-dimensional vector. Further, in this embodiment, the lengths of the feature vectors obtained from the first trained model and the second trained model are the same, but the lengths may be different. When using feature quantities of different lengths, a known method for calculating the similarity between vectors of unequal length, such as Earth Mover's Distance, may be used.

This concludes the description of the first embodiment.

<Embodiment 2>
In this embodiment, the present invention is applied to modes other than switching depending on whether or not a mask or sunglasses are worn. In the first embodiment, one-to-one images are input, and it is determined whether the images are the same object. In this embodiment, an example of a configuration will be described assuming a use case such as an automatic door gate that opens and closes using facial recognition. The feature amounts of N people are registered in advance in the image processing apparatus of this embodiment. During verification, a single image taken with a camera in front of the gate is input as an input image, and it is checked whether the input person is the same as one of the registered N people or does not correspond to any of them. judge.

In the first embodiment, the presence or absence of a mask is determined and the feature quantity conversion unit is switched. In this embodiment, the facial image for registration (frontal face with good lighting conditions) and the facial image for inquiry (poor lighting conditions, large face angle, etc. depending on the camera installation situation) are used. The shooting conditions are very different. Therefore, we will learn and use feature quantity conversion units corresponding to each.

FIG. 10 shows an example of the functional configuration of the image processing device 3. The basic configuration is similar to that shown in FIG. The difference is that a feature registration section 108 and a processing mode setting section 109 are newly provided. The flow of the matching process is shown in FIG. FIG. 11(A) shows the registration operation of a person, and FIG. 11(B) shows the matching operation between the input image and the registered person.

When the image processing device 3 starts the registration operation, the processing mode setting unit 109 sets the current operation mode to the registration operation mode (S601). In S602, the first feature converter 105 acquires a conversion parameter set (first parameter set) for the registered operation mode. Apply the obtained parameter set to the trained model. Next, in S604, the first image acquisition unit 101 inputs all N person images for registration one by one (S604), the feature value conversion unit 105 converts them into feature values (S605), and the feature registration unit 108 is registered as a feature amount of each person. The registered image is assumed to be a frontal face of a person photographed under good conditions. Therefore, the first feature converter is trained in advance mainly using frontal faces.

Next, when the image processing apparatus starts a matching operation, the processing mode setting unit 109 sets the operation mode to the matching operation mode (S701). First, in S702, the second feature converter 106 acquires a parameter set (second parameter set) selected according to the situation from among a plurality of learned parameter sets. The second parameter set is learned in advance using people from various angles as learning data.

In S703, the second image acquisition unit 102 acquires one captured input image. Note that depending on the positional relationship between the camera and the gate door, where the person appears in the image cannot be determined in advance. Therefore, a face detector may be provided inside the second image acquisition unit 102 to detect the face and cut out only the image around the face. (A widely known face detector may be used.) Next, the second feature converter 106 acquires a second feature from the input image (S704). In S705 to S707, the feature matching unit 107 calculates the similarity between the input image feature and each registered feature one by one (S706), and if there is a candidate person whose similarity is higher than a predetermined value, The result is output (S708). Although not shown in the processing flow, in an actual use case, the opening/closing operation of the gate door is performed based on the above results. Specifically, if the person included in the second image matches one of the registered persons, the gate is controlled to be opened; if the person included in the second image does not match any of the registered persons, the gate is not opened and the gate is opened as necessary. Output a notification to the administrator. The authentication result may be output to a display device near the entrance gate.

FIG. 12 is a flowchart of learning processing according to the second embodiment. A schematic diagram is also shown in FIG. The difference here from the first embodiment is that the first learning model and the second learning model are learned at the same time. It will be explained that the learning method of this embodiment is also applicable to such a method. Note that an example of the hardware configuration is the same as that shown in FIG. 2, and an example of the functional configuration of the image processing apparatus is the same as that shown in FIG. 14.

In S801 of FIG. 12, the image acquisition unit 201 acquires a first learning image group that is a collection of only frontal images that simulate the photographing conditions of the registered images. In S802, the feature amount conversion unit 202 acquires a first learning feature amount from a first learning image group based on a first learning model using a first parameter set. In S803, the image acquisition unit 201 acquires a second learning image group. The second image group includes various human images from different angles, including a top-down view and the like assuming the input image. In S804, the feature amount conversion unit 202 acquires a second learning feature amount from the second learning image group based on the second learning model using the second parameter set.

In S805, the learning unit 203 randomly selects images from each image group to create a subject pair (intra-class pair) and an other-person pair (inter-class pair), and calculates a loss value based on the similarity between the feature amounts. seek. As the loss, a triplet loss known in Non-Patent Document 2 is used as described below. (Non-patent document 2: Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and cluster In CVPR, 2015).

(Formula 7)
Loss value = Σ _i [inter-class pair similarity score (f _i, f _j )
− Intra-class pair similarity score (f _i, f _k )+m ] ⁺ ,
However, m is a constant for the loss margin value to make learning robust, and [・] ⁺ is (Formula 8)
[x] ⁺ =x If x>0
[x] ⁺ =0 Otherwise
This is a function defined by .
Here, f _i is a feature amount of the person image I _i , f _j is a feature amount of a person different from the image I _i , and f _k is a feature amount of another image I _k of the same person as I _i .

Note that the person images I _i are randomly selected from the first learning set or the second learning set, and the person images I _j and I _k are sampled accordingly to create inter-class pairs and intra-class pairs. At this time, when the person image I _i is selected from the first learning set, the person images I _j and I _k are selected from the second learning set, and when the person image I i is selected from the second learning set, the person images I j and I _k are selected from the second learning set. Images I _j and I _k are selected from the first training set. Thereby, the first learning model and the second learning model can be trained in conjunction with each other.

In S806, the learning unit 203 performs learning update of parameters using the error back propagation method in a direction that reduces the loss value of each of the first learning model and the second learning model. As a result, as shown in the schematic diagram in Figure 13, a loss value based on the similarity is calculated for each output of the two learning models, and this is back-propagated to each feature conversion unit as an error to update the learning. will be held.

As described above, an example has been described in which the first learning model and the second learning model process images with different characteristics and simultaneously perform learning on both models. As a derivative method, a combination of learning two learning models at the same time in the initial stage, fixing the first feature amount and learning only the second feature amount in the latter half can also be considered.

<Embodiment 3>
In the above-described embodiment, both the state determination and the feature amount conversion obtain the state and feature amount from the image. In this embodiment, an embodiment will be described in which an intermediate feature amount is generated from an image, and state determination and feature amount conversion are performed based on the intermediate feature amount. Here, the state includes, for example, attributes of a person such as gender, race, and age. In this embodiment, when obtaining feature amounts for identifying a person included in an image, some parameters of the learning model are changed depending on the attributes of the person. On the other hand, a common learning model layer is used for determining attributes (states) of a person and converting features. As a result, the processing of state determination and feature quantity conversion is shared, and speed and memory efficiency are improved.

In this embodiment, using FIGS. 15 to 18, the same one-to-one images as in the first embodiment are input, and "one-to-one image matching processing" is performed to determine whether the subjects are the same object. I will explain about it. Next, using FIGS. 19 and 20, "1-to-N image matching processing" is performed to determine whether the person appearing in the input image is the same as any registered person from among the N people registered in advance. The case will be explained below. Note that the hardware configuration is the same as that of the information processing apparatus shown in FIG. 2 in the first and second embodiments.

<One-to-one image matching process>
FIG. 15 shows an example of the functional configuration of the image processing device 15. The basic configuration is similar to that shown in FIG. The difference is that the first feature amount conversion unit 1501 generates intermediate feature amounts. Along with this, the parameter determination section 1502, the second feature amount conversion section 1504, and the third feature amount conversion section 1505 (third feature acquisition section) operate based on the intermediate feature amount. The parameter determining unit 1502 determines the parameters of the learned model according to the state of the object (in the case of a person, the attribute) included in the image. The parameter determination unit 1502 estimates the state of an object included in the image based on the intermediate feature amount of the image. The estimation method estimates that the attribute of interest is the attribute of interest if the degree of matching with the representative feature amount of the attribute of interest is greater than or equal to a predetermined threshold. Alternatively, the state of the object included in the image is estimated based on a third learned model that outputs feature amounts related to the state of the object from the image. Furthermore, the parameter determination unit 1502 determines transformation parameters that are associated in advance according to the estimated state (person's attributes). That is, if the attributes of the object included in the first image and the attributes of the object included in the second image are the same, the same learned model (or feature conversion parameter) is determined. If the attributes of the object included in the first image and the attributes of the object included in the second image are different, different learned models (or model parameters) are determined. Furthermore, the storage unit 1503 stores conversion parameters to be supplied to the second feature amount conversion unit 1504 and the third feature amount conversion unit 1505.

FIG. 16 is a schematic diagram of the matching process of this embodiment. The input image is converted by a first feature converting unit 1501 into an intermediate feature related to the state of the object. Using the converted intermediate feature amount, the parameter determination unit 1502 determines conversion parameters according to the state. The state of an object includes gender, race, etc. Alternatively, it may be age, facial orientation, presence or absence of a mask, etc., but is not limited to these. The storage unit 1503 stores conversion parameters 1602 specific to state Y and predetermined general conversion parameters 1601 corresponding to all states. For example, if the state determination for the input image is "state Y", the conversion parameter 1602 for state Y is set in the third feature quantity conversion unit 1505. Note that if the target object does not fit into a specific learned state, a predetermined parameter may be given as a dummy. Then, the third feature converter 1505 converts the intermediate feature into a facial feature based on the parameters determined by the parameter determiner 1502. Note that in the embodiment described above, it was called a feature amount, but in order to make it easier to distinguish it from an intermediate feature amount, it is called a facial feature amount. Next, the registered image is also converted into facial features, and the feature matching unit 107 matches the facial features of the input image and the registered image.

As a result, the processing speed can be increased because the parts to be converted into intermediate feature quantities are shared. In addition, the size of the models of the parameter determination section and the second and third feature conversion sections can be reduced. Further, by reducing the model size, the size of the conversion parameters managed in the storage unit 1503 can also be reduced, and the reading speed of the conversion parameters can also be increased. Note that in the first embodiment, the parameter determining unit 1502 determines the state of the object (whether or not a mask is worn) by a method such as template matching. However, the parameter determination unit 1502 may also be configured by a deep neural network, similar to the second and third feature conversion units. Similarly, the first feature converter may also be configured as a deep neural network. A specific state determination method will be described later using FIG. 21.

In this way, by holding conversion parameters specific to a specific state, matching that is robust against changes in state can be achieved. In addition, even if state determination fails, since all transformation parameters share the feature space, feature quantity transformation that has failed significantly is not performed. Therefore, matching that is robust to the performance of state determination can be achieved. Further, in order to enhance this property, each transformation parameter may be learned so that it can perform feature amount transformation for images other than the corresponding state to some extent. For example, in addition to images in a corresponding state as learning data, a small amount of images in other states may be included for learning. Alternatively, learning may be performed by changing the loss function, such as reducing the loss value, in other states.

Next, the procedure of the verification process will be explained using FIG. 17. In this process, one-by-one images are input, and it is determined whether the images are the same object. In this example, the state determined by the parameter determination unit 1502 will be described as "gender."

In S1701, the first image acquisition unit 101 acquires a first image (first image) including a person.

In S1702, the first feature converting unit 1501 converts the first image into an intermediate feature (first intermediate feature).

In S1703, the parameter determining unit 1502 determines whether the image is in the first image state (first state) based on the first intermediate feature amount. Specifically, it is determined whether the gender of the person appearing in the first image is male (or not female).

In S1704, the parameter determination unit 1502 reads the conversion parameter corresponding to the first state from the storage unit 1503 based on the determination result, and sets it in the second feature value conversion unit 1504.

In S1705, the second feature amount conversion unit 1504 converts the first intermediate feature amount to obtain a facial feature amount (first facial feature amount). Here, if the first state is male according to the determination result in S1703, the second feature conversion unit 1504 uses a learned model that is set with parameters that are good at identifying males. , we will get features from the image.

In S1706, the second image acquisition unit 102 acquires a second image (second image) including a person.

In S1707, the first feature converting unit 1501 converts the second image into an intermediate feature (second intermediate feature).

In S1708, the parameter determination unit 1502 determines the state of the second image (second state) from the second intermediate feature amount. Specifically, it is determined whether the gender of the person appearing in the second image is male (or not female).

In S1709, the conversion parameters corresponding to the second state are read from the storage unit 1503 and set in the third feature value conversion unit 1505.

In S1710, the third feature amount conversion unit 1505 converts the second intermediate feature amount to obtain a facial feature amount (second facial feature amount). Here, if the first image and the second image are both images of men, the parameters of the trained model set in the second feature conversion unit 1504 and the third feature conversion unit 1505 will be the same. . On the other hand, for example, if the first image is a male image and the second image is a female image, the parameters of the learned model set in the second feature conversion unit 1504 and the third feature conversion unit 1505 are different. .

In S1711, the feature amount matching unit 107 calculates the similarity score of the two feature amounts obtained in S1705 and S1710. By subjecting the similarity score to threshold processing, it is possible to determine whether or not the person appearing in the two images is the same.

Next, using FIG. 18, a verification processing procedure different from that in FIG. 17 will be explained. When the condition determined by the parameter determining unit 1502 is race, gender, etc., it can be determined that the person is a different person if the condition is different. In this process, the states of the two images are determined in advance, and if it is determined that the judgment result regarding the state of the object included in the images is high and the states are different, then the state of the facial features is determined. Skip the conversion process. This can reduce processing. Further, when it is determined that both sheets are in the same state, processing can be reduced by reading out the conversion parameters at once.

Steps S1801 to S1803 in FIG. 18 are the same as steps S1701 to S1703 in FIG. ). In S1804 to S1806, similarly to S1706 to S1708, the first feature conversion unit 1501 converts the second image into intermediate feature amounts to obtain the state of the second image (second state).

In S1807, the parameter determining unit 1502 determines whether the first state and the second state obtained in S1803 and S1806 are the same. If they are the same, the process moves to S1808; otherwise, the process moves to S1812.

In S1808, the parameter determining unit 1502 reads the conversion parameters corresponding to the first state from the storage unit 1503 and sets them in the second feature converter 1504 and the third feature converter 1505.

In S1809, the second feature amount conversion unit 1504 converts the first intermediate feature amount into a facial feature amount (first facial feature amount).

In S1810, the third feature amount conversion unit 1505 converts the second intermediate feature amount into a facial feature amount (second facial feature amount).

In S1811, the feature matching unit 107 calculates a similarity score between the first facial feature and the second facial feature.

In S1812, it is determined whether the state score (state score) output by the parameter determination unit 1502 is high. Therefore, the parameter determination unit 1502 is configured to output the score together with the state. For example, the parameter determination unit 1502 is configured as a deep neural network and configured to obtain an output for each state. Then, learning is performed so that the output corresponding to the state of the image is the largest. The state may be determined as the state in which the output is maximized, and the output value may be used as the state score. A specific method for determining the state score will be described later using FIG. 21. If the state score is greater than the predetermined threshold, the process moves to S1813. Otherwise, the process moves to S1814.

In S1813, the feature matching unit 107 outputs the similarity between the first image and the second image as zero. In other words, if the confidence level for state determination is greater than or equal to a predetermined value and the states (person attributes) of the objects are different, it can be determined that the objects are unlikely to be the same object.

In S1814, the parameter determination unit 1502 reads the conversion parameter corresponding to the first state from the storage unit 1503 and sets it in the second feature amount conversion unit 1504.

In S1815, the second feature amount conversion unit 1504 converts the first intermediate feature amount to obtain a facial feature amount (first facial feature amount).

In S1816, the conversion parameters corresponding to the second state are read from the storage unit 1503 and set in the third feature value conversion unit 1505.

In S1817, the third feature amount conversion unit 1505 converts the second intermediate feature amount to obtain a facial feature amount (second facial feature amount).

In S1818, the feature amount matching unit 107 calculates the similarity score of the two feature amounts obtained in S1815 and S1817. Similarly to the embodiment described above, two objects are determined to be the same if the similarity score is greater than or equal to a predetermined threshold, and are determined to be different objects if the similarity score is less than the threshold.

<1 to N image matching process>
FIG. 19 shows an example of the functional configuration of the image processing device 19. The basic configuration is similar to that shown in FIG. The difference is that a processing mode setting section 1901 and a feature amount registration section 1902 are provided. The flow of the verification process is shown in FIG. FIG. 20(A) shows the registration operation of a person, and FIG. 20(B) shows the matching operation between the input image and the registered person.

In the registration operation, the parameter determining unit 1502 determines a conversion parameter according to the race status of the registered person acquired in advance. This is because the race of the registered person can be accurately known at the time of registration, so there is no need to estimate it from the image. A specific process flow will be explained using FIG. 20(A).

In S2001a, the processing mode setting unit 109 sets the current operation mode to the registered operation mode.

In S2002a, the processing mode setting unit 109 acquires the racial status of the registered person. For example, a list of race status for each registered person is stored in advance in the storage unit H104 such as an HDD, and the list is acquired. Alternatively, the race status of the person to be registered is acquired from the acquisition unit H105 such as a keyboard.

S2003a is the start of a loop for sequentially processing registered persons. It is assumed that registered persons are assigned numbers sequentially starting from 1. In order to refer to a registered person using a variable i, first initialize i to 1. Further, when i is less than or equal to the number of registered persons, the process moves to S2005a, and when this is not satisfied, the process exits the loop and ends the process. Based on this, the corresponding conversion parameters are read from the storage unit 1503 and set in the second feature amount conversion unit 1504.

In S2005a, the first image acquisition unit 101 acquires a registered image of person i.

In S2006a, the first feature converter 1501 converts the registered image into an intermediate feature.

In S2007a, the second feature amount converting unit 1504 converts the intermediate feature amount to obtain a facial feature amount.

In S2008a, it is registered in the feature registration unit 1902 as a facial feature amount of person i. In addition, the racial status of person i is also registered.

S2009a is the end of the registered person loop, in which 1 is added to i and the process returns to S2003a.

Next, a comparison operation between an input image and a registered person will be explained using FIG. 20(B). During the matching operation, since the status of the input image, such as race, is unknown, processing is performed based on the status estimated from the image. Further, when the status of race, gender, etc. is different, it can be determined that the person is a different person. Therefore, when the state of the input image, such as race, can be estimated with high certainty, the processing speed is improved by narrowing down the registered persons to be matched. A specific process flow will be explained using FIG. 20(B). In this example, the state sought by the parameter determination unit 1502 is "race."

In S2001b, the processing mode setting unit 109 sets the operation mode to the verification operation mode. This prevents the status from being acquired from the processing mode setting unit 109.

In S2002b, the second image acquisition unit 102 acquires the inquiry image (second image).

In S2003b, the first feature converting unit 1501 converts the second image into an intermediate feature (second intermediate feature).

In S2004b, the parameter determination unit 1502 determines the state of the second image (second state) from the second intermediate feature amount. Specifically, the race of the person appearing in the second image is determined.

In S2005b, the parameter determining unit 1502 determines a conversion parameter corresponding to the second state from the storage unit 1503 according to the second state. The third feature conversion unit 1505 sets the determined conversion parameters to the (third) trained model.

In S2006b, the third feature amount conversion unit 1505 converts the second intermediate feature amount to obtain a facial feature amount (second facial feature amount).

In S2007b, it is determined whether the state score (state score) output by the parameter determination unit 1502 is high. If the state score is greater than the predetermined threshold, the process moves to S2008b. Otherwise, the process moves to S2009b.

In S2008b, the feature matching unit 107 narrows down registered persons in the same state as the second state as candidate persons. In other words, in this embodiment, the search is narrowed down to registered persons of the same race.

S2009b is the start of a loop for sequentially processing registered persons. If the registered persons have been narrowed down in S2008b, the feature value matching unit 107 sequentially performs a matching process on the narrowed down registered persons. Therefore, in order to sequentially refer to the registered persons using the variable i, numbers are first assigned sequentially starting from 1 to the registered persons to be processed, and i is initialized to 1. Further, when i is less than or equal to the number of registered persons to be processed, the process moves to S2010b, and when this is not satisfied, the process exits the loop and moves to S2012b.

In S2010b, the feature matching unit 107 obtains the facial feature of person i stored in the feature registration unit 1902. Then, the feature matching unit 107 calculates a similarity score between the second facial feature obtained in S2006b and the facial feature of person i.

S2011b is the end of the registered person loop, in which 1 is added to i and the process returns to S2009b.

In S2012b, the output unit 1900 outputs the result if there is a person whose similarity score obtained in S2010b is equal to or greater than a predetermined value. Note that the output unit 1900 outputs the matching result in the feature value matching unit 107, that is, the result of face authentication, to a display device or the like.

<Example of status determination method>
A method for determining the state from an image using the first feature converter 1501 and the parameter determiner 1502 will be described. The first feature converter 1501 and parameter determiner 1502 are configured using the above-mentioned DNN. The parameter determining unit 1502 is configured to set the number of outputs of the neural network to be the same as the number of states, and obtain the output through the Softmax function.

Next, it learns to determine states from images. In this embodiment, a state label is associated with each dimension of the output of the Softmax function of the parameter determination unit 1502, and learning is performed such that the corresponding state of the image takes 1 and the other states take 0. The learning flow will be explained using FIG. 21.

In S2101, a parameter set used by the first feature converter 1501 is initialized with random numbers or the like. Alternatively, initialization may be performed using a parameter set obtained by learning face recognition using the method described in FIG. 5(A) and the like described above.

In S2102, the parameter set used by the parameter determination unit 1502 is initialized with random numbers or the like.

In S2103, a group of facial images to which state labels have been added is acquired. For example, if the status is race, a group of facial images labeled with race are acquired.

In S2104, the parameter determination unit 1502 estimates the label of the state. Using an image as input, forward processing of the DNN is performed to obtain the value of the Softmax function.

In S2015, loss is calculated based on Equation 9, which is known as cross entropy.

(Formula 9)
Loss value = -Σp(i)log(q(i))
Here, p(i) indicates information on a correct label that takes 1 when the i-th state value is correct and takes 0 otherwise. q(i) indicates the value of the Softmax function corresponding to the i-th state.

In S2016, the parameter sets of the first feature converter 1501 and the parameter determiner 1502 are updated so that the loss value becomes smaller. By using an error backpropagation method commonly used in DNNs, the loss value is slightly updated in the direction of decreasing it.

In S2107, it is determined whether learning has ended. For example, when the amount of decrease in the loss value becomes small, it is determined that learning has ended. Alternatively, it may be determined that learning has ended when learning has been repeated a predetermined number of times. If learning is completed, the process moves to S2108. Otherwise, the process returns to S2103.

In S2108, the parameter set of the first feature converter 1501 is stored.

In S2109, the parameter set of the parameter determination unit 1502 is stored.

By using the parameter set of the first feature converter 1501 and the parameter determiner 1502 thus obtained, the state of the image can be determined. Specifically, the value of the Softmax function for the image is obtained, and it is determined that the state corresponds to the dimension that takes the largest value. Note that the value of the Softmax function obtained at this time takes a larger value when the degree of certainty is higher, so the value of the Softmax function can also be used as the state score.

As described above, the processing speed is increased by standardizing the state determination and the calculation of the intermediate feature amount of the feature amount conversion. In addition, the model size for state determination and feature value conversion can be reduced, and memory usage can also be reduced. Furthermore, since the conversion parameters managed in the storage unit 1503 can also be made smaller, the reading speed of the conversion parameters can be increased.

In addition, when differences in states such as race and age match differences in people, when it is determined with high certainty that the states are different, feature amount conversion is skipped and the degree of similarity is estimated to be low. This makes it possible to speed up the processing. Note that estimating the degree of similarity to be low based on the difference in state can be applied even in the case where the calculation of the intermediate feature amount between the state determination and the feature amount conversion is not standardized. In other words, the present invention is applicable even when the state determination and the feature amount conversion are both performed using images as input, as in the first and second embodiments. Further, as the state, an attribute that is difficult to change during a person's lifetime may be set. Alternatively, if the operating period is short, appearance attributes such as age, presence of beard, hairstyle, etc. may be used. Further, an alternative attribute such as skin color may be used instead of race. Therefore, the conditions of use are not limited to race or gender.

<Other derivative forms>
Although the description in this specification has focused on matching people, the present invention is applicable to various tasks related to matching identity and calculating similarity. For example, it can be applied to tasks such as detecting objects in a specific category, image inquiry tasks to extract designs of specific shapes from videos, and similar image searches.

The conditions determined by the condition determination unit 103 and the processing mode setting unit 109 include the image quality of the input image, the angle of view of the object, the size of the object, the clarity of the view of the object, the brightness and darkness of the illumination, the occlusion of the object, and the attachments of the object. the presence or absence of an object, the subtype of the object, or a combination thereof.

Further, here, two types of parameters are used depending on the state of the object, but it is also possible to use three or more types of parameters and switch between them.

In addition, although the embodiment of image recognition has been mainly illustrated here, collation and similarity search of information other than images, such as audio signals and music, can also be considered. By using the method of converting text into features as in Patent Document 2, it is also possible to apply it to tasks such as matching and searching for documents with similar meanings in text information such as books, SNS logs, and forms. It will be done. Note that since each category of books, SNS, etc. has its own unique vocabulary and format, there is room for performance to be improved by using different feature amount conversion means for each document category.

Further, in the embodiment, the explanation has mainly been given to checking whether or not objects are the same, but it is also possible to estimate the similarity value between objects by regression estimation. To do this, for example, the true similarity between the pair of object i and object j is given as a teacher value as shown in the following equation, and the loss value is defined as the square error with the estimated similarity score.

(Formula 10)
Loss value = Σ _i Σ _j (true pair similarity score (f _i, f _j )
- Pair similarity score (f _i, f _j )) ²
The parameters of the feature amount converter 105 and the feature amount converter 106 may be learned respectively so as to reduce this loss value. However, here, f _{i and} f _j are pairs of feature amounts of images transformed by the first trained model and the second trained model, respectively. As described above, it has been shown that the present invention is applicable to various tasks.

The present invention is also realized by performing the following processing. That is, software (programs) that implement the functions of the embodiments described above are supplied to the system or device via a data communication network or various storage media. This is a process in which the computer (or CPU, MPU, etc.) of the system or device reads and executes the program. Further, the program may be recorded on a computer-readable recording medium and provided.

1 Image processing device 101 First image acquisition section 102 Second image acquisition section 103 Object parameter determination 104 Storage section 105 First feature amount conversion section 106 Second feature amount conversion section 107 Feature amount matching section

Claims (24)

A first acquisition means for acquiring a first feature amount of a first object that matches a predetermined condition in a first image and is obtained using a first trained model. and,
a second acquisition of a second feature amount of the second object that does not match the predetermined condition in the second image, the second feature amount obtained using the second trained model; means and
Based on the first feature amount and the second feature amount, whether the first object in the first image and the second object in the second image are the same object. a verification means for determining whether the
has
When the first object and the second object are the same object, the first and second trained models are configured such that when the first object and the second object are the same object, the first and second trained models the first feature amount obtained using the trained model; and the second feature obtained using the second trained model as the feature amount of the second object that does not meet the predetermined condition. An image processing device characterized in that learning is performed so that a feature quantity becomes a similar feature quantity . further comprising determining means for determining whether the object in the image matches the predetermined condition;
When the determination means determines that the third object in the third image meets the predetermined condition, the second acquisition means acquires a third feature of the third object, obtaining a third feature obtained using the first trained model;
The matching means determines whether the first object in the first image and the third object in the third image are the same based on the first feature amount and the third feature amount. The image processing device according to claim 1, wherein the image processing device determines whether or not it is an object . The determination means includes the image quality of the input image, the angle of view of the object, the size of the object, the clarity of the view of the object, the brightness and darkness of illumination, the occlusion of the object, the presence or absence of attachments or attachments to the object, the subtype of the object, 3. The image processing apparatus according to claim 2, wherein it is determined whether or not an object in an image matches the predetermined condition based on a result of detecting at least one state. The image processing apparatus according to any one of claims 1 to 3, wherein the first object is a person wearing a mask. When the first object and the second object are the same object, the first object obtained using the first trained model as a feature of the first object that meets the predetermined condition. The first learning is performed so that the feature amount of the second object obtained by using the second trained model is similar to the feature amount of the second object that does not meet the predetermined condition. The image processing apparatus according to any one of claims 1 to 4, further comprising learning means for learning the trained model and the second trained model. The image processing apparatus according to claim 5 , wherein the learning means learns each of the first trained model and the second trained model based on a plurality of image groups. The plurality of image groups include a first image group serving as a reference and a second image group obtained by converting the reference image group,
The learning means is configured to determine the feature amount of the object included in the first image group when the object included in the first image group and the object included in the second image group are the same object. 7. The image processing apparatus according to claim 6, wherein the image processing apparatus learns so that the degree of similarity between and the feature amount of the object included in the second image group becomes larger than a predetermined value. 8. The image processing apparatus according to claim 7, wherein the second image group is an image group in which an attachment is synthesized with a specific object in the first image group. The image processing device according to any one of claims 1 to 8, wherein at least one of the first trained model and the second trained model is a neural network including a plurality of layers. The image according to any one of claims 1 to 8 , wherein the first trained model and the second trained model are neural networks in which parameters of some layers are shared. Processing equipment. The image processing apparatus according to any one of claims 1 to 9, wherein at least one of the first trained model and the second trained model is a transformer network. The image processing according to any one of claims 1 to 11 , wherein the second trained model is trained based on feature amounts extracted based on the first trained model. Device. 12. The image processing apparatus according to claim 1 , wherein the first trained model and the second trained model are trained simultaneously or alternately. a third acquisition unit that acquires intermediate feature quantities of the first image based on a third trained model that outputs feature quantities related to the state of the object from the image;
14. The method according to claim 1, further comprising parameter determining means for determining parameters of the first learned model based on intermediate feature amounts of the acquired first image. The image processing device according to item 1 . The third acquisition means further acquires an intermediate feature amount of the second image,
The parameter determining means determines parameters of the second trained model based on the intermediate feature amount of the acquired second image,
The parameters of the second trained model are such that the attribute of the object indicated by the intermediate feature of the first image is different from the attribute of the object indicated by the intermediate feature of the acquired second image. 15. The image processing apparatus according to claim 14, wherein if the first learned model is selected, parameters are determined to be different from parameters of the first trained model. 15. The first acquisition means acquires the first feature amount using an intermediate feature amount of the first image acquired by the third acquisition means. image processing device . 16. The second acquisition means acquires the second feature amount using an intermediate feature amount of the second image acquired by the third acquisition means. image processing device . a first acquisition means that acquires a first feature quantity from a first image based on a first trained model that extracts features from the image;
a second acquisition unit that acquires a second feature amount from the second image based on a second trained model that extracts features from the image, which is determined according to the state of the second image;
Comparing means for determining whether an object included in the first image and an object included in the second image are the same based on the first feature amount and the second feature amount. death,
The image processing device is characterized in that the second trained model is trained based on feature amounts extracted based on the first trained model. a first acquisition means that acquires a first feature quantity from a first image based on a first trained model that extracts features from the image;
a second acquisition unit that acquires a second feature amount from the second image based on a second trained model that extracts features from the image, which is determined according to the state of the second image;
a matching unit that determines whether an object included in the first image and an object included in the second image are the same based on the first feature amount and the second feature amount;
a third acquisition unit that acquires intermediate feature quantities of the first image based on a third trained model that outputs feature quantities related to the state of the object from the image;
An image processing apparatus comprising: parameter determining means for determining parameters of the first learned model based on intermediate feature amounts of the first image. 20. The second trained model is a model in which the second feature amount is learned in the same feature space as the first trained model. The image processing device described. a registration means for registering a first feature amount obtained using the first trained model for each image of a registered person that meets a predetermined condition ;
The first trained model or a second trained model different from the first trained model is selected depending on whether the person in the second image matches the predetermined condition. a selection means to
acquisition means for acquiring a second feature amount as a feature amount of the person in the second image based on the learned model selected by the selection means ;
a verification unit that determines whether the person in the second image is the same as any of the registered persons based on the first feature amount and the second feature amount ;
has
The first and second trained models meet the predetermined condition when the registered person who matches the predetermined condition and the person who does not match the predetermined condition in the second image are the same person. The first feature amount obtained by using the first trained model as the feature amount of the registered person who matches, and the second learned model as the feature amount of the person who does not match the predetermined condition. An image processing apparatus characterized in that learning is performed so that the second feature obtained by using the second feature becomes a similar feature . A first acquisition step of acquiring a first feature amount of a first object that matches a predetermined condition in the first image and is obtained using the first trained model. and a second feature amount of the second object that does not match the predetermined condition in the second image, the second feature amount obtained using the second trained model. an acquisition step; based on the first feature amount and the second feature amount, the first object in the first image and the second object in the second image are the same object; A verification step to determine whether or not there is a
has
When the first object and the second object are the same object, the first and second trained models are configured such that when the first object and the second object are the same object, the first and second trained models the first feature amount obtained using the trained model; and the second feature obtained using the second trained model as the feature amount of the second object that does not meet the predetermined condition. An image processing method characterized in that the image processing method is characterized in that learning is performed so that the quantity becomes a similar feature quantity . a registration step of registering a first feature amount obtained using the first trained model for each image of a registered person that meets predetermined conditions ;
The first trained model or a second trained model different from the first trained model is selected depending on whether the person in the second image matches the predetermined condition. a selection process to
an acquisition step of acquiring a second feature amount as a feature amount of the person in the second image based on the trained model selected in the selection step ;
a matching step of determining whether a person included in the second image is the same as one of the registered persons based on the first feature amount and the second feature amount ;
has
The first and second trained models meet the predetermined condition when the registered person who matches the predetermined condition and the person who does not match the predetermined condition in the second image are the same person. The first feature amount obtained by using the first trained model as the feature amount of the registered person who matches, and the second learned model as the feature amount of the person who does not match the predetermined condition. An image processing method characterized in that learning is performed so that the second feature obtained using the image processing method is similar to the second feature . A program for causing a computer to function as each means included in the image processing apparatus according to any one of claims 1 to 21 .

Images