KR20240087443A - Training method and apparatus of object search model for unsupervised domain adaptation - Google Patents

Abstract

A method of training an object detection model for unsupervised domain adaptation according to an embodiment includes detecting a source object of source data belonging to a source domain based on a trained object detection model, and adding the object detection model to the object detection model. Detecting a target object of target data belonging to a target domain based on a target domain, extracting source feature data from the detected source object and target feature data from the detected target object based on a feature extraction model, Determining a reliability score for the detected target object based on intermediate feature data generated from the source feature data and the target feature data, clustering the target object detected from the target data based on the target feature data (clustering), refining the cluster for the target object by applying uniqueness logic, and using an objective function based on the refined clustering result and the reliability score to determine the feature. It may include training an object identification model including an extraction model.

Description

Method and apparatus for training an object search model for unsupervised domain adaptation {TRAINING METHOD AND APPARATUS OF OBJECT SEARCH MODEL FOR UNSUPERVISED DOMAIN ADAPTATION}

A method and apparatus for training an object search model for unsupervised domain adaptation are provided.

Video data-based AI service can refer to a service that utilizes AI technologies to extract the types and characteristics of objects present in videos and images, calculates meaningful information from the extracted features, and provides it to users. Through this, AI service users can receive services such as medical, security, defect detection, crime recognition, and situational recognition with high recognition rates and accuracy. In addition, the technology to find a specific person from videos featuring people is a part of AI image processing technology among video data-based AI services, and is currently making rapid progress, with visual intelligence already achieving a visual recognition rate that exceeds the human level.

Technology for finding specific people from videos can be broadly divided into person re-identification technology and person search technology. Person identification technology has the disadvantage of requiring limited filming environments and conditions, such as using images of only one person or images of people previously detected in the entire image to find a given person. However, human search technology is different in that it detects people from images such as CCTV images that appear in multiple people and then searches for a given person among them, so there are no separate restrictions on the application of the technology. Therefore, human search technology can be more easily applied to real life than human identification technology and can be applied in a wider range of fields. In particular, human search technology can be applied to a wide range of security and surveillance systems that can recognize people or situations by analyzing images collected from CCTV, black boxes, etc. Meanwhile, person search technology is a technology that uses both person detection technology and person identification technology, and has much more complex and diverse challenging tasks than the above-mentioned person identification technology. Accordingly, human search technology requires the development of various technologies to solve challenging tasks.

Meanwhile, machine learning is a field of AI that learns using a source domain and can check the learning results using a target domain. In addition, machine learning can be broadly classified into supervised supervision and unsupervised learning methods. Supervised learning may correspond to a method of learning by providing a machine learning model with data and a label corresponding to the correct answer. Unsupervised learning may correspond to a method of classifying features by analyzing data without labels and learning using the classified features. People search technology can be applied to a machine learning model by applying supervised learning and unsupervised learning, or a learning method mixed with each other can be applied to a machine learning model.

The background technology described above is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and cannot necessarily be said to be known technology disclosed to the general public before this application.

Republic of Korea Patent Publication No. 10-2022-0068373 (announced on May 26, 2022) presents a pedestrian tracking device and method in a CCTV environment. Republic of Korea Patent Publication No. 10-2112859 (announced on May 13, 2020) presents a method of training a deep learning model for labeling work and a device using the same.

A method and apparatus for training an object search model for unsupervised domain adaptation according to an embodiment can improve the clustering process for labeling by applying uniqueness information when learning object identification, and the reliability calculated by a partial classification technique is applied. Performance degradation caused by domain gaps can be improved by using the objective function.

However, technical challenges are not limited to the above-mentioned technical challenges, and other technical challenges may exist.

A method of training an object detection model for unsupervised domain adaptation according to an embodiment includes detecting a source object of source data belonging to a source domain based on a trained object detection model, and adding the object detection model to the object detection model. Detecting a target object of target data belonging to a target domain based on a target domain, extracting source feature data from the detected source object and target feature data from the detected target object based on a feature extraction model, Determining a reliability score for the detected target object based on intermediate feature data generated from the source feature data and the target feature data, clustering the target object detected from the target data based on the target feature data (clustering), refining the cluster for the target object by applying uniqueness logic, and using an objective function based on the refined clustering result and the reliability score to determine the feature. It may include training an object identification model including an extraction model.

The training method of the object detection model may include performing supervised training on the object detection model using training data of the source domain.

The training method of the object detection model may further include performing unsupervised training of the object identification model using the source feature data and the target feature data after training the object detection model.

Determining the reliability score includes generating a classification result for the plurality of parts from the intermediate feature data based on a part classification model, and classifying the plurality of parts in the classification result. It may include calculating the reliability score based on the average of the predicted probability of a correct answer.

The step of classifying into parts includes training the part classification model in a direction to decrease the part loss function calculated based on the correct prediction probability of classifying the plurality of parts. It can be included.

The step of refining the cluster includes refining the cluster for the target object by grouping the clustered target object by image and applying uniqueness logic to each group, and negating the cluster based on the refined cluster. It may include resetting objects and positive objects.

The step of training the object identification model includes adjusting the parameters of the feature extraction model so that the distance between the feature vectors of the negative object and the reference object becomes larger, and the distance between the feature vectors of the positive object and the reference object becomes closer. An updating step may be further included.

The step of refining the cluster may further include training the feature extraction model in a direction that reduces the loss function calculated based on the refined clustering result.

The step of training the object identification model may further include training the object identification model in a direction to decrease the classification domain adaptation loss function calculated based on the reliability score and the refined clustering result.

The step of training the object identification model may further include training the object identification model in a direction to decrease the feature domain adaptation loss function calculated based on the reliability score and the distance value between domains.

An apparatus for training an object search model for unsupervised domain adaptation according to an embodiment includes a memory storing instructions for training an object detection model and an object identification model, and a processor executing the instructions stored in the memory, The processor executes instructions stored in the memory to detect a source object of source data belonging to a source domain and a target object of target data belonging to a target domain from the trained object detection model, and detects the source object based on the feature extraction model. Extracting source feature data from an object and target feature data from the target object, determining a reliability score for the target object based on intermediate data generated from the source feature data and the target feature data, and adding the target feature data to the target feature data. an object identification model comprising clustering the target object based on the target object, applying uniqueness logic to refine clusters for the target object, and the feature extraction model using an objective function based on the refined clustering result and the reliability score. can be trained.

The processor may perform supervised training on the object detection model using training data of the source domain and unsupervised training on the object identification model using the source feature data and the target feature data. .

The processor generates a classification result for a plurality of parts from the intermediate feature data based on a part classification model, and the reliability is based on the average of correct prediction probabilities for the plurality of parts in the classification result. A score can be calculated.

The processor may train the part classification model in a direction that decreases the part classification loss function calculated based on the correct prediction probability by which the plurality of parts are classified.

The processor groups the clustered target objects by image, applies uniqueness logic to each group, refines the cluster for the target object, and resets the negative and positive objects based on the refined cluster. .

The processor may update the parameters of the feature extraction model so that the distance between the feature vectors of the negative object and the reference object becomes larger, and the distance between the feature vectors of the positive object and the reference object becomes closer.

The processor may train the feature extraction model in a direction that reduces the loss function calculated based on the refined clustering result.

The processor may train the object identification model in a direction that reduces the classification domain adaptation loss function calculated based on the reliability score and the refined clustering result.

The processor may train the object identification model in a direction that reduces the feature domain adaptation loss function calculated based on the reliability score and the distance value between domains.

A computer-readable recording medium storing one or more computer programs according to an embodiment may include instructions for performing a method.

Figure 1 shows an example diagram of an unsupervised domain adaptation technique.
Figure 2 shows a block diagram of an apparatus for training an object search model for unsupervised domain adaptation according to an embodiment.
Figure 3 shows a flowchart according to domain criteria operated by a processor in a training device for an object search model for unsupervised domain adaptation according to an embodiment.
Figure 4 shows a flowchart of a method for training an object search model for unsupervised domain adaptation according to one embodiment.
Figure 5 shows an example of a clustering method and a method of refining clustered clusters among the training methods of an object search model for unsupervised domain adaptation according to an embodiment.
Figure 6 shows a flowchart of the inference step of a method for training an object search model for unsupervised domain adaptation according to one embodiment.
Figure 7 shows the results of comparing the performance of an object search model to which a training method according to an embodiment is applied with another object search model.
Figure 8 shows the results of comparing the performance of an object search model to which training methods according to an embodiment are sequentially applied.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and are intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or excessively formal sense. No.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

Human search technology can be broadly classified into two types depending on the learning method of the machine learning model. Person search techniques can be illustratively classified into Supervised Person Search and Weakly Supervised Person Search.

In supervised person search technology, both the machine learning model for person detection and the machine learning model for person identification can be learned using supervised learning. Map person search technology can show a high recognition rate when the probability distribution of the source domain and the probability distribution of the target domain match or are similar, and the bounding box label representing the person's location information and the ID label representing the person's identity information All can have characteristics provided to a machine learning model. Unlike person identification technology, mapped person search technology uses large-scale datasets and is an early technology that performs the two tasks of person detection and person identification using an end-to-end network. am. In addition, the supervised person search technology is an application technology that includes a technology that mitigates the impact of incorrect person detection results through a network using an attention structure and a technology that reflects the quality of the detected results in the person identification stage through a weighting technique. development has taken place. In addition, the supervised person search technology is developed to alleviate the high computational cost of the existing Faster R-CNN (Region-Based Convolutional Neural Network) by proposing an anchor-free machine learning model in the person detection stage. It was also done.

Meanwhile, the source domain and target domain may not always have the same probability distribution, and if the number of data is not sufficient, there may be cases where learning from a different domain may be necessary. Accordingly, the difference in probability distributions between the source domain and the target domain is called a domain gap, and the domain gap can cause performance degradation. However, in relation to map person search technology, technology has not been developed for cases where the probability distributions of the source domain and target domain do not match. Additionally, at the same time, supervised human search technology has the problem of incurring enormous economic and time costs as it generates labels from large-scale data sets for learning.

To solve this problem, weakly supervised human search technology has the characteristic of providing only bounding box labels without ID labels of the target domain. Accordingly, a weak supervised person search technology may use supervised learning for the person detection model and unsupervised learning for the person identification model. Weakly supervised person search technology has been developed through a Siamese network to learn the context of the background in which people appear in the clustering stage so that it is consistent for each object, and people who appear together in one video are identified in another video. We have developed technology for a clustering technique using the logic that it is highly likely to appear in . In addition, weakly supervised person search technology has also been developed to learn only the characteristics of people that deviate from background information using various contextual situations. However, weakly supervised human search technology still has the problem of having to generate bounding box labels in the target domain for learning, and there is also the problem that technology for the probability distribution mismatch between the source domain and target domain has not yet been developed.

The purpose of solving the problem of probability distribution mismatch between the source domain and the target domain may be to obtain good results when applying the test in the target domain. Accordingly, easily transferring knowledge acquired through the source domain to the target domain during domain adaptation can be a way to solve the above-mentioned problem. Research is being actively conducted to apply these domain adaptation techniques to the field of human identification technology rather than the field of human search technology.

Figure 1 shows an example diagram of an unsupervised domain adaptation technique.

According to one embodiment, the unsupervised domain adaptation technology may assume a situation where the source domain and the target domain are different. Bounding box labels can be detected in both the image of the source domain (e.g., source image) 110 and the image of the target domain (e.g., target image) 130, but the ground truth label for each bounding box is It can be assumed that it exists only in the source domain 110. In other words, training data belonging to the source domain may include a source image, a pre-given ground truth bounding box label for the source image, and a ground truth label for each ground truth bounding box.

By way of example, the source domain may represent a domain in which data (e.g., source data, source image) acquired in an arbitrary environment (e.g., source environment) is distributed, and the target domain may represent an environment different from the source environment (e.g., : may indicate a domain in which data (e.g., target data, target image) acquired from the target environment is distributed. Mathematically, the probability distribution of data belonging to the source domain and the probability distribution of data belonging to the target domain can be interpreted as different. For example, images acquired by a surveillance camera and images acquired by a smartphone camera exhibit different characteristics and thus may belong to different domains.

For example, a machine learning model (e.g., the object detection model in Figure 2 below) can be trained using training data belonging to the source domain described above. For example, the machine learning model may generate detection results of bounding box labels and/or extraction results of ID labels (ID1, ID2, ID3, ID4) for each detected bounding box from each image (111, 112) of the source domain. You can. The machine learning model may extract features of the source object based on the detection result of the bounding box label and/or the extraction result of the ID label (ID1, ID2, ID3, ID4) for each detected bounding box. In the source domain space 120, a distribution using the feature vector of the source object extracted from each image 111 and 112 in the feature space may appear. The machine learning model can classify the feature vector of the source object into groups 121 and 122 for each image. For example, image 111 may be classified into group 121 and image 112 may be classified into group 122. The machine learning model trained as described above can also be used for images belonging to the target domain. Using a machine learning model trained on the source domain 110 on data (e.g., images) 131, 132, 133 belonging to the target domain 130 is based on knowledge learned using the source domain 110. (knowledge) may be interpreted as being transferred to the target domain 130. Therefore, the images 131, 132, and 133 belonging to the target domain for which it is difficult to secure a true value label or for which a true value label cannot be secured are grouped 141, 142, 143). In the target domain space 140, a distribution using the feature vector of the target object extracted from each image 131, 132, and 133 in the feature space may appear. In the target domain space 140, the feature vector of the target object may be classified into groups 141, 142, and 143 for each image using knowledge transferred from the source domain 110. For example, image 131 may be classified into group 141, image 132 may be classified into group 142, and image 133 may be classified into group 143. Accordingly, the target domain may not require a label during the training process of the machine learning model.

Unsupervised Domain Adaptive Person Re-Identification may include reducing the distance (e.g., difference) between the source domain and the target domain while the target domain is unlabeled. One way to reduce the distance (e.g. difference) between the source domain and the target domain is to use a generative model to transform the style of the source domain to be like the target domain or to use the source domain and target domain together for learning. However, the unsupervised domain adaptive person identification technology is a person identification technology, not a person search technology, and does not have a person detection step, so there is a problem that it can only be applied in limited situations where only one person is present in the video. Accordingly, the present invention derived a solution to the above-mentioned problems by applying unsupervised domain adaptive human identification technology.

Figure 2 shows a block diagram of an apparatus for training an object search model for unsupervised domain adaptation according to an embodiment.

An electronic device (eg, an object search model training device 200) according to an embodiment may include a communication unit 210, a memory 220, and a processor 230. The object search model training apparatus 200 may train an object search model before searching for an object (eg, a person). The object search model may include an object detection model and an object identification model. For example, the electronic device may detect an object from an image using a trained object detection model and identify the object detected by the object detection model using the trained object identification model.

The communication unit 210 is connected to an external device and can receive images necessary for training and inference of an object search model from the external device. In addition, the object search model training device 200 can also be used as an object search device as described in FIG. 6 below, and thus can receive images requiring object search through the communication unit 210.

The memory 220 may store instructions for training an object detection model and an object identification model, and hyperparameter values required for training. According to one embodiment, hyperparameters are values that the user directly sets when modeling. and Includes but is not limited to this. Additionally, the memory 220 may temporarily or long-term store data received from an external device through the communication unit 210. The memory 220 may be a non-volatile storage device or a volatile storage device that continues to retain stored information even when power is not supplied. For example, memory 220 may include compact flash (CF) cards, secure digital (SD) cards, memory sticks, solid-state drives (SSD), and micro SD cards. It may include magnetic computer storage devices such as NAND flash memory, hard disk drives (HDD), and optical disc drives such as CD-ROM, DVD-ROM, etc. there is.

Instructions may include all operations performed by the processor 230. The processor 230 may execute instructions stored in the memory 220 to train an object detection model and an object identification model.

Figure 3 shows a flowchart according to domain criteria operated by a processor in a training device for an object search model for unsupervised domain adaptation according to an embodiment.

The processor 230 may perform operation 3100 using the object detection model 3110 and may perform operation 3200 using the object identification model 3210.

In operation 3100, the training device may perform supervised training of the object detection model 3110 using training data belonging to a source domain. The supervised training process can be performed using a source domain with the correct answer. The object detection model 3110 for which supervised training has been completed can also be used for data in the target domain for which there is no correct answer when training the object identification model 3210, which will be described later.

The object detection model 3110 may be designed and trained to output a bounding box and object information (e.g., an ID label indicating the type of object) for the input image. For example, if the input is a source domain (e.g., an entire scene image), the output of the object detection model 3110 is the source bounding box label where the object is predicted to be and the source object detection ID label. (e.g. type of detected object) may be included. For example, the processor 230 may train the object detection model 3110 using ground truth (GT) labels (e.g., bounding box labels and object detection ID labels) of the source domain. For reference, in human search, the detected object must be a person, and if it is not a person, it may be treated as a background. The training device may train the object detection model 3110 by processing the bounding boxes corresponding to the remaining object detection ID labels as background, other than the bounding box in which the object detection ID label is detected to indicate a person.

Meanwhile, the object detection model 3110 on which training has been completed can be used in a training operation of the object identification model 3210 and/or an inference operation using an object search model, which will be described later. At this time, data belonging to the target domain rather than the source domain may be input to the object detection model 3110. When the input is a target domain, the output of the object detection model 3110 may be an object detection result (eg, target bounding box label and target object detection ID label). The processor 230 may perform operation 3200 using an object detected in the target domain.

Additionally, the object detection model may include a Faster R-CNN model. Faster R-CNN can be composed of a deep learning-based Region Proposal Network (RPN) and a Fast R-CNN model. A deep learning-based region proposal network (RPN) can detect objects by selectively searching regions that are likely to contain objects. Then, the Fast R-CNN model can receive the detected object and predict the object's class and location. Faster R-CNN is a very fast single integrated network, so you only need to train and run one network.

In operation 3200, the training device may perform unsupervised training of the object identification model 3210 using the source domain, target domain, and intermediate domain. The unsupervised training process can be performed using the object identification model 3210, and the inference step is described later in FIG. 6.

The processor 230 may train the object identification model 3210 to identify the object detected by the object detection model 3110. The object identification model 3210 is designed to receive an image (e.g., a patch image corresponding to the bounding box) corresponding to the object detection result (e.g., a bounding box classified as a person) and output an object identification ID label. and can be trained. As will be described later, the object identification ID label may be a type of temporary identifier for distinguishing objects in bounding boxes detected in multiple images, rather than information indicating the actual identity of the object. For example, among bounding boxes detected in a plurality of images, the same object identification ID label may be assigned to bounding boxes that are determined to contain the same object. For another example, different object identification ID labels may be assigned to bounding boxes determined to contain different objects among bounding boxes detected in a plurality of images.

The object identification model 3210 according to one embodiment includes a feature extraction model 3211, an intermediate domain module (IDM) 3212, a GAP layer 3213, an FC layer 3214, a part classification model 3215, and a clustering algorithm. It may include (3216). The processor 230 may extract features of the source object and target object detected by the object detection model 3110 using the feature extraction model 3211. The source object may represent an object detected based on the object detection model 3110 from the source image, and the target object may represent an object detected based on the object detection model 3110 from the target image. The processor 230 extracts feature data (e.g., feature vector) for the source object by inputting the source patch image generated by cropping the portion corresponding to the source object from the source image into the feature extraction model 3211. can do. Similarly, the processor 230 may extract feature data for the target object by inputting a target patch image generated by cropping a portion corresponding to the target object from the target image into the feature extraction model 3211.

The processor 230 may generate intermediate feature data by mixing source feature data and target feature data extracted by the feature extraction model 3211. Intermediate feature data may represent feature data belonging to an intermediate domain. Exemplarily, the processor 230 may generate intermediate feature data by mixing source feature data and target feature data based on the IDM 3212. Generation of intermediate feature data based on IDM (3212), 'Dai, Y.; Liu, J.; Sun, Y.; Tong, Z.; Zhang, C.; and Duan, L.- Y. 2021. Idm: An intermediate domain module for domain adaptive person re-id. Reference may be made to 'In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 11864-11874'. The intermediate domain may represent a domain that mediates the source domain and the target domain. Knowledge trained using the source domain can be easily transferred to the target domain during domain adaptation via an intermediate domain.

The processor 230 may individually pass the source feature data, target feature data, and intermediate feature data through a global average pooling (GAP) layer 3213 to reduce the number of features. The processor 230 can directly apply intermediate feature data with a reduced number of features to the domain adaptation loss function. Additionally, the processor 230 may directly apply source feature data and target feature data with a reduced number of features to the feature extraction loss function. The fully-connected (FC) layer 3214 can output the result of multiplying the last feature with a matrix. Accordingly, the processor 230 passes the source feature data, target feature data, and intermediate feature data with a reduced number of features through the GAP layer 3213 individually through the FC layer 3214 to output result values in the form of a matrix. there is. The results in the form of a matrix output in this way can be used to calculate the values of the domain adaptation loss function and the feature extraction loss function, respectively.

The processor 230 may apply the above-described intermediate feature data to the part classification model 3215 to classify it into a plurality of parts. According to this classification result, the processor 230 may apply the correct prediction probability for a plurality of parts to the part classification loss function. The processor 230 may train the part classification model 3215 using a part classification loss function. Additionally, the processor 230 may calculate a reliability score based on the average of the correct prediction probabilities for a plurality of parts. Additionally, the processor 230 may pass the target feature data through the clustering algorithm 3216 to perform clustering on the target object based on the target feature data. The processor 230 may refine the clustered cluster to form a refined cluster. The processor 230 may train the feature extraction model 3211 by applying the refined clustering result to the feature extraction loss function. Additionally, the processor 230 may train the object identification model 3210 by applying the refined clustering result and the above-described reliability score to the domain adaptation loss function. As a result, the processor 230 can train the object identification model 3210 while also training models within the object identification model 3210. In addition, operations by the above-described feature extraction model (3211), IDM (3212), GAP layer (3213), FC layer (3214), part classification model (3215), and clustering algorithm (3216) may be performed simultaneously. It is not limited and can be done sequentially.

Figure 4 shows a flowchart of a method for training an object search model for unsupervised domain adaptation according to one embodiment.

In step 410, the processor may detect the source object of source data belonging to the source domain based on the trained object detection model. The processor can perform supervised training on an object detection model using GT labels (e.g., bounding box label, ID label) of the source domain. The source object can use the GT label for training in the next step without being detected by the object detection model.

In step 420, the processor may detect the target object of target data belonging to the target domain based on the object detection model. There are multiple images in the domain and the object may be, for example, one or more people.

In step 430, the processor may extract source feature data from the detected source object and target feature data from the detected target object based on the feature extraction model. For example, the processor may extract source feature data and target feature data using a Re-ID model.

At step 440, the processor may determine a trust score for the target object based on intermediate feature data generated from the source feature data and the target feature data. The processor may generate intermediate feature data by inputting source feature data and target feature data into the IDM. The reliability score is described later in Equations 1 and 2.

In step 450, the processor may cluster target objects detected from target data based on target feature data. The processor can assign a pseudo label to the target object through clustering.

At step 460, the processor may apply uniqueness logic to refine the cluster for the target object. Uniqueness logic implies that “the same person cannot appear in one video,” and the method of refining a cluster by applying uniqueness logic is described later in Figure 5.

At step 470, the processor may train an object identification model including a feature extraction model using an objective function based on the refined clustering results and reliability scores. The objective function is a function that must find the maximum or minimum value in the optimization algorithm of the trained model, and a loss function can be used in the present invention. Loss (or error) can be measured as a number between 0 and 1, with 0 being a perfect model. Accordingly, the model can be optimized so that the loss is as close to 0 as possible. The objective function based on the reliability score is described later in Equation 4.

Detecting objects in a target domain through an object detection model has a high probability of inaccurate object detection results because there is no bounding box label information in the target domain. Additionally, object identification using the object identification model is performed based on objects detected using the object detection model, which may result in successively lower identification results. To solve this problem, the processor can check the object detection result by calculating a reliability score value. The processor may determine the reliability score based on the matrix value of the part classified by the part classification model.

The part classification model can designate multiple divided parts as one part by applying partial max pooling to intermediate feature data. For example, a part classification model can divide a detected object (e.g., a person) horizontally and label each part. So, for example, the first label could be 0, and the second label could be 1. Additionally, the processor can classify parts using part classification consisting of one layer of a multi-layer perceptron (MLP). Given a mini-batch, the output of the part classification model is P , and P is The prediction score matrix of can be expressed. can be the number of parts, elements of matrix P May represent the probability that the i-th part is classified as the j-th part. Additionally, the processor can train the part classification model in a direction that reduces the loss function of the part classification model. The loss function of the part classification model can be expressed as Equation 1.

In the above equation 1, can represent the correct answer label (GT label) of the ith part, can represent the predicted probability of a correct answer, can be 0.04 as a hyperparameter.

Meanwhile, the reliability score can be determined as the average of the predicted probabilities and can be expressed as Equation 2.

In the aforementioned equation 2, is the aforementioned In the same way as, the prediction probability of the correct answer can be expressed, and the reliability score may also indicate the degree of classification of the part. Therefore If the value is 1, this may mean that all parts are well classified and the detected target object is accurately detected. if If the value is less than 1, it may mean that the detected target object was detected incorrectly. For example, if the detected target object is a background rather than a person, it may be difficult for the part classification model to classify the part, thus lowering the probability of accurate classification. can be smaller than 1.

The processor needs an ID label to train an object identification model, but since the target domain of the present invention does not have an ID label, assignment of an ID label is necessary. The processor may assign capital labels (e.g., object identification ID labels) using a clustering algorithm called density-based spatial clustering of applications with noise (DBSCAN). Accordingly, the processor can perform clustering using the density difference. However, this clustering method has the disadvantage of low clustering accuracy. Therefore, in order to improve this, the present invention can refine the clustered cluster by applying uniqueness logic.

Figure 5 shows an example of a clustering method and a method of refining clustered clusters among the training methods of an object search model for unsupervised domain adaptation according to an embodiment.

(510), (520) are images (510) and (520) in feature space ( ) can represent the distribution using the feature vector of the target object detected from. The processor may assign an ID label based on the clustering result. For example, the processor may designate the anchor object at (510) and (520) as the (511) object. The density of clustering can be centered around transportation characteristics. Accordingly, the clustering algorithm can classify (511) objects, (512) objects, (513) objects, and (514) objects that use the same means of transportation as the same object and cluster them into one cluster. Since the same ID label is assigned to one cluster, the (511) object, (512) object, (513) object, and (514) object can have the same ID label. To optimize the feature extraction model, the processor may designate the object with the closest distance among objects with different ID labels as a negative object. Accordingly, at 510, the processor may designate the object (515) as a negative object. Additionally, the processor can designate the object with the greatest distance among objects with the same ID label as a positive object. Accordingly, at 510, the processor may designate the object 512 as a positive object.

However, although the (511) object and the (513) object are different objects, they are within the same cluster, so distinction is necessary. To this end, the processor can group each object within the cluster by image. Accordingly, according to the uniqueness logic that “people who appear in the same video are different people,” one image ( In the group 521 for ), objects other than (511), which are standard objects, can be excluded from the cluster. Therefore, the refined clustering result is that the (511) object, (512) object, and (514) object may have the same ID label. According to these results, the processor can change the negative object from the (515) object to the (513) object.

Meanwhile, the feature extraction model can extract features of the source domain, target domain, and intermediate domain. The processor can train the feature extraction model in a direction that reduces the loss function calculated based on the refined clustering results. The loss function of the feature extraction model can be composed of a batch-hard triplet loss function and a cross-entropy-based classification loss function.

The batch-hard triplet loss function can be expressed as Equation 3.

In Equation 3 described above, the processor can randomly select K feature vectors from P IDs for a mini-batch with a batch size of PXK. is the reference object, is a positive object, can represent a negative object. D(·) represents the Euclidean distance, and m is a hyperparameter that can be specified by the user.

The processor uses a batch-hard triplet loss function to create a feature extraction model so that the distance between the (513) object (e.g., negated object) and the (511) object (e.g., reference object) in FIG. 5 increases. can be trained. Additionally, the processor can train the feature extraction model to make the distance between the (512) object (e.g., positive object) and (511) object (e.g., reference object) closer through the triplet loss (batch-hard triplet loss) function. there is. Additionally, the loss function is a function that quantifies the difference between the actual value and the predicted value, allowing the model to be optimized while reducing the loss. In addition, when a model is trained, parameters by training are determined, and optimizing the model may have the same meaning as optimizing the specified parameters. Accordingly, optimizing the model may mean updating parameters in a way that reduces the loss function. Ultimately, training a feature extraction model may mean updating the parameters of the feature extraction model.

The cross-entropy-based classification loss function can be expressed as Equation 4.

In the aforementioned equation 4, represents the probability of being the ith ID, and k may represent the source domain and target domain. In the case of a source domain, it can indicate that it is a correct ID, and in the case of a target domain, it can indicate that it is a false ID given as a result of clustering. N can represent the number of IDs, is a hyperparameter and can be 1. can be 1 because the correct answer cluster is always known when the input domain is the source domain. However, if the input domain is the target domain, uniqueness logic is applied, so if objects grouped in the same cluster (e.g. objects with the same ID) come from the same image, it may be 0. Additionally, when the input domain is the target domain, it may be 1 if objects grouped in the same cluster (e.g., objects with the same ID) come from different images.

Meanwhile, the object identification model can apply a domain adaptation loss function as the objective function. The domain adaptation loss function can be divided into a classification part and a feature part to calculate the loss. In the classification part, the domain adaptation loss function can be expressed as Equation 5 based on the above-described reliability score and refined clustering results.

In the aforementioned equation 5, can represent the predicted value that the feature of the intermediate domain has the label of the ith ID. can represent the weight of how much to mix the source domain and target domain when creating an intermediate domain, is a hyperparameter and can be 1.

Additionally, in the feature part, the domain adaptation loss function can be expressed as Equation 6 based on the reliability score and the distance value between domains.

In the aforementioned equation 6, represents the characteristics of the source domain and target domain, may represent the characteristics of the intermediate domain. The feature domain adaptation loss function can calculate the distance between features of the domain using L2 norm (Euclidean Distance). Accordingly, the feature domain adaptation loss function can calculate the distance between the features of the intermediate domain and the features of the source domain or the distance between the features of the intermediate domain and the features of the target domain. Reliability gamma can be calculated for each sample in the minibatch, and domain adaptation loss can be calculated for all samples in the minibatch. As a result, the domain adaptive loss function can limit the extent to which it affects model training when the reliability scores of mid-domain features are insufficient.

Figure 6 shows a flowchart of the inference step of a method for training an object search model for unsupervised domain adaptation according to one embodiment.

After training the object identification model, the processor can perform the inference step using test data. Test data may include queries containing an image of a single object (e.g., a person) and gallery images containing multiple object images. The inference step may be a test method that finds objects such as queries in gallery videos. When test data is input to the object detection model, the processor can detect test objects in the test data. The processor may extract test feature data 611 by inputting the detected test object into the trained feature extraction model 610. The processor may calculate similarity between feature data based on the extracted test feature data 611. The processor can find objects such as the query in descending order of calculated similarity. This allows users to accurately find objects, such as queries, among multiple objects. In the inference stage, unlike the training stage described above, the object identification model 600 may consist of only the feature extraction model 610.

Figure 7 shows the results of comparing the performance of an object search model to which a training method according to an embodiment is applied with another object search model.

The performance of the object search model to which the training method according to an embodiment of the present invention is applied can be confirmed using the benchmark datasets CUHK-SYSU and PRW. The CUHK-SYSU data set provides 18,184 images and can include 96,143 bounding box labels and 8,432 ID labels. Among them, the training data (e.g. source domain) is 11,204 images and includes 5,532 ID labels, and the test data (e.g. target domain) may include 6,978 gallery images and 2,900 query images. The PRW dataset is captured from images taken by 6 different cameras and can contain 11,816 image counts, 43,110 bounding box labels, and 932 ID labels. The training data is 5,704 images and includes 482 ID labels, and the test data can include 6,112 gallery images and 2057 query images. Additionally, mAP and Top-k were used as performance indicators. In the CUHK-SYSU dataset, the performance of supervised machine learning models and weakly supervised machine learning models can be achieved similar to that of target domains without labels. Additionally, the PRW dataset shows the best performance compared to other supervised machine learning models and weakly supervised machine learning models.

Figure 8 shows the results of comparing the performance of an object search model to which training methods according to an embodiment are sequentially applied.

In Figure 8, the baseline technique may mean applying only basic IDM to the object identification model. The with DA (domain adaptation) technique may mean applying a part classification model that calculates a reliability score using partial classification to an object identification model. The with CR (clustering refinement) technique may mean refining clusters by applying uniqueness logic and applying the results to an object identification model. CUHK -> PRW may mean setting the CHUCK data set as the source domain and setting the PRW data set as the target domain. Also, on the contrary, PRW -> CUHK may mean that the PRW data set is set as the source domain and the CHUCK data set is set as the target domain. The performance results show that each technique applied individually is also superior to the baseline. However, the final performance results show that the proposed technique (training method according to an embodiment of the present invention) in which both the with DA technique and the with CR technique are applied is the best.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. It may be possible. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (20)

In a method of training an object search model for unsupervised domain adaptation performed by a processor,
Detecting a source object of source data belonging to a source domain based on a trained object detection model;
detecting a target object of target data belonging to a target domain based on the object detection model;
extracting source feature data from the detected source object and target feature data from the detected target object based on a feature extraction model;
determining a reliability score for the detected target object based on intermediate feature data generated from the source feature data and the target feature data;
Clustering the target object detected from the target data based on the target feature data;
Applying uniqueness logic to refine a cluster for the target object; and
Training an object identification model including the feature extraction model using an objective function based on the refined clustering result and the reliability score.
Training method of object search model including. According to paragraph 1,
Performing supervised training for the object detection model using training data of the source domain
Training method of object search model including.
According to paragraph 2,
After training the object detection model, performing unsupervised training of the object identification model using the source feature data and the target feature data.
A training method for an object search model further comprising:
According to paragraph 1,
The step of determining the reliability score is,
generating a classification result for a plurality of parts from the intermediate feature data based on a part classification model; and
Calculating the reliability score based on the average of correct prediction probabilities for the plurality of parts in the classification result.
Training method of object search model including. According to clause 4,
The step of classifying into parts is,
Training the part classification model in a direction to decrease the part loss function calculated based on the correct prediction probability by which the plurality of parts are classified.
Training method of object search model including.
According to paragraph 1,
The step of refining the cluster is,
grouping the clustered target objects by image and refining clusters for the target objects by applying uniqueness logic to each group; and
Resetting negative objects and positive objects based on the refined cluster
Training method of object search model including.
According to clause 6,
The step of training the object identification model is,
Updating the parameters of the feature extraction model so that the distance between the feature vectors of the negative object and the reference object becomes larger and the distance between the feature vectors of the positive object and the reference object becomes closer.
A training method for an object search model further comprising:
According to paragraph 1,
The step of refining the cluster is,
Training the feature extraction model in a direction where the loss function calculated based on the refined clustering result decreases.
A training method for an object search model further comprising:
According to paragraph 1,
The step of training the object identification model is,
Training the object identification model in a direction to decrease the classification domain adaptation loss function calculated based on the reliability score and the refined clustering result.
A training method for an object search model further comprising:
According to paragraph 1,
The step of training the object identification model is,
Training the object identification model in a direction to decrease the feature domain adaptation loss function calculated based on the reliability score and the distance value between domains.
A training method for an object search model further comprising:
A computer-readable recording medium storing one or more computer programs including instructions for performing the method of any one of claims 1 to 10.
In an apparatus for training an object search model for unsupervised domain adaptation,
a memory storing instructions for training an object detection model and an object identification model; and
Processor executing instructions stored in the memory
Including,
The processor,
Detect a source object of source data belonging to a source domain and a target object of target data belonging to a target domain from the object detection model trained by executing instructions stored in the memory, and extract source features from the source object based on the feature extraction model. Extract target feature data from data and the target object, determine a reliability score for the target object based on intermediate data generated from the source feature data and the target feature data, and determine the target feature data based on the target feature data. An object that clusters objects, applies uniqueness logic to refine clusters for the target object, and trains an object identification model including the feature extraction model using an objective function based on the refined clustering results and the confidence score. Training device for search model.
According to clause 12,
The processor,
Training of an object detection model that performs supervised training on the object detection model using training data of the source domain and unsupervised training on the object identification model using the source feature data and the target feature data. Device.
According to clause 12,
The processor,
Generate classification results for a plurality of parts from the intermediate feature data based on a part classification model, and calculate the reliability score based on the average of correct prediction probabilities for the plurality of parts in the classification results. A training device for an object search model.
According to clause 14,
The processor,
A training device for an object search model that trains the part classification model in a direction in which a part classification loss function calculated based on the correct prediction probability by which the plurality of parts are classified decreases.
According to clause 12,
The processor,
Grouping the clustered target objects by image, refining clusters for the target objects by applying uniqueness logic to each group, and training an object search model that resets negative and positive objects based on the refined clusters. Device.
According to clause 16,
The processor,
A training device for an object search model that updates parameters of the feature extraction model so that the distance between the feature vectors of the negative object and the reference object becomes larger and the distance between the feature vectors of the positive object and the reference object becomes closer.
According to clause 12,
The processor,
A training device for an object search model that trains the feature extraction model in a direction in which the loss function calculated based on the refined clustering result decreases.
According to clause 12,
The processor,
A training device for an object search model that trains the object identification model in a direction that reduces the classification domain adaptation loss function calculated based on the reliability score and the refined clustering result.
According to clause 12,
The processor,
A training device for an object search model that trains the object identification model in a direction in which the feature domain adaptation loss function calculated based on the reliability score and the distance value between domains decreases.

Images