TWI796072B - Identification system, method and computer readable medium thereof - Google Patents

Abstract

The present invention is an identification system and method thereof, including a data processing module, a feature extraction module, and a classification module. A person’s image of a person form a received image data is cut to form an input data by the data processing module. After a feature vector of the input data is captured by the feature extraction module, whether the person is a clerk with uniform or a consumer without uniform is determined based on the feature vector by the classification module and the flow of people can be counted. In the present invention, the problem that the conventional face recognition technology is not suitable for the crowd counting of retail stores can be solved. The present invention also provides a computer-readable medium for executing the method of the present invention.

Description

Identity recognition system, method and computer-readable medium thereof

The present invention relates to the technology of image recognition, in particular to a body recognition system, method and computer-readable medium for respectively recognizing people's bodies.

For a store, the number of visitors represents the popularity of the store and can also increase the chance of consumption. Therefore, how to correctly count the number of visitors to the store in a day or a period of time is very important for the self-evaluation of the store. In other words, in the application of smart retail, it is usually necessary to collect consumer behavior as a basis for improving the retail service experience. Traditionally, the collection of consumer behavior must be done through the collection of questionnaires or the identification devices such as counters worn by shop assistants. count. Today, the more common methods of collecting consumer behaviors are mostly using image recognition technology for people counting, hot spot identification, and consumer gender and age group identification, so as to obtain consumers' reach without affecting consumers. The purpose of store-related information.

When analyzing consumer-related information, the number of consumers who come to the store is the most basic indicator (Metric). However, there are usually consumers and shop assistants who provide services to consumers at the same time in the store, so how to distinguish consumption in the flow of people count In order to obtain the real number of consumers, it is one of the important themes of smart retail at present. In this regard, in order to provide smart retail counting of real consumers, a common method It uses Face Recognition (Face Recognition) technology to identify and deduct the clerk who has pre-registered Face ID (Face ID). However, due to the impact of the new crown pneumonia (COVID-19) epidemic, people will wear it in public. Wearing a mask will affect the recognition effect of face recognition. Furthermore, according to the experiment of the National Institute of Standards and Technology (NIST), the error rate of face recognition is between 5% and 50% when identifying the face wearing a mask. The improvement of face recognition technology has improved, but the face recognition still has the problem of being affected by the angle of the face. Therefore, the known face recognition technology still has deficiencies in identity recognition.

In addition, because retail stores usually wear uniforms, there is another method of clothing recognition (Clothes Recognition) to identify uniforms to distinguish them. Very demanding. In addition, when training the model, a static image of a model or a set of clothing is usually input, but consumers or shop assistants usually present dynamic situations in retail stores, so the conventional clothing recognition method is not suitable for Streaming videos recorded by Closed-Circuit Television (CCTV) in retail stores where people are crowded, angles are uncertain or covered, or, using Logo Recognition and Optical Character Recognition (Optical Character Recognition) Recognition) to identify the logo (Logo) or words on the uniform, also has the same problem as the aforementioned clothing recognition technology, that is, the method of logo recognition or optical character recognition still needs to input a clear and relatively complete image, so The same does not apply to the field of retail stores.

In view of the above problems, how to identify characters in larger-scale streaming videos, especially, even if the size of the characters displayed on the streaming video screen is relatively small or partially covered, can still obtain relatively rich information. Identifying relevant information will become the goal that those skilled in the art are eager to pursue.

In order to solve the above-mentioned problems in the prior art, the present invention discloses an identity recognition system, which includes: a data processing module, which is used to receive image data, and use a multi-object tracking algorithm to track people in the image data, so as to obtain information from the image data. The input data is formed by cutting out multiple person images corresponding to the person in the image data; the feature extraction module is coupled with the data processing module, and uses the person re-identification model to extract the feature vector of the person in the input data and a classification module, coupled with the feature extraction module, for inputting the feature vector into the classification model to determine the identity of the person.

In one embodiment, the data processing module includes a posture and direction estimation model for performing multi-directional sampling on the plurality of person images, wherein the posture and direction estimation model is used to perform direction recognition on the plurality of person images, so as to The input data is formed by filtering images of people in different directions.

In another embodiment, the image of the person in different directions includes the orientation of the person in front, back, left and/or right.

In another embodiment, the data processing module performs face anonymization and blurring on the plurality of person images to form the input data.

In another embodiment, the feature vector extraction includes obtaining feature vectors of the person's uniform, hat and/or identification card.

In yet another embodiment, during the model training process, the data processing module pre-marks the multiple person images according to whether the person wears a uniform or not to form a training sample for the person re-identification model to use the training samples for training.

The present invention further discloses a method for identifying an identity, which is executed by a computer. The method includes the following steps: receiving image data; using a multi-object tracking algorithm to track a person in the image data, so as to cut out the corresponding person from the image data Multiple character images are generated to form input data; through The person re-identification model extracts the feature vector of the person in the input data; and through the classification model, judges whether the person is a consumer or a shop assistant according to the feature vector.

In one embodiment, the step of forming the input data further includes performing multi-directional sampling on the plurality of person images through the pose and direction estimation model, so as to filter out the person images in different directions to form the input data. In another embodiment, the image of the person in different directions includes the orientation of the person in front, back, left and/or right.

In another embodiment, the step of forming the input data further includes performing face anonymization and blurring on the plurality of person images to form the input data.

In another embodiment, the feature vector extraction includes obtaining feature vectors of the person's uniform, hat and/or identification card.

In yet another embodiment, the present invention further includes in the process of model training, first marking the plurality of person images according to whether the person wears a uniform or not, so as to form a training sample, so that the person re-identification model can use the training sample to train.

The present invention further discloses a computer-readable medium, which is used in a computing device or a computer, and stores instructions to execute the aforementioned identification method.

As can be seen from the above, the identity recognition system, method and computer-readable medium of the present invention use the data processing module to trim the received image data to form input data, and the input data is processed by the feature extraction module. The feature vector extraction of the characters is carried out, so that the classification module can classify and count the characters according to the feature vectors, so as to judge the identity of the characters through uniform recognition to achieve the purpose of counting people.

1: Identity recognition system

11: Data processing module

12: Feature extraction module

13: Classification module

S201~S207: steps

S501~S507: Process

U, NU: space

FIG. 1 is a system architecture diagram of the identity recognition system of the present invention.

Fig. 2 is a flow chart of training the person re-identification model of the identity recognition system of the present invention.

Fig. 3 is a schematic diagram of classification training performed by the identity recognition system of the present invention.

4A-4B are schematic diagrams of classification inference performed by the identity recognition system of the present invention.

Fig. 5 is a flow chart of the identification method of the present invention.

The following describes the technical content of the present invention through specific embodiments, and those skilled in the art can easily understand the advantages and effects of the present invention from the content disclosed in this specification. However, the present invention can also be implemented or applied in other different specific implementation forms.

FIG. 1 is a system architecture diagram of the identity recognition system of the present invention. As shown in the figure, the identity recognition system 1 of the present invention includes a data processing module 11, a feature extraction module 12, and a classification module 13, wherein the data processing module 11 is used to cut out characters from the received image data The image is used to form the input data. After the feature extraction module 12 extracts the feature vector of the input data, the classification module 13 judges the identity of the person in the image according to the feature vector, such as a consumer or a shop assistant, or even Retail store managers, event promotional models, delivery staff, and people counting, in short, through the feature vector extraction to judge the identity, how to extract and identify the characteristics of the people in the image is the key, the following is the consumer and The identification of the clerk will be described as an example.

The description of the identity recognition system 1 of the present invention is described in detail as follows.

The data processing module 11 is used for receiving image data. In one embodiment, the data processing module 11 receives a streaming video recorded by an external closed-circuit television (CCTV) as image data, or It is the image data stored in the database built by the identity recognition system 1 of the present invention or captured by a camera. In other words, the data processing module 11 utilizes Multiple Object Tracking (Multiple Object Tracking, MOT) technology, wherein the multiple object tracking technology is specifically a multiple object tracking algorithm, which can target people (such as consumers or shop assistants) in the image data To track, specifically, the data processing module 11 can use the multi-object tracking algorithm to display the tracking of the person in the form of a frame to select the person. After the person is tracked by the multi-object tracking technology, the image data corresponding to The tracked person is cropped, that is, cropped according to the frame of the selected person. Since the image data is a continuous streaming video, after the person is cropped, multiple person images will be generated to form the input data.

In one embodiment, the data processing module 11 includes a posture and direction estimation model for multi-sampling a person from multiple person images. Since a single person is clipped from the image data, there are usually multiple types of people. Images are used as samples, that is, images of various types of people will include images of people in various directions or angles. Therefore, even after continuous or random sampling, it is still possible to obtain samples that are almost repeated in appearance. Therefore, in order to obtain as much as possible Characters face samples in different directions, so the present invention adopts a pose orientation estimation (Pose Orientation Estimation, POE) model for further analysis, so that the data processing module 11 first recognizes the orientation of the characters through the pose orientation estimation model, and then the multiple Multi-directional sampling is performed on the personal character image, and finally only the character images in different directions are retained, so that the input data is formed from the multi-directional sampling of the character images, that is, the multi-directional sampling of the present invention is aimed at the front and back of the character. , left, and/or right orientations, specifically, samples of the directions in which the person faces, for example, the front, the rear, the left, and/or the right. Accordingly, by providing multi-directional person images of a single person, the correctness of the recognition results of the body recognition system 1 of the present invention can be improved, and redundant similar data can be eliminated, which can help reduce the amount of data calculation.

In one embodiment, the data processing module 11 performs face anonymization and blurring on the plurality of person images to form the input data, that is, the step of processing the person images by the data processing module 11 also includes: The face parts in the person image are blurred so that the body recognition system 1 does not need to recognize facial features. The present invention blurs the face parts of the person in the person image so that the feature extraction module 12 can When performing feature vector extraction, it can focus on parts other than the face, such as clothes, hats or identification cards, or even the logo (LOGO) part on the clothes, making the recognition effect of the identity recognition system 1 of the present invention more accurate ; In addition, through the blurring of the faces of the characters, the purpose of protecting the privacy of consumers can be achieved at the same time.

The feature extraction module 12 uses a Person Re-identification (Person Re-identification) model to extract the feature vector of the person in the input data, wherein the feature vector extraction of the person includes obtaining the uniform and LOGO on the person. (such as a retail store's trademark, store logo, etc.), hat and/or identification card, etc., which can be used to identify the feature vector other than the face of the person (such as a person who is a clerk or a consumer).

The classification module 13 is to input the feature vectors extracted by the person re-identification model into the classification model, so that the classification model judges whether the person corresponding to the input data is wearing a uniform and their identity according to the feature vectors, so as to classify, especially It is that the present invention classifies shop assistants and non-shop assistants (consumers), and can count the number of people after the classification, so as to achieve the purpose of counting the number of consumers.

Fig. 2 is a flow chart of training the person re-identification model of the identity recognition system of the present invention. As shown in the figure, before the operation of the identity recognition system, the person re-identification model can be trained in advance. In addition to establishing the person re-identification model, the person re-identification model can also be adjusted and improved, so as to achieve correct and accurate image data. The effect of accurate consumer counting. Its training steps are as follows.

In step S201, data are prepared. In addition to obtaining pre-stored image data from the database built by the identity recognition system of the present invention, this step can also receive images from the camera built by the system. The image data captured by the video camera or external CCTV device is cut out from the image data by a multi-object tracking algorithm as a sample.

In step S202, multi-directional sampling is performed. In this step, multiple person images corresponding to a person are used to identify the direction of the person through the pose and direction estimation model, so as to obtain images of the person facing different directions as samples. In one embodiment, as a result of multi-directional sampling, samples in front, back, left, and right directions can be retained.

In step S203, face anonymous blurring is performed. Since the present invention intends to make the person re-identification model focus on the feature extraction of the clerk's exclusive features such as the person's uniform, LOGO, hat or identification card, the face is anonymously blurred, wherein, after the face is anonymously blurred There is almost no impact on the recognition ability of person re-identification; moreover, because the identity recognition does not need to recognize the facial features, so in order to improve the recognition of the person re-identification model of the present invention in the actual field (such as in a retail store) Therefore, when processing the input data (i.e. samples), the face can be anonymously blurred. On the one hand, it can protect the privacy of consumers. On the other hand, it can make the person re-identification model focus on the characteristics of clothing for identification, reducing Dependence on facial features.

In step S204, mark the data. This step is to mark whether the person in the sample is wearing a uniform or not as a condition. In short, in order to distinguish whether the sample is wearing a uniform or not, and then estimate whether the person is a clerk, this step can first Use the marking method to mark the characters in the sample as wearing uniforms or not wearing uniforms.

In step S205, feature extraction. In this step, the sample is input into the person re-identification model to extract the feature vectors. Based on this, the person re-identification model can extract the feature vectors for the characters who wear uniforms or not, and then the characters who wear uniforms can be obtained. The eigenvector of .

In step S206, classify. In this step, the feature vector is input into a classification model such as a Support Vector Machine (SVM), so as to classify the samples into two categories: shop assistants or consumers. In addition, since the number of shop assistants in retail stores is usually far less than the number of consumers, the obtained data is Imbalanced Data. Therefore, when training the support vector machine, the category weight setting is used (for example, sklearn.svm. The category weight setting of the balanced (balanced) mode of SVC) to automatically increase the weight of the clerk category with fewer samples, so as to avoid the tendency of the person re-identification model to identify the person as a consumer category with more samples.

For example, if there are 1,000 samples of consumers and 100 samples of shop assistants in the image data used for training, at this time, if all the shop assistants are misjudged as consumers, the error rate is only 100/ (1,000+100), about 9.1%, and its error rate may be ignored. For this, the present invention increases the weight of the clerk, that is, the ratio of the weight of the consumer to the weight of the clerk is 1:10, so that 1,000 consumers The importance is the same as 100 clerks. Accordingly, if all clerks are misjudged, the error rate is 100*10/(1,000*1+100*10)=50%. In other words, if all clerks are misjudged , which is equivalent to misjudgment of half of the data, which increases the severity of misjudgment. Therefore, the present invention increases the severity of misjudgment by the model for samples of shop assistants with a relatively small number of samples by adjusting and setting weights, so as to avoid the problem of misjudgment. In a specific embodiment, the setting weight of the present invention can be automatically adjusted through a balanced (balanced) mode, that is, the balanced mode uses the formula: total number of samples/(number of categories*number of samples in this category), to achieve automatic basis The sample size is used to adjust the aforementioned weights. For example, following the aforementioned example, the consumer weight = (1,000+100)/(2*1,000)=0.55, and the clerk weight = (1,000+100)/(2*100 )=5.5, therefore, the ratio of consumer weight to clerk weight is 0.55:5.5, which is 1:10, which is the same as the previous example, so that the model imbalance caused by misjudgment can be avoided.

In step S207, the weights are optimized. This step calculates the loss value (Loss) based on the classification result and the ground truth, updates the weights (Weight) of the person re-identification model and the classification model to make the Loss smaller, and then trains repeatedly until the termination condition is reached. Among them, in the ground truth In terms of category calculation, t is the true category of the sample, which can be 1 and -1 to represent the clerk (1) and the consumer (-1) respectively. When the result of the classification model prediction is y=0.7, and the real category of the sample The category is t=1, then the loss is: max(0,1-t*y)=max(0,1-1*0.7)=0.3, which means that if y is closer to t, the loss will be smaller.

Next, in order to enable the person re-identification model of the body recognition system of the present invention to correctly classify after training, the training is performed as follows.

As shown in FIG. 3 , it is a schematic diagram of classification training performed by the identity recognition system of the present invention. In order to allow the classification model that performs identity recognition to be classified correctly after training, the training goal is set to train the full-body feature extraction ability of the person re-identification model and ensure that the following conditions are true: (a) when the person is similarly dressed and is the same person, the feature The vectors should be almost the same; (b) when the clothes are similar, even if they are different individuals, the feature vectors should be similar; (c) when the clothes are different, the feature vectors should not be similar.

Firstly, input the labeled uniform samples (samples 1-3) and non-uniform samples (samples 4-6) into the person re-identification model to train the person re-identification model’s ability to extract the full-body features of the person, so that the person re-identification For the model, the feature vectors extracted by the same person who are similarly dressed are the same or nearly the same, the feature vectors extracted by different individuals who are similarly dressed are similar, and the feature vectors extracted by different individuals are not similar.

Afterwards, the training classification model is used to classify the feature vectors. Under the aforementioned classification conditions, the feature vectors corresponding to those who wear uniforms and those who do not wear uniforms will fall in the space U (Uniform, that is, those who wear uniforms) and the spaces separated by the decision boundary. Space NU (Non-Uniform, that is, not wearing a uniform).

Next, after the classification model is trained, inference can be performed, as shown in Figures 4A and 4B, which are schematic diagrams of classification inference performed by the identity recognition system of the present invention. The identity recognition system of this system performs multi-directional sampling and anonymous face blurring on the input samples, deduces the feature vectors from the input samples through the person re-identification model, and then uses the classification model to classify the feature vectors, for those who wear uniforms and those who do not wear uniforms For the uniform sample, as shown in Figure 4A, if the feature vector falls in the space U (Uniform), that is, the character is wearing a uniform; otherwise, if it falls in the space NU (Non-Uniform), that is, the character is not wearing a uniform, as shown in the figure 4B.

FIG. 5 is a flow chart of the identification method of the present invention. As shown in the figure, the identity recognition method of the present invention is implemented in electronic devices such as computers, servers, or cloud systems. Specifically, the identity recognition method of the present invention includes the following procedures.

In the process S501, image data is received. This process can obtain the stored image data from the database, and can also receive the image data captured from the installed camera or CCTV device.

In the process S502, the person is tracked. This process is to track the person in the image data through a multi-object tracking algorithm, so as to cut out a plurality of person images corresponding to the person in the image data.

In the process S503, multi-directional sampling is performed. This process uses the pose direction estimation model to perform multi-directional sampling on the cropped images of multiple characters. Specifically, the multi-directional sampling is to sample the front, rear, left and/or right orientations of the character, based on The multiple person images sampled in multiple directions form the input data.

In process S504, the face is anonymously blurred. This process is to anonymously blur the faces of the multiple person images, making the facial parts blurred and unable to be identified, and the input data is formed based on this. The purpose is not only to protect personal information, but also to reduce the time required for feature extraction. Impact.

In the process S505, input data is formed. This process is to form the input data of the above-mentioned multiple person images that have been trimmed, multi-directionally sampled, and face anonymously blurred.

In the process S506, feature vector extraction is performed on the features of the person in the input data through the person re-identification model. The aforementioned feature vector extraction includes vector extraction of the character's uniform, LOGO, hat, and/or identification card.

In the process S507, it is judged whether the character wears a uniform and the identity of the character according to the feature vector through the classification model. This process is to use a classification model to determine the identity of the person image. In the present invention, it is mainly determined whether the person is a consumer or a shop assistant, and then the flow of people can be counted.

In one embodiment, in step S506 , before using the person re-identification model, the person re-identification model may be trained to obtain the person re-identification model or the optimization of the subsequent person re-identification model. Specifically, the training steps include data preparation, multi-directional sampling, anonymous face blurring, data labeling, feature extraction, classification, and weight optimization. The detailed content has been described in Figure 2 and the corresponding paragraphs, so it will not be repeated.

In one embodiment, the identity recognition system and method of the present invention are applied in actual application fields. In short, training is conducted in 9 large retail stores, and the training samples include 5,077 samples without uniforms and 755 samples with Wearing uniform samples, the present invention uses the trained person re-identification model to conduct 1,074 test samples in another retail store, among which the average precision rate (Precision) and recall rate (Recall) are both above 93%. Details As shown in Table 1 below.

In addition, the present invention also discloses a computer-readable medium, which is applied to a computing device or computer having a processor (for example, CPU, GPU, etc.) and/or memory, and stores instructions, and can be used by this computing device or The computer executes the computer-readable medium through the processor and/or memory, so as to execute the above-mentioned method and steps when executing the computer-readable medium.

In one embodiment, the modules, units, devices, etc. of the present invention include a microprocessor and a memory, and algorithms, data, programs, etc. are stored in the memory or chip, and the microprocessor can load data from the memory Or algorithms or programs for data analysis or calculation. In other words, the identity recognition system of the present invention can be executed on electronic devices, such as general computers, tablets or servers, which perform data analysis and calculation after receiving image data, so the procedures performed by the identity recognition system can be performed through software Designed and built on electronic devices with processors, memory and other components to run on various electronic devices; in addition, each module or unit of the identity recognition system can be composed of independent components, such as designed as a computer Devices, memory, storage, or firmware with processing units can all be components of the present invention, and related models such as person re-identification models or classification models can also be selected as software programs, hardware or firmware architectures presented.

To sum up, the identity recognition system and method of the present invention and its computer-readable medium are used to distinguish consumers and shop assistants by identifying uniforms and related features in retail stores, and perform character recognition on the whole body characteristics of people through person re-identification The extraction of the vector, and then input the feature vector into the classification model used to identify uniforms to judge the identity of the person, for example, the person is a consumer and a shop assistant; in addition, during the training process, the samples are marked, such as There are uniforms and non-uniforms, so that the model can learn and recognize the common characteristics of shop assistants such as uniforms, hats, identification cards, etc. Therefore, the present invention solves the problem that in the past, facial recognition could not identify shop assistants who did not register Face ID, and the recognition rate was affected by masks and face angles. In other words, even if there are new shop assistants who have not registered in any system, as long as the new Personnel wear shop assistant uniforms, and can still be identified as shop assistants by the ID system even when their backs are turned to the camera.

The above-mentioned embodiments are for illustrative purposes only, and are not intended to limit the present invention. Anyone skilled in the art can make modifications and changes to the above-mentioned embodiments without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention is defined by the scope of patent application attached to the present invention, as long as it does not affect the effect and implementation purpose of the present invention, it should be included in this disclosed technical content.

1: Identity recognition system

11: Data processing module

12: Feature extraction module

13: Classification module

Claims (11)

An identity recognition system, comprising: a data processing module, used to receive image data, and use a multi-object tracking algorithm to track a person in the image data, so as to cut out a plurality of person images corresponding to the person in the image data , and then perform face anonymization and fuzzing on the multiple person images to form input data; the feature extraction module is coupled with the data processing module, and uses the person re-identification model to perform feature vectors for the person in the input data and the classification module is coupled with the feature extraction module to input the feature vector into the classification model to determine the identity of the person, wherein, in the model training process, the category weight setting is adopted , to automatically increase the weight of the person category with the feature vector, so as to adjust the person re-identification model and the classification model. The identity recognition system as described in Claim 1, wherein the data processing module includes a posture direction estimation model for multi-directional sampling of the multiple person images, wherein the posture direction estimation model is for the multiple The direction recognition is performed on the person images to filter out the person images in different directions to form the input data. The identity recognition system according to Claim 2, wherein the images of the person in different directions include the orientation of the person in front, back, left or right. The identity recognition system according to Claim 1, wherein the feature vector extraction includes obtaining the feature vector of the person's uniform, hat or identification card. The identity recognition system as described in Claim 1, wherein, during the model training process, the data processing module pre-marks the plurality of person images according to whether the person wears a uniform or not, so as to form a training sample for the The person re-identification model is trained using the training samples. A method for identifying an identity, which is executed by a computer, and the method includes the following steps: receiving image data; using a multi-object tracking algorithm to track a person in the image data, so as to cut out a plurality of people corresponding to the person in the image data image, and then perform face anonymization and fuzzing on the multiple person images to form input data; use the person re-identification model to extract the feature vector of the person in the input data; and use the classification model to judge the person according to the feature vector The person's body is separated, wherein, in the model training process, the category weight setting is adopted to automatically increase the weight of the person category with the feature vector, so as to adjust the person re-identification model and the classification model. The identity recognition method as described in Claim 6, wherein the step of forming input data further includes performing multi-directional sampling on the plurality of person images through a posture and direction estimation model, so as to filter out person images in different directions to form the input data . The identity recognition method as described in Claim 7, wherein the images of the person in different directions include the orientation of the person in front, back, left or right. The identity recognition method as described in Claim 6, wherein the feature vector extraction includes obtaining feature vectors of the person's uniform, hat or identification card. The identity recognition method as described in Claim 6 further includes in the model training process, first marking the multiple person images according to whether the person wears a uniform or not, so as to form a training sample for the person re-identification model to use the training samples for training. A computer-readable medium, used in a computing device or a computer, stores instructions to execute the identity identification method described in any one of claims 6-10.

Images