CN109829436B - A Multi-Face Tracking Method Based on Deep Apparent Features and Adaptive Aggregation Networks - Google Patents

Abstract

The invention relates to a multi-face tracking method based on deep apparent features and an adaptive aggregation network. First, a face recognition data set is used to train an adaptive aggregation network; then a face detection method based on a convolutional neural network is used to obtain the human face. The position of the face target to be tracked is initialized, and the face features are extracted; then the Kalman filter is used to predict the position of each face tracking target in the next frame, and the position of the face is located again in the next frame. Face extraction features; finally, an adaptive aggregation network is used to aggregate the face feature sets in the tracking track of each tracked face target, and dynamically generate a face depth apparent feature fused with multi-frame information. The position and the fused features are calculated and matched with the face position and its features obtained through detection in the current frame, and the tracking status is updated. The present invention can improve the performance of face tracking.

Description

A Multi-Face Tracking Method Based on Deep Apparent Features and Adaptive Aggregation Networks

technical field

The invention relates to the field of pattern recognition and computer vision, in particular to a multi-face tracking method based on deep apparent features and an adaptive aggregation network.

Background technique

In recent years, with social progress and the continuous development of technology, video face recognition has gradually become a hot research field, attracting the research interest of many experts and scholars at home and abroad. As the entrance and basis of video face recognition, face detection And tracking technology has developed rapidly and is widely used in intelligent monitoring, virtual reality perception interface, video conferencing and other fields. Because the real video background is complex and changeable, and the face is a non-rigid target, it may exist in the video sequence. With large pose or expression changes, it is still a great challenge to implement a robust face tracking algorithm in real scenes.

In order to analyze a face, we must first capture the face, which can be achieved through face detection technology and face tracking technology. Only by accurately locating and tracking the face target in the video, we can update the face. Detailed analysis, such as face recognition, pose estimation, etc. Target tracking technology is undoubtedly one of the most important technologies in intelligent security. Face tracking technology is a specific application of current tracking technology. It uses tracking algorithms to process moving faces in video sequences, and keeps track of this face area. This technology has good application prospects in scenarios such as intelligent security and video surveillance.

Face tracking plays an important role in video surveillance, but currently in real scenes, due to the large changes in face poses and the overlap and occlusion between tracking targets, practical applications are still difficult.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to propose a multi-face tracking method based on deep apparent features and an adaptive aggregation network, which can improve the performance of face tracking.

The present invention adopts the following scheme to realize: a multi-face tracking method based on deep apparent feature and self-adaptive aggregation network, which specifically includes the following steps:

Step S1: using the face recognition data set to train the adaptive aggregation network;

Step S2: According to the initial input video frame, use the convolutional neural network to obtain the position of the face, initialize the face target to be tracked, extract the face features and save;

Step S3: use Kalman filter to predict the position of each face target in the next frame, and locate the position of the face again in the next frame, and extract features for the detected face;

Step S4: Use the adaptive aggregation network trained in step S1 to aggregate the face feature set in the tracking track of each tracked face target, and dynamically generate a face deep apparent feature fused with multi-frame information. The predicted position and the fused feature are compared with the detected face position and its features in the current frame, and the similarity is calculated and matched, and the tracking status is updated.

Further, step S1 specifically includes the following steps:

Step S11: collect public face recognition data sets, and obtain pictures and names of relevant persons;

Step S12: Integrate pictures of common characters in multiple datasets by using a fusion strategy, use the pre-trained MTCNN model for face detection and face key point location, and apply similarity transformation for face alignment, and at the same time all the training set The images are subtracted from the mean of each channel on the training set, data preprocessing is done, and the adaptive aggregation network is trained.

Further, the adaptive aggregation network is formed by a deep feature extraction module and an adaptive feature aggregation module in series, which accepts one or more face images of the same person as input, and outputs the aggregated features, wherein the deep feature extraction The module adopts 34 layers of ResNet as the backbone network, and the adaptive feature aggregation module contains a feature aggregation layer; let B represent the number of input samples, {z _t } represents the output feature set of the deep feature extraction module, where t=1, 2, ...,B represents the input sample number, and the calculation method of the feature aggregation layer is:

a=∑ _t o _t z _t ;

In the formula, q represents the weight of each component of the feature vector z _t , which is a parameter that can be learned. By using the face recognition signal as the supervision signal, the back-propagation and gradient descent methods are used for learning, and v _t is the output of the sigmoid function, representing The score of each eigenvector z _t , ranging between 0 and 1, o _t is the output of L1 normalization, so that ∑ _t o _t =1, a is an eigenvector after B eigenvectors are aggregated.

Further, step S2 specifically includes the following steps:

j为第j个检测到人脸的编号，Jⁱ为第帧检测到的人脸数量，

其中

表示第i帧中第j个人脸的位置，x,y,w,h分别表示人脸区域的左上角坐标及其宽度和高度，

其中

表示第i帧中第 j个人脸的关键点，c₁,c₂,c₃,c₄,c₅分别表示人脸的左眼，右眼，鼻子，左嘴角，右嘴角的坐标；Step S21: Let i represent the number of the ith frame of the input video, initially i=1, use the pre-trained MTCNN model to simultaneously detect the position D ⁱ of all faces and the position C ⁱ of the corresponding facial key points, wherein

j is the number of the jth detected face, J ⁱ is the number of detected faces in the jth frame,

in

Represents the position of the jth face in the ith frame, x, y, w, h represent the upper left corner coordinates of the face area and its width and height, respectively,

in

Represents the key points of the jth face in the ith frame, c ₁ , c ₂ , c ₃ , c ₄ , and c ₅ respectively represent the coordinates of the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth;

为其分配一个唯一的身份ID_k,k＝1,2,...,Kⁱ，其中k表示第k个跟踪目标的编号，Kⁱ表示在第i帧时跟踪目标的人数，并初始化其对应的跟踪器T_k＝{ID_k,P_k,L_k,E_k,A_k}，其中ID_k表示第k 个跟踪目标的唯一身份标识，P_k表示分配给第k个目标的人脸位置坐标，L_k表示第 k个目标的面部关键点坐标，E_k表示第k个目标的人脸特征列表，A_k表示第k个目标的生命周期，初始化Kⁱ＝Jⁱ，

A_k＝1；Step S22 ^: For each face position D _ji and its facial key point coordinates

It is assigned a unique identity ID _k , k=1,2,...,K ⁱ , where k represents the number of the k-th tracking target, K ⁱ represents the number of people tracking the target at the i-th frame, and initializes its The corresponding tracker T _k ={ID _k ,P _k ,L _k ,E _k ,A _k }, where ID _k represents the unique identification of the k-th tracked target, and P _k represents the face assigned to the k-th target Position coordinates, L _k represents the facial key point coordinates of the k-th target, E _k represents the face feature list of the k-th target, A _k represents the life cycle of the k-th target, initialization K ⁱ =J ⁱ ,

A _k = 1;

Step S23: For the position P _k of each face in T _k , crop the image to obtain the corresponding face image, use the corresponding face key point position L _k , apply the similarity transformation to align the face, and obtain the aligned face image. face image;

Step S24: Input the aligned face image into the adaptive aggregation network to obtain the corresponding deep apparent feature of the face, and add it to the feature list E _k of T _k in the tracker.

Further, step S3 specifically includes the following steps:

Step S31: Represent the state of each tracked face target in the following form:

分别表示(u,v,s,r)在图像坐标空间中的速度；In the formula, m represents the state of the tracked face target, u and v represent the center coordinates of the tracked face area, s is the area of the face frame, r is the aspect ratio of the face frame,

respectively represent the speed of (u, v, s, r) in the image coordinate space;

的形式，其中

表示第i帧中第k个跟踪目标的人脸位置转化后的形式；Step S32: Convert the face position P _k =(x, y, w, h) in each tracker T _k into

in the form of

Represents the transformed form of the face position of the k-th tracking target in the i-th frame;

作为第i帧第k个跟踪目标的直接观测结果，其由人脸检测而来，采用基于线性匀速运动模型的卡尔曼滤波器对第k个跟踪目标在第i+1帧中的状态

进行预测；Step S33: put

As the direct observation result of the k-th tracking target in the i-th frame, it is obtained from face detection, and the Kalman filter based on the linear uniform motion model is used to analyze the state of the k-th tracking target in the i+1-th frame.

make predictions;

Step S34: in the i+1th frame, the MTCNN model is used to perform face detection and facial key point positioning again to obtain the position D ⁱ⁺¹ of the human face and the facial key point C ⁱ⁺¹ ;

基于其面部关键点

应用相似变换完成人脸对齐，并输入自适应聚合网络提取特征，得到特征集合Fⁱ⁺¹，其中Fⁱ⁺¹表示第i+1 帧中所有人脸的特征集合。Step S35: for each face position

based on its facial key points

Apply similarity transformation to complete face alignment, and input adaptive aggregation network to extract features to obtain feature set F ⁱ⁺¹ , where F ⁱ⁺¹ represents the feature set of all faces in the i+1th frame.

Further, step S4 specifically includes the following steps:

Step S41: For the tracker T _k of each face, input the set E _k of all the features in its historical motion trajectory into the adaptive aggregation network to obtain the aggregated feature f _k , where f _k represents the historical motion trajectory of the kth tracking target. An aggregated feature output after fusion of all feature vectors in ;

转化为

的形式；Step S42: Calculate the position state of the k-th target predicted by the Kalman filter in the i-th frame in the next frame

transform into

form;

和目标k聚合后的特征f_k，以及第i+1帧中的由人脸检测得到的人脸位置Dⁱ⁺¹及其特征集合Fⁱ⁺¹，计算如下关联矩阵：Step S43: Combine

The feature f _k aggregated with the target k, and the face position D ⁱ⁺¹ and its feature set F ⁱ⁺¹ obtained by face detection in the i+1th frame, calculate the following correlation matrix:

G=[g _jk ], j=1,2,...,J ⁱ⁺¹ ,k=1,2,...,K ⁱ ;

为第i+1帧中第j个人脸检测框与第i帧中由卡尔曼滤波器预测的第k个目标在第i+1帧中的位置状态

之间的重合程度，

为第i+1帧中第j个人脸特征

与第i帧中第k个目标聚合特征f_k之间的余弦相似度，λ为超参数，用于平衡两个度量的权重；In the formula, J ⁱ⁺¹ is the number of faces detected in the i+1th frame, K ⁱ is the number of tracked targets in the i-th frame,

is the position state of the jth face detection frame in the i+1th frame and the kth target predicted by the Kalman filter in the ith frame in the i+1th frame

the degree of overlap between

is the jth face feature in the i+1th frame

Cosine similarity with the k-th target aggregated feature f _k in the i-th frame, λ is a hyperparameter used to balance the weights of the two metrics;

关联到第k个跟踪目标；Step S44: Using the correlation matrix G as the cost matrix, the Hungarian algorithm is used to calculate the matching result, and the face detection frame in the i+1th frame is

Associated with the kth tracking target;

Step S45: Correspond the subscript in the matching result to the item in the association matrix G, and filter all items g _jk less than T _similarity , and delete it from the matching result, where T _similarity is a set hyperparameter, indicating that the matching is successful The minimum similarity threshold of ;

与第k个跟踪目标关联成功，则更新对应跟踪器T_k中的位置状态

人脸关键点位置

生命周期A_k＝A_k+1，以及将对应的人脸特征

添加到特征列表E_k，若检测框

关联失败，则创建新的跟踪器；Step S46: In the matching result, if the detection frame is

If the association with the k-th tracking target is successful, the position state in the corresponding tracker T _k is updated

Face key point location

Life cycle A _k =A _k +1, and the corresponding face features

add to the feature list E _k , if the detection frame

If the association fails, create a new tracker;

Step S47: For each tracker T _k , if its life cycle A _k >T _age , delete the tracker, where T _age is a set hyperparameter, indicating the longest time a tracking target can survive.

Compared with the prior art, the present invention has the following beneficial effects:

1. A multi-face tracking method based on deep apparent features and an adaptive aggregation network constructed by the present invention can effectively track the faces in the video, improve the accuracy of face tracking, and reduce target switching. number of times.

2. The present invention can track the human face in the video online while ensuring the tracking effect.

3. In the face tracking process, the predicted position of the face has a large uncertainty, and the face may undergo large posture changes and occlusions. The information between position and depth features improves the performance of face tracking.

4. In the face tracking process, it is difficult to effectively utilize all the features in the same target tracking trajectory and effectively compare multiple feature sets. It learns the importance of each feature in the feature set and fuses it effectively, which improves the effect of face tracking.

Description of drawings

FIG. 1 is a schematic flowchart of an embodiment of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments.

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

As shown in FIG. 1 , this embodiment provides a multi-face tracking method based on deep apparent features and an adaptive aggregation network, which specifically includes the following steps:

Step S1: using the face recognition data set to train the adaptive aggregation network;

Step S2: according to the initial input video frame, use the face detection method based on convolutional neural network to obtain the position of the face, initialize the face target to be tracked, extract the face features and save;

Step S3: use the Kalman filter to predict the position of each face target in the next frame, and use the face detection method again in the next frame to locate the position of the face, and extract features for the detected face;

Step S4: Use the adaptive aggregation network trained in step S1 to aggregate the face feature set in the tracking track of each tracked face target, and dynamically generate a face deep apparent feature fused with multi-frame information. The predicted position and the fused feature are compared with the detected face position and its features in the current frame, and the similarity is calculated and matched, and the tracking status is updated.

In this embodiment, step S1 specifically includes the following steps:

Step S11: collect public face recognition data sets, and obtain pictures and names of relevant persons;

Step S12: Integrate pictures of common characters in multiple datasets by using a fusion strategy, use the pre-trained MTCNN model for face detection and face key point location, and apply similarity transformation for face alignment, and at the same time all the training set The images are subtracted from the mean of each channel on the training set, data preprocessing is done, and the adaptive aggregation network is trained.

In this embodiment, the adaptive aggregation network is composed of a deep feature extraction module and an adaptive feature aggregation module in series, which accepts one or more face images of the same person as input, and outputs the aggregated features, wherein The deep feature extraction module uses 34 layers of ResNet as the backbone network, and the adaptive feature aggregation module contains a feature aggregation layer; let B represent the number of input samples, {z _t } represents the output feature set of the deep feature extraction module, where t=1 ,2,...,B represents the input sample number, and the calculation method of the feature aggregation layer is:

a=∑ _t o _t z _t ;

In the formula, q represents the weight of each component of the feature vector z _t , which is a parameter that can be learned. By using the face recognition signal as a supervision signal, the back-propagation and gradient descent methods are used for learning, and v _t is the output of the sigmoid function, representing The score of each eigenvector z _t , ranging between 0 and 1, o _t is the output of L1 normalization, so that ∑ _t o _t =1, a is an eigenvector after B eigenvectors are aggregated.

In this embodiment, step S2 specifically includes the following steps:

j为第j个检测到人脸的编号，Jⁱ为第帧检测到的人脸数量，

其中

表示第i帧中第j个人脸的位置，x,y,w,h分别表示人脸区域的左上角坐标及其宽度和高度，

其中

表示第i帧中第 j个人脸的关键点，c₁,c₂,c₃,c₄,c₅分别表示人脸的左眼，右眼，鼻子，左嘴角，右嘴角的坐标；Step S21: Let i represent the number of the ith frame of the input video, initially i=1, use the pre-trained MTCNN model to simultaneously detect the position D ⁱ of all faces and the position C ⁱ of the corresponding facial key points, wherein

j is the number of the jth detected face, J ⁱ is the number of detected faces in the jth frame,

in

Represents the position of the jth face in the ith frame, x, y, w, h represent the upper left corner coordinates of the face area and its width and height, respectively,

in

Represents the key points of the jth face in the ith frame, c ₁ , c ₂ , c ₃ , c ₄ , and c ₅ respectively represent the coordinates of the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth;

为其分配一个唯一的身份ID_k,k＝1,2,...,Kⁱ，其中k表示第k个跟踪目标的编号，Kⁱ表示在第i帧时跟踪目标的人数，并初始化其对应的跟踪器T_k＝{ID_k,P_k,L_k,E_k,A_k}，其中ID_k表示第k 个跟踪目标的唯一身份标识，P_k表示分配给第k个目标的人脸位置坐标，L_k表示第k个目标的面部关键点坐标，E_k表示第k个目标的人脸特征列表，A_k表示第k个目标的生命周期，初始化Kⁱ＝Jⁱ，

A_k＝1；Step S22 ^: For each face position D _ji and its facial key point coordinates

It is assigned a unique identity ID _k , k=1,2,...,K ⁱ , where k represents the number of the k-th tracking target, K ⁱ represents the number of people tracking the target at the i-th frame, and initializes its The corresponding tracker T _k ={ID _k ,P _k ,L _k ,E _k ,A _k }, where ID _k represents the unique identification of the k-th tracked target, and P _k represents the face assigned to the k-th target Position coordinates, L _k represents the facial key point coordinates of the k-th target, E _k represents the face feature list of the k-th target, A _k represents the life cycle of the k-th target, initialization K ⁱ =J ⁱ ,

A _k = 1;

Step S23: For the position P _k of each face in T _k , crop the image to obtain the corresponding face image, use the corresponding face key point position L _k , apply the similarity transformation to align the face, and obtain the aligned face image. face image;

Step S24: Input the aligned face image into the adaptive aggregation network to obtain the corresponding deep apparent feature of the face, and add it to the feature list E _k of T _k in the tracker.

In this embodiment, step S3 specifically includes the following steps:

Step S31: Represent the state of each tracked face target in the following form:

分别表示(u,v,s,r)在图像坐标空间中的速度；In the formula, m represents the state of the tracked face target, u and v represent the center coordinates of the tracked face area, s is the area of the face frame, r is the aspect ratio of the face frame,

respectively represent the speed of (u, v, s, r) in the image coordinate space;

的形式，其中

表示第i帧中第k个跟踪目标的人脸位置转化后的形式；Step S32: Convert the face position P _k =(x, y, w, h) in each tracker T _k into

in the form of

Represents the transformed form of the face position of the k-th tracking target in the i-th frame;

作为第i帧第k个跟踪目标的直接观测结果，其由人脸检测而来，采用基于线性匀速运动模型的卡尔曼滤波器对第k个跟踪目标在第i+1帧中的状态

进行预测；Step S33: put

As the direct observation result of the k-th tracking target in the i-th frame, it is obtained from face detection, and the Kalman filter based on the linear uniform motion model is used to analyze the state of the k-th tracking target in the i+1-th frame.

make predictions;

Step S34: in the i+1th frame, the MTCNN model is used to perform face detection and facial key point positioning again to obtain the position D ⁱ⁺¹ of the human face and the facial key point C ⁱ⁺¹ ;

基于其面部关键点

应用相似变换完成人脸对齐，并输入自适应聚合网络提取特征，得到特征集合Fⁱ⁺¹，其中Fⁱ⁺¹表示第i+1 帧中所有人脸的特征集合。Step S35: for each face position

based on its facial key points

Apply similarity transformation to complete face alignment, and input adaptive aggregation network to extract features to obtain feature set F ⁱ⁺¹ , where F ⁱ⁺¹ represents the feature set of all faces in the i+1th frame.

In this embodiment, step S4 specifically includes the following steps:

Step S41: For the tracker T _k of each face, input the set E _k of all the features in its historical motion trajectory into the adaptive aggregation network to obtain the aggregated feature f _k , where f _k represents the historical motion trajectory of the kth tracking target. An aggregated feature output after fusion of all feature vectors in ;

转化为

的形式；Step S42: Calculate the position state of the k-th target predicted by the Kalman filter in the i-th frame in the next frame

transform into

form;

和目标k聚合后的特征f_k，以及第i+1帧中的由人脸检测得到的人脸位置Dⁱ⁺¹及其特征集合Fⁱ⁺¹，计算如下关联矩阵：Step S43: Combine

The feature f _k aggregated with the target k, and the face position D ⁱ⁺¹ and its feature set F ⁱ⁺¹ obtained by face detection in the i+1th frame, calculate the following correlation matrix:

G=[g _jk ], j=1,2,...,J ⁱ⁺¹ ,k=1,2,...,K ⁱ ;

为第i+1帧中第j个人脸检测框与第i帧中由卡尔曼滤波器预测的第k个目标在第i+1帧中的位置状态

之间的重合程度，

为第i+1帧中第j个人脸特征

与第i帧中第k个目标聚合特征f_k之间的余弦相似度，λ为超参数，用于平衡两个度量的权重；In the formula, J ⁱ⁺¹ is the number of faces detected in the i+1th frame, K ⁱ is the number of tracked targets in the i-th frame,

is the position state of the jth face detection frame in the i+1th frame and the kth target predicted by the Kalman filter in the ith frame in the i+1th frame

the degree of overlap between

is the jth face feature in the i+1th frame

Cosine similarity with the k-th target aggregated feature f _k in the i-th frame, λ is a hyperparameter used to balance the weights of the two metrics;

关联到第k个跟踪目标；Step S44: Using the correlation matrix G as the cost matrix, the Hungarian algorithm is used to calculate the matching result, and the face detection frame in the i+1th frame is

Associated with the kth tracking target;

Step S45: Correspond the subscript in the matching result to the item in the association matrix G, and filter all items g _jk less than T _similarity , and delete it from the matching result, where T _similarity is a set hyperparameter, indicating that the matching is successful The minimum similarity threshold of ;

与第k个跟踪目标关联成功，则更新对应跟踪器T_k中的位置状态

人脸关键点位置

生命周期A_k＝A_k+1，以及将对应的人脸特征

添加到特征列表E_k，若检测框

关联失败，则创建新的跟踪器；Step S46: In the matching result, if the detection frame is

If the association with the k-th tracking target is successful, the position state in the corresponding tracker T _k is updated

Face key point location

Life cycle A _k =A _k +1, and the corresponding face features

add to the feature list E _k , if the detection frame

If the association fails, create a new tracker;

Step S47: For each tracker T _k , if its life cycle A _k >T _age , delete the tracker, where T _age is a set hyperparameter, indicating the longest time a tracking target can survive.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flows of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The above are only preferred embodiments of the present invention, and are not intended to limit the present invention in other forms. Any person skilled in the art may use the technical content disclosed above to make changes or modifications to equivalent changes. Example. However, any simple modifications, equivalent changes and modifications made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solutions of the present invention still belong to the protection scope of the technical solutions of the present invention.

Claims (2)

1. a multi-face tracking method based on deep apparent feature and self-adaptive aggregation network, is characterized in that: comprise the following steps: Step S1: using the face recognition data set to train the adaptive aggregation network; Step S2: According to the initial input video frame, use the convolutional neural network to obtain the position of the face, initialize the face target to be tracked, extract the face features and save; Step S3: use Kalman filter to predict the position of each face target in the next frame, and locate the position of the face again in the next frame, and extract features for the detected face; Step S4: Use the adaptive aggregation network trained in step S1 to aggregate the face feature set in the tracking track of each tracked face target, and dynamically generate a face deep apparent feature fused with multi-frame information. The predicted position and the fused features are calculated and matched with the face position and its features obtained through detection in the current frame, and the tracking status is updated; Step S1 specifically includes the following steps: Step S11: collect public face recognition data sets, and obtain pictures and names of relevant persons; Step S12: Integrate pictures of common characters in multiple datasets by using a fusion strategy, use the pre-trained MTCNN model for face detection and face key point location, and apply similarity transformation for face alignment, and at the same time all the training set The average value of each channel on the training set is subtracted from the images, data preprocessing is completed, and the adaptive aggregation network is trained; Step S2 specifically includes the following steps: Step S21: let i represent the number of the ith frame of the input video, initially i=1, use the pre-trained MTCNN model to simultaneously detect the position D ⁱ of all faces and the position C ⁱ of the corresponding facial key points, wherein

j is the number of the j-th detected face, J ⁱ is the number of detected faces in the i-th frame,

in

Represents the position of the jth face in the ith frame, x, y, w, h represent the upper left corner coordinates of the face area and its width and height, respectively,

in

Represents the key points of the jth face in the ith frame, c ₁ , c ₂ , c ₃ , c ₄ , and c ₅ respectively represent the coordinates of the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth; Step S22: For the position of each face

and its facial keypoint coordinates

It is assigned a unique identity ID _k , k=1,2,...,K ⁱ , where k represents the number of the k-th tracking target, K ⁱ represents the number of people tracking the target at the i-th frame, and initializes its The corresponding tracker T _k ={ID _k ,P _k ,L _k ,E _k ,A _k }, where ID _k represents the unique identification of the k-th tracking target, and P _k represents the face assigned to the k-th target Position coordinates, L _k represents the facial key point coordinates of the k-th target, E _k represents the face feature list of the k-th target, A _k represents the life cycle of the k-th target, initialization K ⁱ =J ⁱ ,

A _k = 1; Step S23: For the position P _k of each face in T _k , crop the image to obtain the corresponding face image, use the corresponding face key point position L _k , apply the similarity transformation to align the face, and obtain the aligned face image. face image; Step S24: input the aligned face image into the adaptive aggregation network, obtain the corresponding face depth apparent feature, and add it to the feature list E _k of T _k in the tracker; Step S3 specifically includes the following steps: Step S31: Represent the state of each tracked face target in the following form:

In the formula, m represents the state of the tracked face target, u and v represent the center coordinates of the tracked face area, s is the area of the face frame, r is the aspect ratio of the face frame,

respectively represent the speed of (u, v, s, r) in the image coordinate space; Step S32: Convert the face position P _k =(x, y, w, h) in each tracker T _k into

in the form of

Represents the transformed form of the face position of the k-th tracking target in the i-th frame; Step S33: put

As the direct observation result of the k-th tracking target in the i-th frame, it is obtained from face detection, and the Kalman filter based on the linear uniform motion model is used to analyze the state of the k-th tracking target in the i+1-th frame.

make predictions; Step S34: in the i+1th frame, the MTCNN model is used to perform face detection and facial key point positioning again to obtain the position D ⁱ⁺¹ of the human face and the facial key point C ⁱ⁺¹ ; Step S35: for each face position

based on its facial key points

Apply similarity transformation to complete face alignment, and input adaptive aggregation network to extract features to obtain feature set F ⁱ⁺¹ , where F ⁱ⁺¹ represents the feature set of all faces in the i+1th frame; Step S4 specifically includes the following steps: Step S41: For the tracker T _k of each face, input the set E _k of all the features in its historical motion trajectory into the adaptive aggregation network to obtain the aggregated feature f _k , where f _k represents the historical motion trajectory of the kth tracking target. An aggregated feature output after fusion of all feature vectors in ; Step S42: Calculate the position state of the k-th target predicted by the Kalman filter in the i-th frame in the next frame

transform into

form; Step S43: Combine

The feature f _k aggregated with the target k, and the face position D ⁱ⁺¹ and its feature set F ⁱ⁺¹ obtained by face detection in the i+1th frame, calculate the following correlation matrix: G=[g _jk ], j=1,2,...,J ⁱ⁺¹ ,k=1,2,...,K ⁱ ; In the formula, J ⁱ⁺¹ is the number of faces detected in the i+1th frame, K ⁱ is the number of tracked targets in the i-th frame,

is the position state of the jth face detection frame in the i+1th frame and the kth target predicted by the Kalman filter in the ith frame in the i+1th frame

the degree of overlap between

is the jth face feature in the i+1th frame

Cosine similarity with the k-th target aggregated feature f _k in the i-th frame, λ is a hyperparameter used to balance the weights of the two metrics; Step S44: Using the correlation matrix G as the cost matrix, the Hungarian algorithm is used to calculate the matching result, and the face detection frame in the i+1th frame is

Associated with the kth tracking target; Step S45: Correspond the subscript in the matching result to the item in the association matrix G, and filter all items g _jk less than T _similarity , and delete it from the matching result, where T _similarity is a set hyperparameter, indicating that the matching is successful The minimum similarity threshold of ; Step S46: In the matching result, if the detection frame is

If the association with the k-th tracking target is successful, the position state in the corresponding tracker T _k is updated

Face key point location

Life cycle A _k =A _k +1, and the corresponding face features

add to the feature list E _k , if the detection frame

If the association fails, create a new tracker; Step S47: For each tracker T _k , if its life cycle A _k >T _age , delete the tracker, where T _age is a set hyperparameter, indicating the longest time a tracking target can survive. 2. a kind of multi-face tracking method based on deep apparent feature and self-adaptive aggregation network according to claim 1, is characterized in that: described self-adaptive aggregation network is connected in series by deep feature extraction module and self-adaptive feature aggregation module It accepts one or more face images of the same person as input, and outputs aggregated features. The deep feature extraction module uses 34 layers of ResNet as the backbone network, and the adaptive feature aggregation module contains a feature aggregation layer; Let B represent the number of input samples, {z _t } represent the output feature set of the deep feature extraction module, where t=1,2,...,B represents the input sample number, and the calculation method of the feature aggregation layer is:

a=∑ _t o _t z _t ; In the formula, q represents the weight of each component of the feature vector z _t , which is a parameter that can be learned. By using the face recognition signal as the supervision signal, the back-propagation and gradient descent methods are used for learning, and v _t is the output of the sigmoid function, representing The score of each eigenvector z _t , ranging between 0 and 1, o _t is the output of L1 normalization, so that ∑ _t o _t =1, a is an eigenvector after B eigenvectors are aggregated.

Images