CN114758424B

CN114758424B - Intelligent payment equipment based on multiple verification mechanisms and payment method thereof

Info

Publication number: CN114758424B
Application number: CN202210663952.6A
Authority: CN
Inventors: 应旭斌
Original assignee: Zhonglvlian Zhejiang Ecological Technology Co ltd
Current assignee: Zhonglvlian Zhejiang Ecological Technology Co ltd
Priority date: 2022-06-14
Filing date: 2022-06-14
Publication date: 2022-09-02
Anticipated expiration: 2042-06-14
Also published as: CN114758424A

Abstract

The application relates to the field of intelligent payment, and particularly discloses an intelligent payment device based on a multiple verification mechanism and a payment method thereof, the intelligent payment equipment excavates deep local implicit associated characteristic information from a face image and an eyeball area image of an object to be verified through a deep neural network model so as to improve the safety and convenience of payment by combining face recognition and iris recognition, and in the process, in order to improve the fusion effect and improve the classification accuracy, the iris feature map is further weighted by using a data dense cluster mechanism based on self attention, therefore, the parameter adaptive variability of the fused feature map to the classification target function can be improved through the adaptive dependence of the data dense cluster in the iris feature map, so that the classification accuracy is improved, and the safety and the convenience of payment are improved.

Description

Intelligent payment equipment based on multiple verification mechanisms and payment method thereof

Technical Field

The present invention relates to the field of intelligent payment, and more particularly, to an intelligent payment device based on multiple verification mechanisms and a payment method thereof.

Background

Public transport, especially subway, as the important component in the transportation industry at present plays a key role in solving the problem of road congestion and improving traffic efficiency, but the form and payment speed of riding and paying become a problem of great concern. The speed and the convenience of payment not only influence the getting-on time but also determine the riding experience of passengers, and more determine the traveling efficiency of people, and also bring certain influence on the road traffic capacity.

At present, most of payment modes of taking a bus are cash, card swiping and code scanning payment, and some uncertain factors such as forgetting to take a bus card, defaulting of a mobile phone, shutdown of the mobile phone, recognition errors in code scanning and the like often appear in the taking process, which bring inconvenience to outgoing of people so that payment cannot be carried out, and lead to an embarrassed situation that a bus cannot be taken.

Therefore, an intelligent payment device for public transportation is expected to operate in a non-inductive payment manner to improve the payment efficiency of people traveling by bus and make the traveling more convenient.

Disclosure of Invention

The present application is proposed to solve the above-mentioned technical problems. The embodiment of the application provides intelligent payment equipment based on a multiple verification mechanism and a payment method thereof, deep local implicit associated feature information is mined from a face image and an eyeball area image of an object to be verified through a deep neural network model to improve the safety and accuracy of payment by combining face recognition and iris recognition, in addition, in the process, in order to improve the fusion effect and improve the classification accuracy, a data dense cluster mechanism based on self attention is further used for weighting an iris feature map, so that the parameter adaptive variability of the fused feature map on a classification target function can be improved through the adaptive dependence of the data dense cluster in the iris feature map to improve the classification accuracy, and the safety and the convenience of payment are improved.

According to an aspect of the present application, there is provided an intelligent payment device based on a multiple verification mechanism, comprising:

the face acquisition module is used for acquiring a face image of the object to be verified acquired by a camera of the intelligent payment equipment;

the face detection module is used for enabling the face image of the object to be verified to pass through a first convolution neural network model serving as a face area detection network so as to obtain a face feature map;

an eyeball region extraction module, which is used for extracting an eyeball region feature map corresponding to eyeballs from the face feature map based on the positions of the eyeballs in the face image;

the eyeball area pixel enhancement module is used for enabling the eyeball area characteristic graph to pass through a generator model serving as a pixel enhancer to obtain a generated eyeball area image;

the iris feature extraction module is used for enabling the generated eyeball area image to pass through a second convolution neural network model with a significance detection module so as to obtain an iris feature image;

a feature distribution correction module, configured to weight the iris feature map based on the face feature map by using a self-attention-based data dense cluster mechanism to obtain a weighted iris feature map, where the weighting of the iris feature map by using the self-attention-based data dense cluster mechanism is performed based on a spatial interaction feature map obtained by multiplying the face feature map and the iris feature map by a position point, and a distance between the face feature map and the iris feature map;

the characteristic distribution fusion module is used for fusing the weighted iris characteristic image and the face characteristic image to obtain a classification characteristic image;

the verification result generation module is used for enabling the classification characteristic graph to pass through a classifier to obtain a classification result, and the classification result is that an object to be detected corresponds to an object label in a database; and

and the payment module is used for paying the fee required by the vehicle taking based on the amount of money in the payment account associated with the object tag.

In the intelligent payment device based on the multiple verification mechanism, the first convolutional neural network model is Fast R-CNN, Fast R-CNN or RetinaNet.

In the above intelligent payment device based on multiple verification mechanisms, the iris feature extraction module is further configured to encode input data in the following manner by each layer of the second convolutional neural network: carrying out convolution coding on the input data by using first convolution cores of all layers of the second convolution neural network so as to obtain a convolution characteristic diagram; performing reel-to-reel encoding on the convolution feature map by using a second convolution unit of each layer of the second convolution neural network and a second convolution kernel to obtain a re-convolution feature map, wherein the first convolution unit and the second convolution unit form the significant feature detection module, and the size of the first convolution kernel is larger than that of the second convolution kernel; performing mean pooling based on a local feature matrix on the re-convolution feature map by using pooling units of each layer of the second convolution neural network to obtain a pooled feature map; and carrying out nonlinear activation on the feature values of all positions in the pooled feature map by using the activation units of all layers of the second convolutional neural network to obtain an activated feature map; wherein the output of the last layer of the second convolutional neural network is the iris feature map.

In the above intelligent payment device based on multiple verification mechanisms, the feature distribution correction module includes: the space interaction feature map generation unit is used for calculating the point-by-point multiplication between the face feature map and the iris feature map to obtain the space interaction feature map; the data difference measurement unit is used for calculating the square root of the Euclidean distance between the face feature map and the iris feature map; the attention unit is used for dividing the characteristic value of each position in the space interaction characteristic map by the square root of the Euclidean distance between the face characteristic map and the iris characteristic map to obtain an attention characteristic map; an exponential operation unit, which is used for calculating a natural exponential function value taking the characteristic value of each position in the attention characteristic map as power to obtain an exponential attention characteristic map; a first class probability unit, configured to pass the exponential attention feature map through the classifier to obtain a first class probability index; the second-class probability unit is used for enabling the iris feature map to pass through the classifier to obtain a second-class probability index; the action unit is used for calculating the product of the first class probability index and the second class probability index as the weighting coefficient of the iris feature map; and the correction unit is used for weighting the iris characteristic diagram by the weighting coefficient to obtain the weighted iris characteristic diagram.

In the above intelligent payment device based on multiple verification mechanisms, the feature distribution correction module is further configured to weight the iris feature map by using a self-attention-based data dense cluster mechanism according to the following formula to obtain the weighted iris feature map;

wherein the formula is:

whereinF _face A graph representing the facial features of the human face,F _iris the iris feature map is represented by a graph of the iris,

the expression point multiplication, softmax (·) represents probability values obtained by a characteristic diagram through a classifier, d (·, ·) represents distances between characteristic diagrams, exp (·) represents exponential operation of the characteristic diagram, the exponential operation of the characteristic diagram represents calculation of natural exponential function values raised by characteristic values of positions in the characteristic diagram, and division of the characteristic diagram by parameters represents division of the characteristic values of the positions in the characteristic diagram by the parameters respectively.

In the above intelligent payment device based on multiple verification mechanisms, the feature distribution fusion module is further configured to: fusing the weighted iris feature map and the face feature map according to the following formula to obtain the classification feature map;

wherein the formula is:

F=αwF _iris +βF _face

wherein,Fin order to be able to classify the feature map,F _iris for the characteristic map of the iris in question,F _face in order to obtain the face feature map,wF _iris representing said weighted iris feature map "+"represents the addition of elements at the corresponding positions of the weighted iris feature map and the face feature map,αandβis a weighting parameter for controlling the balance between the weighted iris feature map and the face feature map.

In the above intelligent payment device based on multiple verification mechanisms, the verification result generation module is further configured to: the classifier processes the classification feature map to generate a classification result according to the following formula:softmax{(W _n , B _n ):...:(W ₁ , B ₁ )|Project(F) Therein ofProject(F) Representing the projection of the classification feature map as a vector,W ₁ toW _n Is a weight matrix of the fully connected layers of each layer,B ₁ toB _n A bias matrix representing the layers of the fully connected layer.

In the above intelligent payment device based on multiple verification mechanisms, the payment module is further configured to generate an insufficient-charge prompt in response to that the amount of money in the payment account associated with the object tag is smaller than the charge required for the ride.

According to another aspect of the present application, a payment method of an intelligent payment device based on a multiple verification mechanism includes:

acquiring a face image of an object to be verified, which is acquired by a camera of intelligent payment equipment;

the face image of the object to be checked is processed by a first convolution neural network model serving as a face area detection network to obtain a face characteristic image;

extracting an eyeball area feature map corresponding to eyeballs from the face feature map based on the positions of the eyeballs in the face image;

enabling the eyeball area characteristic graph to pass through a generator model serving as a pixel enhancer to obtain a generated eyeball area image;

passing the generated eyeball area image through a second convolutional neural network model with a significance detection module to obtain an iris characteristic diagram;

weighting the iris feature map by using a self-attention-based data dense cluster mechanism to obtain a weighted iris feature map based on the face feature map, wherein the weighting of the iris feature map by using the self-attention-based data dense cluster mechanism is performed based on a spatial interaction feature map obtained by multiplying the face feature map and the iris feature map by position points and the distance between the face feature map and the iris feature map;

fusing the weighted iris feature map and the face feature map to obtain a classification feature map;

the classification characteristic graph is processed by a classifier to obtain a classification result, and the classification result is that an object to be detected corresponds to an object label in a database; and

and paying the fee required by the bus based on the amount of money in the payment account associated with the object tag.

In the payment method of the intelligent payment equipment based on the multiple verification mechanisms, the first convolutional neural network model is Fast R-CNN, Fast R-CNN or RetinaNet.

In the above payment method of the intelligent payment device based on multiple verification mechanisms, passing the generated eyeball area image through a second convolutional neural network model with a significance detection module to obtain an iris feature map, including: encoding input data in layers of the second convolutional neural network as follows: performing convolution coding on the input data by using first convolution units of each layer of the second convolution neural network and first convolution cores to obtain a convolution characteristic diagram; performing reel-to-reel encoding on the convolution feature map by using a second convolution unit of each layer of the second convolution neural network and a second convolution kernel to obtain a re-convolution feature map, wherein the first convolution unit and the second convolution unit form the significant feature detection module, and the size of the first convolution kernel is larger than that of the second convolution kernel; performing local feature matrix-based mean pooling on the re-convolution feature map by pooling units of each layer of the second convolution neural network to obtain a pooled feature map; and carrying out nonlinear activation on the feature values of all positions in the pooled feature map by using the activation units of all layers of the second convolutional neural network to obtain an activated feature map; wherein the output of the last layer of the second convolutional neural network is the iris feature map.

In the above payment method of an intelligent payment device based on a multiple verification mechanism, based on the face feature map, weighting the iris feature map by using a data dense cluster mechanism based on self-attention to obtain a weighted iris feature map, the method includes: calculating the position-based multiplication between the face feature map and the iris feature map to obtain the space interaction feature map; calculating the square root of the Euclidean distance between the face feature image and the iris feature image; dividing feature values of all positions in the space interaction feature map by the square root of the Euclidean distance between the face feature map and the iris feature map to obtain an attention feature map; calculating natural exponential function values taking the feature values of all positions in the attention feature map as powers to obtain an exponential attention feature map; passing the exponential attention feature map through the classifier to obtain a first class probability index; passing the iris feature map through the classifier to obtain a second class probability index; calculating the product of the first class probability index and the second class probability index as the weighting coefficient of the iris feature map; and weighting the iris characteristic diagram by the weighting coefficient to obtain the weighted iris characteristic diagram.

In the above payment method of an intelligent payment device based on multiple verification mechanisms, based on the face feature map, the iris feature map is weighted by using a data dense cluster mechanism based on self-attention to obtain a weighted iris feature map, which includes: weighting the iris feature map by using a self-attention-based data dense cluster mechanism according to the following formula to obtain the weighted iris feature map;

wherein the formula is:

whereinF _face The face feature map is represented and the face feature map is displayed,F _iris the iris feature map is represented by a graph of the iris,

the method comprises the steps of representing point multiplication, softmax (DEG) represents probability values obtained by a classifier of a feature map, D (DEG) represents distances between the feature maps, exp (DEG) represents exponential operation of the feature map, the exponential operation of the feature map represents calculation of a natural exponent function value taking feature values of all positions in the feature map as power, and the division of the feature map by parameters represents division of the feature values of all the positions in the feature map by the parameters.

In the above payment method of an intelligent payment device based on a multiple verification mechanism, fusing the weighted iris feature map and the face feature map to obtain a classification feature map, including: fusing the weighted iris feature map and the face feature map to obtain the classification feature map according to the following formula;

wherein the formula is:

F=αwF _iris +βF _face

In the above payment method of an intelligent payment device based on a multiple verification mechanism, passing the classification feature map through a classifier to obtain a classification result, the method includes: the classifier processes the classification feature map to generate a classification result according to the following formula;

wherein the formula is:softmax{(W _n , B _n ):...:(W ₁ , B ₁ )|Project(F) Therein ofProject(F) Representing the projection of the classification feature map as a vector,W ₁ toW _n Is a weight matrix of the fully connected layers of each layer,B ₁ to is thatB _n A bias matrix representing the layers of the fully connected layer.

In the above payment method for an intelligent payment device based on a multiple verification mechanism, the payment of the vehicle taking charge based on the amount of money in the payment account associated with the object tag includes: and generating an under-fare prompt in response to the amount of money in the payment account associated with the object tag being less than the fare required for the ride.

Compared with the prior art, according to the intelligent payment equipment based on the multiple verification mechanisms and the payment method thereof, deep local implicit associated feature information is mined from the face image and the eyeball area image of the object to be verified through the deep neural network model to improve the safety and accuracy of payment by combining face recognition and iris recognition, and in the process, in order to improve the fusion effect and improve the accuracy of classification, the iris feature map is further weighted by using a data dense cluster mechanism based on self-attention, so that the parameter adaptive variability of the fused feature map on a classification target function is improved through the adaptive dependence of the data dense cluster in the iris feature map to improve the classification accuracy, and the safety and convenience of payment are improved.

Drawings

The above and other objects, features and advantages of the present application will become more apparent by describing in more detail embodiments of the present application with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings, like reference numbers generally indicate like parts or steps.

Fig. 1 is an application scenario diagram of an intelligent payment device based on a multiple verification mechanism according to an embodiment of the present application.

Fig. 2 is a block diagram of an intelligent payment device based on a multiple verification mechanism according to an embodiment of the present application.

Fig. 3 is a block diagram of a feature distribution correction module in an intelligent payment device based on a multiple verification mechanism according to an embodiment of the present application.

Fig. 4 is a flowchart of a payment method of an intelligent payment device based on a multiple verification mechanism according to an embodiment of the present application.

Fig. 5 is a schematic structural diagram of a payment method of an intelligent payment device based on a multiple verification mechanism according to an embodiment of the present application.

Detailed Description

Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only a few embodiments of the present application, and not all embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein.

Overview of a scene

As mentioned above, public transportation, especially subway as an important component in the transportation industry, plays a key role in solving the problem of road congestion and improving traffic efficiency, but the form of bus payment and payment speed become a great concern. The speed and convenience of payment not only influence the boarding time, but also determine the riding experience of passengers, and further determine the traveling efficiency of people, and certain influence can be brought to the road traffic capacity.

At present, most of payment modes of taking a bus are cash, card swiping and code scanning payment, and some uncertain factors such as forgetting to take a bus card, defaulting of a mobile phone, shutdown of the mobile phone, recognition errors in code scanning and the like often appear in the taking process, which bring inconvenience to outgoing of people so that payment cannot be carried out, and lead to an embarrassed situation that a bus cannot be taken. Therefore, an intelligent payment device for public transportation is expected to be operated in a non-inductive payment manner to improve the payment efficiency of people traveling in a bus and make the traveling more convenient.

Accordingly, it is considered that face-brushing payment is a common non-inductive payment means, but in the field of public transportation, face-brushing payment cannot well meet the requirements of application scenarios. The inventor finds that people can do various decorations when going out, for example, women can dress up and wear sun shading equipment, so that people cannot accurately recognize faces when brushing faces and paying, and cannot pay or pay wrongly. Therefore, in the present application, the present invention thermally attempts to improve security and convenience of payment by combining face recognition and iris recognition.

Specifically, in the technical scheme of the application, firstly, a camera of the intelligent payment device is used for collecting a face image of an object to be verified. Then, a convolution neural network with excellent performance in the aspect of local implicit feature extraction of the image is used for deep mining of implicit features of the face image of the object to be verified. It should be understood that the target detection method based on deep learning divides the network into two categories, Anchor-based (Anchor-based) and Anchor-free (Anchor-free) according to whether an Anchor window is used in the network. Anchor window based methods such as Fast R-CNN, RetinaNet, etc., anchor window based methods such as CenterNet, ExtremeNet, RePoints, etc. The method based on the anchor window can enable regression of target classification and boundary box coordinates among networks, the prior is added to enable training to be stable, the recall capability of the network target can be effectively improved, and the method is obviously improved for small target detection. Therefore, in the technical scheme of the application, the face image of the object to be verified is used as a first convolution neural network model of the face area detection network to obtain the face feature map. In particular, here, the first convolutional neural network model is Fast R-CNN, Fast R-CNN or RetinaNet.

Then, an eyeball region feature map corresponding to the eyeballs is extracted from the face feature map further based on the positions of the eyeballs in the face image. Here, the convolutional neural network is used to not deform at the position of the extracted feature, so that the eyeball area feature map corresponding to the eyeball in the face feature map can be determined according to the position of the eyeball in the face image in the original image. It should be understood that, because the resolution of the eyeball image in the face image of the object to be verified, which is usually acquired, is not high, and the iris cannot be clearly recognized, in the technical solution of the present application, the eyeball region feature map is further passed through a generator model as a pixel enhancer to obtain a generated eyeball region image.

And then, the generated eyeball area image is subjected to local high-dimensional implicit feature extraction in a second convolutional neural network model with a significance detection module to obtain an iris feature map. Specifically, in the embodiment of the present application, the input data is convolution-encoded by a first convolution core with a first convolution unit of each layer of the second convolutional neural network to obtain a convolution feature map; performing coil-winding encoding on the convolution feature map by using a second convolution unit of each layer of the second convolution neural network and a second convolution kernel to obtain a re-convolution feature map, wherein the first convolution unit and the second convolution unit form the significant feature detection module, and the size of the first convolution kernel is larger than that of the second convolution kernel; then, performing mean pooling based on a local feature matrix on the re-convolution feature map by using pooling units of each layer of the second convolution neural network to obtain a pooled feature map; secondly, carrying out nonlinear activation on the feature values of all positions in the pooled feature map by using the activation units of all layers of the second convolutional neural network to obtain an activated feature map; wherein the output of the last layer of the second convolutional neural network is the iris feature map.

It should be understood that when obtaining the iris feature map, extracting the local region in the image by using the candidate frame as the reference window, and obtaining the iris feature map by resolution enhancement and saliency detection, the iris feature map is in a data-intensive processHigher than the face feature image in degree, if the face feature image is directly fusedF _face And iris feature mapF _iris It may cause a classification bias of the fused feature map with respect to the face feature map and the iris feature map.

Therefore, the iris feature map is weighted by using a self-attention-based data dense cluster mechanism, specifically:

Here, the self-attention-based data-dense clustering mechanism enables spatial interaction of local features and global features based on a reference window and expresses similarity between data-dense object instances by a measure of data dissimilarity represented by a feature map distance. By weighting the iris feature map and fusing the weighted iris feature map with the face feature map, the adaptive dependence of the dense data clusters in the iris feature map can be used for improving the parameter adaptive variability of the fused feature map on a classification target function, so that the classification accuracy is improved.

And then fusing the weighted iris feature map and the face feature map to obtain a classification feature map, and then enabling the classification feature map to pass through a classifier to obtain a classification result for representing the object to be detected to correspond to the object label in the database. Further, the fee required by the vehicle is paid based on the amount of money in the payment account associated with the object tag. In an embodiment of the application, in response to the amount of money in the payment account associated with the object tag being less than the fee required for the ride, a fee insufficiency prompt is generated.

Based on this, this application has proposed a intelligent payment equipment based on multiple check-up mechanism, it includes: the face acquisition module is used for acquiring a face image of an object to be verified, which is acquired by a camera of the intelligent payment equipment; the face detection module is used for enabling the face image of the object to be verified to pass through a first convolution neural network model serving as a face area detection network so as to obtain a face feature map; an eyeball region extraction module, which is used for extracting an eyeball region feature map corresponding to eyeballs from the face feature map based on the positions of the eyeballs in the face image; the eyeball area pixel enhancement module is used for enabling the eyeball area characteristic graph to pass through a generator model serving as a pixel enhancer to obtain a generated eyeball area image; the iris feature extraction module is used for enabling the generated eyeball area image to pass through a second convolution neural network model with a significance detection module so as to obtain an iris feature image; a feature distribution correction module, configured to weight the iris feature map based on the face feature map by using a self-attention-based data dense cluster mechanism to obtain a weighted iris feature map, where the weighting of the iris feature map by using the self-attention-based data dense cluster mechanism is performed based on a spatial interaction feature map obtained by multiplying the face feature map and the iris feature map by a position point, and a distance between the face feature map and the iris feature map; the characteristic distribution fusion module is used for fusing the weighted iris characteristic diagram and the face characteristic diagram to obtain a classification characteristic diagram; the verification result generation module is used for enabling the classification characteristic graph to pass through a classifier to obtain a classification result, and the classification result is that an object to be detected corresponds to an object label in a database; and the payment module is used for paying the fee required by the vehicle taking based on the amount of money in the payment account associated with the object tag.

Fig. 1 illustrates an application scenario diagram of an intelligent payment device based on a multiple verification mechanism according to an embodiment of the present application. As shown in fig. 1, in this application scenario, first, a face image of an object to be verified is acquired by a camera (e.g., C as illustrated in fig. 1) in an intelligent payment device (e.g., T as illustrated in fig. 1). Then, the acquired face image of the object to be verified is input into a server (for example, a cloud server S as illustrated in fig. 1) deployed with an intelligent payment algorithm based on a multiple verification mechanism, wherein the server can process the face image of the object to be verified by the intelligent payment algorithm based on the multiple verification mechanism to generate a classification result indicating that the object to be verified corresponds to the object tag in the database, and further, the passenger-required fee is paid based on the amount of money in the payment account associated with the object tag.

Having described the basic principles of the present application, various non-limiting embodiments of the present application will now be described with reference to the accompanying drawings.

Exemplary System

Fig. 2 illustrates a block diagram of an intelligent payment device based on a multiple verification mechanism according to an embodiment of the present application. As shown in fig. 2, the intelligent payment device 200 based on multiple verification mechanisms according to an embodiment of the present application includes: the face acquisition module 210 is configured to acquire a face image of an object to be verified acquired by a camera of the intelligent payment device; a face detection module 220, configured to pass the face image of the object to be verified through a first convolutional neural network model serving as a face area detection network to obtain a face feature map; an eyeball region extraction module 230, configured to extract an eyeball region feature map corresponding to an eyeball from the face feature map based on the position of the eyeball in the face image; an eyeball region pixel enhancement module 240, configured to pass the eyeball region feature map through a generator model as a pixel enhancer to obtain a generated eyeball region image; an iris feature extraction module 250, configured to pass the generated eyeball region image through a second convolutional neural network model with a saliency detection module to obtain an iris feature map; a feature distribution correction module 260, configured to weight the iris feature map based on the face feature map by using a self-attention-based data dense cluster mechanism to obtain a weighted iris feature map, where the weighting of the iris feature map by using the self-attention-based data dense cluster mechanism is performed based on a spatial interaction feature map obtained by multiplying the face feature map and the iris feature map by a position point, and a distance between the face feature map and the iris feature map; a feature distribution fusion module 270, configured to fuse the weighted iris feature map and the face feature map to obtain a classification feature map; a verification result generation module 280, configured to pass the classification feature map through a classifier to obtain a classification result, where the classification result is that an object to be detected corresponds to an object tag in a database; and a payment module 290, configured to pay the fee required for the ride based on the amount in the payment account associated with the object tag.

Specifically, in this embodiment of the application, the face acquisition module 210 and the face detection module 220 are configured to acquire a face image of an object to be verified acquired by a camera of an intelligent payment device, and obtain a face feature map by using the face image of the object to be verified as a first convolutional neural network model of a face area detection network. As described above, it is considered that face-brushing payment is a common non-inductive payment means, but in the field of public transportation, face-brushing payment cannot well meet the application scenario requirements. The reason for this is that people can do various decorations when going out, for example, women can dress up and wear sun shading equipment, which makes people unable to accurately recognize faces when brushing faces and paying, resulting in unable payment or wrong payment. Therefore, in the technical scheme of the application, the safety and the accuracy of payment are expected to be improved by combining face recognition and iris recognition.

Specifically, in the technical scheme of the application, firstly, a camera of the intelligent payment device is used for collecting a face image of an object to be verified. Then, a convolution neural network with excellent performance in the aspect of local implicit feature extraction of the image is used for deep mining of implicit features of the face image of the object to be verified. It should be understood that the target detection method based on deep learning divides the network into two categories, Anchor-based (Anchor-based) and Anchor-free (Anchor-free) according to whether an Anchor window is used in the network. Such as Fast R-CNN, RetinaNet, etc., and anchorless window-based methods such as CenterNet, ExtrmeNet, RePoints, etc. The method based on the anchor window can enable regression of target classification and boundary box coordinates among networks, training is stable by adding prior, the recall capability of network targets can be effectively improved, and the method is very obvious for small target detection. Therefore, in the technical scheme of the application, the face image of the object to be verified is processed in the first convolution neural network model serving as the face area detection network to extract the local high-dimensional implicit feature distribution in the face image so as to obtain the face feature map. In particular, here, the first convolutional neural network model is Fast R-CNN, Fast R-CNN or RetinaNet.

Specifically, in the embodiment of the present application, the eyeball region extraction module 230 and the eyeball region pixel enhancement module 240 are configured to extract an eyeball region feature map corresponding to an eyeball from the face feature map based on the position of the eyeball in the face image, and pass the eyeball region feature map through a generator model as a pixel enhancer to obtain a generated eyeball region image. That is, in order to improve the security and accuracy of payment by combining face recognition and iris recognition, an eyeball region feature map corresponding to eyeballs is extracted from the face feature map further based on the positions of the eyeballs in the face image. It should be understood that, here, the convolutional neural network is utilized to not deform at the position of extracting the feature, and therefore, in the technical solution of the present application, the eyeball area feature map corresponding to the eyeball in the face feature map may be determined according to the position of the eyeball in the original image in the face image.

Then, it should be understood that, since the resolution of the eyeball image in the face image of the object to be verified, which is usually acquired, is not high, and the iris cannot be clearly recognized, in the technical solution of the present application, the eyeball region feature map is further passed through a generator model as a pixel enhancer to obtain an eyeball region image with enhanced generated features.

Specifically, in this embodiment of the present application, the iris feature extraction module 250 is configured to pass the generated eyeball region image through a second convolutional neural network model having a saliency detection module to obtain an iris feature map. That is, in the technical solution of the present application, in this way, the generated eyeball region image is subjected to local high-dimensional implicit feature extraction in a second convolutional neural network model with a saliency detection module, so as to obtain an iris feature map corresponding to the eyeball region.

More specifically, in this embodiment of the present application, the iris feature extraction module is further configured to encode the input data in the following manner in each layer of the second convolutional neural network: performing convolution coding on the input data by using first convolution units of each layer of the second convolution neural network and first convolution cores to obtain a convolution characteristic diagram; performing reel-to-reel encoding on the convolution feature map by using a second convolution unit of each layer of the second convolution neural network and a second convolution kernel to obtain a re-convolution feature map, wherein the first convolution unit and the second convolution unit form the significant feature detection module, and the size of the first convolution kernel is larger than that of the second convolution kernel; performing local feature matrix-based mean pooling on the re-convolution feature map by pooling units of each layer of the second convolution neural network to obtain a pooled feature map; and carrying out nonlinear activation on the feature values of all positions in the pooled feature map by using the activation units of all layers of the second convolutional neural network to obtain an activated feature map; wherein the output of the last layer of the second convolutional neural network is the iris feature map.

Specifically, in this embodiment, the feature distribution correction module 260 is configured to weight the iris feature map based on the face feature map by using a self-attention-based data dense clustering mechanism to obtain a weighted iris feature map, where the iris feature map is obtained by weighting the iris feature map, and the feature distribution is obtained by using a self-attention-based data dense clustering mechanismWeighting the iris feature map by using a self-attention-based data dense cluster mechanism is performed based on a spatial interaction feature map obtained by multiplying the face feature map and the iris feature map by position points and the distance between the face feature map and the iris feature map. It should be understood that when the iris feature map is obtained, the local region in the image is extracted by using the candidate frame as the reference window, and then the iris feature map is obtained by resolution enhancement and saliency detection, which makes the iris feature map higher than the face feature map in data density, if the face feature map is directly fused, the iris feature map is higher than the face feature mapF _face And the iris feature mapF _iris A classification bias of the fused feature map with respect to the face feature map and the iris feature map may result. Therefore, in the technical solution of the present application, the iris feature map is further weighted by using a self-attention-based data dense cluster mechanism.

Accordingly, in a specific example, first, the position-based point multiplication between the face feature map and the iris feature map is calculated to obtain the spatial interaction feature map. Then, the square root of the Euclidean distance between the face feature image and the iris feature image is calculated. Then, the feature value of each position in the space interaction feature map is divided by the square root of the Euclidean distance between the face feature map and the iris feature map to obtain an attention feature map. Then, a natural exponent function value raised by a feature value of each position in the attention feature map is calculated to obtain an exponential attention feature map. The exponential attention feature map is then passed through the classifier to obtain a first class probability index. Then, the iris feature map is passed through the classifier to obtain a second class probability index. Then, calculating the product between the first class probability index and the second class probability index as the weighting coefficient of the iris feature map. And finally, weighting the iris characteristic diagram by the weighting coefficient to obtain the weighted iris characteristic diagram. It should be appreciated that the self-attention-based data-dense clustering mechanism enables spatial interaction of local features and global features based on a reference window and expresses similarity between data-dense object instances by a measure of dissimilarity of data represented by a feature map distance. Therefore, the iris feature map is weighted and then fused with the face feature map, and the parameter adaptive variability of the fused feature map to a classification target function can be improved through the adaptive dependence of the data dense cluster in the iris feature map, so that the classification accuracy is improved.

More specifically, in an embodiment of the present application, the feature distribution correction module is further configured to: weighting the iris feature map by using a self-attention-based data dense cluster mechanism according to the following formula to obtain the weighted iris feature map;

wherein the formula is:

Fig. 3 illustrates a block diagram of a feature distribution correction module in an intelligent payment device based on a multiple verification mechanism according to an embodiment of the present application. As shown in fig. 3, the feature distribution correction module 260 includes: a spatial interaction feature map generation unit 261, configured to calculate a point-by-point multiplication between the face feature map and the iris feature map to obtain the spatial interaction feature map; a data dissimilarity measurement unit 262, configured to calculate a square root of a euclidean distance between the face feature map and the iris feature map; an attention unit 263, configured to divide feature values at various positions in the spatial interaction feature map by a square root of a euclidean distance between the face feature map and the iris feature map to obtain an attention feature map; an exponential operation unit 264 for calculating a natural exponential function value raised by a feature value of each position in the attention feature map to obtain an exponential attention feature map; a first class probability unit 265, configured to pass the exponential attention feature map through the classifier to obtain a first class probability index; a second-class probability unit 266, configured to pass the iris feature map through the classifier to obtain a second-class probability index; an action unit 267, configured to calculate a product between the first class probability index and the second class probability index as a weighting coefficient of the iris feature map; and a correcting unit 268, configured to weight the iris feature map by using the weighting coefficient to obtain the weighted iris feature map.

Specifically, in this embodiment of the present application, the feature distribution fusion module 270 is configured to fuse the weighted iris feature map and the face feature map to obtain a classification feature map. It should be understood that, in this way, by using the self-attention-based data dense cluster mechanism to weight the iris feature map and then fuse the iris feature map with the face feature map, the parameter adaptive variability of the fused classification feature map on the classification target function can be improved through the adaptive dependence of the data dense cluster in the iris feature map, thereby improving the classification accuracy.

More specifically, in an embodiment of the present application, the feature distribution fusion module is further configured to: fusing the weighted iris feature map and the face feature map according to the following formula to obtain the classification feature map;

wherein the formula is:

F=αwF _iris +βF _face

Specifically, in this embodiment of the application, the verification result generating module 280 and the payment module 290 are configured to pass the classification feature map through a classifier to obtain a classification result, where the classification result is that the object to be verified corresponds to an object tag in a database, and pay the fee required for the vehicle taking based on the amount of money in the payment account associated with the object tag. That is, in the technical solution of the present application, after the classification feature map is obtained, the classification feature map is further passed through a classifier to obtain a classification result indicating that the object to be checked corresponds to the object tag in the database, and further, the fee required for the vehicle taking is paid based on the amount of money in the payment account associated with the object tag. Accordingly, in one particular example, the under-fare prompt is generated in response to the amount of money in the payment account associated with the object tag being less than the fare required for the ride.

More specifically, in the embodiment of the present application, the classifier processes the classification feature map to generate a classification result according to the following formula;

the formula is:softmax{(W _n , B _n ):...:(W ₁ , B ₁ )|Project(F) Therein ofProject(F) Representing the projection of the classification feature map as a vector,W ₁ toW _n Is a weight matrix of the fully connected layers of each layer,B ₁ toB _n A bias matrix representing the layers of the fully connected layer.

In summary, the multiple verification mechanism-based intelligent payment apparatus 200 according to the embodiment of the present application is illustrated, which uses a deep neural network model to mine deep local implicit associated feature information from a face image and an eyeball area image of an object to be verified, so as to improve the security and accuracy of payment by combining face recognition and iris recognition, and in this process, in order to improve the fusion effect and improve the classification accuracy, a self-attention-based data dense cluster mechanism is further used to weight an iris feature map, so that the parameter adaptive variability of the fused feature map on a classification target function can be improved by the adaptive dependence of the data dense cluster in the iris feature map, thereby improving the classification accuracy.

As described above, the intelligent payment device 200 based on multiple verification mechanisms according to the embodiment of the present application may be implemented in various terminal devices, such as a server of an intelligent payment algorithm based on multiple verification mechanisms. In one example, the intelligent payment device 200 based on the multiple verification mechanism according to the embodiment of the present application may be integrated into the terminal device as one software module and/or hardware module. For example, the multi-verification mechanism based intelligent payment device 200 may be a software module in the operating system of the terminal device, or may be an application developed for the terminal device; of course, the intelligent payment device 200 based on the multiple verification mechanism can also be one of many hardware modules of the terminal device.

Alternatively, in another example, the multi-verification mechanism based intelligent payment device 200 and the terminal device may be separate devices, and the multi-verification mechanism based intelligent payment device 200 may be connected to the terminal device through a wired and/or wireless network and transmit the interaction information according to the agreed data format.

Exemplary method

Fig. 4 illustrates a flow chart of a payment method of an intelligent payment device based on a multiple verification mechanism. As shown in fig. 4, a payment method of an intelligent payment device based on multiple verification mechanisms according to an embodiment of the present application includes the steps of: s110, acquiring a face image of an object to be verified, which is acquired by a camera of the intelligent payment equipment; s120, the face image of the object to be checked is processed by a first convolution neural network model serving as a face area detection network to obtain a face feature map; s130, extracting an eyeball area feature map corresponding to eyeballs from the face feature map based on the positions of the eyeballs in the face image; s140, enabling the eyeball area characteristic graph to pass through a generator model serving as a pixel enhancer to obtain a generated eyeball area image; s150, enabling the generated eyeball area image to pass through a second convolutional neural network model with a significance detection module to obtain an iris characteristic diagram; s160, based on the face feature map, weighting the iris feature map by using a self-attention-based data dense cluster mechanism to obtain a weighted iris feature map, wherein the weighting of the iris feature map by using the self-attention-based data dense cluster mechanism is performed based on a spatial interaction feature map obtained by multiplying the face feature map and the iris feature map by position points and the distance between the face feature map and the iris feature map; s170, fusing the weighted iris feature map and the face feature map to obtain a classification feature map; s180, passing the classification characteristic graph through a classifier to obtain a classification result, wherein the classification result is an object label of an object to be detected corresponding to a database; and S190, paying the fee required by the vehicle taking based on the amount of money in the payment account associated with the object tag.

Fig. 5 illustrates an architecture diagram of a payment method of an intelligent payment device based on a multiple verification mechanism according to an embodiment of the present application. As shown in fig. 5, in the network architecture of the payment method of the intelligent payment device based on the multiple verification mechanism, firstly, a face image (e.g., P1 as illustrated in fig. 5) of the obtained object to be verified is passed through a first convolutional neural network model (e.g., CNN1 as illustrated in fig. 5) as a face area detection network to obtain a face feature map (e.g., F1 as illustrated in fig. 5); then, based on the position of the eyeball in the face image, extracting an eyeball region feature map corresponding to the eyeball from the face feature map (for example, as illustrated in fig. 5, F2); then, passing the eyeball region feature map through a generator model (e.g., GM as illustrated in fig. 5) as a pixel enhancer to obtain a generated eyeball region image (e.g., F3 as illustrated in fig. 5); then, passing the generated eye region image through a second convolutional neural network model (e.g., CNN2 as illustrated in fig. 5) with a saliency detection module to obtain an iris feature map (e.g., F4 as illustrated in fig. 5); then, based on the face feature map, weighting the iris feature map using a self-attention-based data dense cluster mechanism (e.g., DCM as illustrated in fig. 5) to obtain a weighted iris feature map (e.g., F as illustrated in fig. 5); then, fusing the weighted iris feature map and the face feature map to obtain a classification feature map (for example, FC as illustrated in fig. 5); then, the classification feature map is passed through a classifier (e.g., a classifier as illustrated in fig. 5) to obtain a classification result, where the object to be detected corresponds to an object label in a database; and finally, paying the fee required by the bus based on the amount of money in the payment account associated with the object tag. Inputting the obtained input signal waveform (e.g., IN1 as illustrated IN fig. 5) and output signal waveform (e.g., IN2 as illustrated IN fig. 4) of the current time point into a convolutional neural network (e.g., CNN as illustrated IN fig. 4), respectively, to obtain an input signal profile (e.g., F1 as illustrated IN fig. 4) and an output signal profile (e.g., F2 as illustrated IN fig. 4); then, pooling the input signal profile and the output signal profile along a channel dimension to obtain an input profile matrix (e.g., M1 as illustrated in fig. 4) and an output profile matrix (e.g., M2 as illustrated in fig. 4), respectively; then, passing the eyeball region feature map through a generator model as a pixel enhancer to obtain a generated eyeball region image (e.g., MT1 as illustrated in fig. 4); then, processing the obtained input signal waveform (for example, IN3 as illustrated IN fig. 4) and output signal waveform (for example, IN4 as illustrated IN fig. 4) of a series of predetermined time points before the current time point IN a manner of processing the input signal waveform and output signal waveform of the current time point, respectively, to obtain a second to an nth transfer matrix (for example, MT2-MTn as illustrated IN fig. 4); then, fusing the weighted iris feature map and the face feature map to obtain a classification feature map (e.g., MD as illustrated in fig. 4); then, a correlation coordinate estimation is performed on the distance matrix to obtain a classification matrix (e.g., MC as illustrated in fig. 4); and, finally, passing the classification matrix through an encoder for regression (e.g., E as illustrated in fig. 4) to obtain the required duty cycle of the PWN excitation wave.

More specifically, in step S110 and step S120, a face image of an object to be verified acquired by a camera of the smart payment device is acquired, and the face image of the object to be verified is passed through a first convolutional neural network model as a face area detection network to obtain a face feature map. It should be understood that, considering that face-brushing payment is a common non-inductive payment means, in the field of public transportation, face-brushing payment cannot well meet the requirements of application scenarios. The reason is that people can make various decorations when going out, for example, women can make up and dress up and wear sun shading equipment, so that people cannot accurately recognize faces when brushing faces to pay, and payment cannot be made or wrong payment is made. Therefore, in the technical scheme of the application, the safety and the accuracy of payment are expected to be improved by combining the face recognition and the iris recognition.

Specifically, in the technical scheme of the application, firstly, a camera of the intelligent payment device is used for collecting a face image of an object to be verified. Then, a convolution neural network with excellent performance in the aspect of local implicit feature extraction of the image is used for deep mining of implicit features of the face image of the object to be verified. It should be understood that the target detection method based on deep learning divides the network into two categories, Anchor-based (Anchor-based) and Anchor-free (Anchor-free) according to whether an Anchor window is used in the network. Such as Fast R-CNN, Faster R-CNN, RetinaNet, etc., and anchorless window-based methods such as CenterNet, ExtremNet, RePoints, etc. The method based on the anchor window can enable regression of target classification and bounding box coordinates among networks, training is stable by adding prior, the recall capability of network targets can be effectively improved, and the method is very obvious in improvement of small target detection. Therefore, in the technical scheme of the application, the face image of the object to be verified is processed in the first convolution neural network model serving as the face area detection network to extract the local high-dimensional implicit feature distribution in the face image so as to obtain the face feature map. In particular, here, the first convolutional neural network model is Fast R-CNN, Fast R-CNN or RetinaNet.

More specifically, in step S130 and step S140, based on the position of the eyeball in the face image, an eyeball region feature map corresponding to the eyeball is extracted from the face feature map, and the eyeball region feature map is passed through a generator model as a pixel enhancer to obtain a generated eyeball region image. That is, in order to improve the security and accuracy of payment by combining face recognition and iris recognition, an eyeball region feature map corresponding to an eyeball is extracted from the face feature map further based on the position of the eyeball in the face image. It should be understood that, here, the convolutional neural network is utilized to not deform at the position of extracting the feature, and therefore, in the technical solution of the present application, the eyeball area feature map corresponding to the eyeball in the face feature map may be determined according to the position of the eyeball in the original image in the face image.

More specifically, in step S150, the generated eyeball area image is passed through a second convolutional neural network model with a saliency detection module to obtain an iris feature map. That is, in the technical solution of the present application, in this way, the generated eyeball region image is subjected to local high-dimensional implicit feature extraction in a second convolutional neural network model with a saliency detection module, so as to obtain an iris feature map corresponding to the eyeball region.

More specifically, in step S160, based on the face feature map, the iris feature map is weighted by using a self-attention-based data dense cluster mechanism to obtain a weighted iris feature map, wherein the weighting of the iris feature map by using the self-attention-based data dense cluster mechanism is performed based on a spatial interaction feature map obtained by multiplying the face feature map and the iris feature map by a position point, and a distance between the face feature map and the iris feature map. It should be understood that when the iris feature map is obtained, the local region in the image is extracted by using the candidate frame as the reference window, and then the iris feature map is obtained by resolution enhancement and saliency detection, which makes the iris feature map higher than the face feature map in data density, if the face feature map is directly fused, the iris feature map is higher than the face feature mapF _face And the iris feature mapF _iris A classification bias of the fused feature map with respect to the face feature map and the iris feature map may result. Therefore, in the technical solution of the present application, the iris feature map is further weighted by using a self-attention-based data dense cluster mechanism.

Accordingly, in a specific example, first, the position-based point multiplication between the face feature map and the iris feature map is calculated to obtain the spatial interaction feature map. Then, the square root of the Euclidean distance between the face feature image and the iris feature image is calculated. Then, the feature value of each position in the space interaction feature map is divided by the square root of the Euclidean distance between the face feature map and the iris feature map to obtain an attention feature map. Then, a natural exponent function value raised by a feature value of each position in the attention feature map is calculated to obtain an exponential attention feature map. The exponential attention feature map is then passed through the classifier to obtain a first class probability index. Then, the iris feature map is passed through the classifier to obtain a second class probability index. Then, the product of the first class probability index and the second class probability index is calculated as the weighting coefficient of the iris feature map. And finally, weighting the iris characteristic diagram by the weighting coefficient to obtain the weighted iris characteristic diagram. It should be appreciated that the self-attention based data-dense clustering mechanism enables spatial interaction of local features with global features based on a reference window and expresses the similarity between data-dense object instances by a measure of dissimilarity in the data represented by the feature map distance. Therefore, the iris feature map is weighted and then fused with the face feature map, so that the parameter adaptive variability of the fused feature map to a classification target function can be improved through the adaptive dependence of the data dense cluster in the iris feature map, and the classification accuracy is improved.

More specifically, in step S170, the weighted iris feature map and the face feature map are fused to obtain a classification feature map. It should be understood that, in this way, by using the self-attention-based data dense cluster mechanism to weight the iris feature map and then fuse the iris feature map with the face feature map, the parameter adaptive variability of the fused classification feature map on the classification target function can be improved through the adaptive dependence of the data dense cluster in the iris feature map, thereby improving the classification accuracy.

More specifically, in step S180 and step S190, the classification feature map is passed through a classifier to obtain a classification result, the classification result is that the object to be detected corresponds to the object tag in the database, and the fee required for the ride is paid based on the amount of money in the payment account associated with the object tag. That is, in the technical solution of the present application, after the classification feature map is obtained, the classification feature map is further passed through a classifier to obtain a classification result indicating that an object to be checked corresponds to an object tag in a database, and further, based on an amount in a payment account associated with the object tag, a fee required for taking a car is paid. Accordingly, in one particular example, an under-fare prompt is generated in response to the amount of money in the payment account associated with the object tag being less than the fare required for the ride.

In summary, the payment method of the intelligent payment device based on the multiple verification mechanisms is clarified, the deep local implicit associated feature information is mined from the face image and the eyeball area image of the object to be verified through the deep neural network model to improve the safety and the accuracy of payment by combining face recognition and iris recognition, and in the process, in order to improve the fusion effect and improve the accuracy of classification, a data dense cluster mechanism based on self-attention is further used for weighting the iris feature map, so that the parameter adaptive variability of the fused feature map on the classification target function can be improved through the adaptive dependence of the data dense cluster in the iris feature map to improve the classification accuracy, and the safety and the convenience of payment are improved.

The foregoing describes the general principles of the present application in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present application are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the foregoing disclosure is not intended to be exhaustive or to limit the disclosure to the precise details disclosed.

The block diagrams of devices, apparatuses, systems referred to in this application are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".

It should also be noted that in the devices, apparatuses, and methods of the present application, the components or steps may be decomposed and/or recombined. These decompositions and/or recombinations are to be considered as equivalents of the present application.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims

1. An intelligent payment device based on multiple verification mechanism, comprising:

the face acquisition module is used for acquiring a face image of an object to be verified, which is acquired by a camera of the intelligent payment equipment;

a feature distribution correction module, configured to weight the iris feature map based on the face feature map by using a self-attention-based data dense cluster mechanism to obtain a weighted iris feature map, where the weighting of the iris feature map by using the self-attention-based data dense cluster mechanism is performed based on a spatial interaction feature map obtained by multiplying the face feature map and the iris feature map by location points, and a distance between the face feature map and the iris feature map;

the payment module is used for paying the fee required by the riding based on the amount of money in the payment account associated with the object tag;

wherein the feature distribution correction module comprises:

the space interaction feature map generation unit is used for calculating the point-by-point multiplication between the face feature map and the iris feature map to obtain the space interaction feature map;

the data difference measurement unit is used for calculating the square root of the Euclidean distance between the face feature map and the iris feature map;

the attention unit is used for dividing the characteristic value of each position in the space interaction characteristic map by the square root of the Euclidean distance between the face characteristic map and the iris characteristic map to obtain an attention characteristic map;

an exponential operation unit, which is used for calculating a natural exponential function value taking the characteristic value of each position in the attention characteristic map as power to obtain an exponential attention characteristic map;

a first class probability unit, configured to pass the exponential attention feature map through the classifier to obtain a first class probability index;

the second-class probability unit is used for enabling the iris feature map to pass through the classifier to obtain a second-class probability index;

the action unit is used for calculating the product between the first class probability index and the second class probability index as a weighting coefficient of the iris feature map; and

and the correction unit is used for weighting the iris characteristic diagram by the weighting coefficient to obtain the weighted iris characteristic diagram.

2. The intelligent payment device based on multiple verification mechanism of claim 1, wherein the first convolutional neural network model is Fast R-CNN, Fast R-CNN or RetinaNet.

3. The intelligent payment device based on multiple verification mechanisms of claim 2, wherein said iris feature extraction module is further configured to encode the input data in the following manner in each layer of said second convolutional neural network:

performing convolution coding on the input data by using first convolution units of each layer of the second convolution neural network and first convolution cores to obtain a convolution characteristic diagram;

performing reel-to-reel encoding on the convolution feature map by using a second convolution unit of each layer of the second convolution neural network and a second convolution kernel to obtain a re-convolution feature map, wherein the first convolution unit and the second convolution unit form the significant feature detection module, and the size of the first convolution kernel is larger than that of the second convolution kernel;

performing mean pooling based on a local feature matrix on the re-convolution feature map by using pooling units of each layer of the second convolution neural network to obtain a pooled feature map; and

carrying out nonlinear activation on the feature values of all positions in the pooled feature map by using the activation units of all layers of the second convolutional neural network to obtain an activation feature map;

wherein the output of the last layer of the second convolutional neural network is the iris feature map.

4. The intelligent payment device based on multiple verification mechanisms of claim 3, wherein the feature distribution correction module is further configured to weight the iris feature map using a self-attention based data dense cluster mechanism to obtain the weighted iris feature map according to the following formula;

wherein the formula is:

5. The intelligent payment device based on multiple verification mechanism of claim 4, wherein the feature distribution fusion module is further configured to: fusing the weighted iris feature map and the face feature map according to the following formula to obtain the classification feature map;

wherein the formula is:

F=αwF _iris +βF _face

6. The intelligent payment device based on multiple verification mechanism of claim 5, wherein the verification result generation module is further configured to: the classifier processes the classification feature map to generate a classification result according to the following formula;

wherein the formula is:softmax{(W _n , B _n ):...:(W ₁ , B ₁ )|Project(F) Therein ofProject(F) Representing the projection of the classification feature map as a vector,W ₁ toW _n Is a weight matrix of the fully connected layers of each layer,B ₁ toB _n A bias matrix representing the layers of the fully connected layer.

7. The intelligent payment device based on multiple verification mechanisms of claim 6, wherein the payment module is further configured to generate an under-fare prompt in response to the amount of money in the payment account associated with the object tag being less than the fare required for the ride.

8. A payment method of intelligent payment equipment based on multiple verification mechanisms is characterized by comprising the following steps:

enabling the generated eyeball area image to pass through a second convolution neural network model with a significance detection module to obtain an iris characteristic diagram;

enabling the classification characteristic graph to pass through a classifier to obtain a classification result, wherein the classification result is that an object to be detected corresponds to an object label in a database; and

paying the fee required by the bus taking based on the amount of money in the payment account associated with the object tag;

based on the face feature map, weighting the iris feature map by using a data dense cluster mechanism based on self attention to obtain a weighted iris feature map, wherein the method comprises the following steps of:

calculating the position-based multiplication between the face feature map and the iris feature map to obtain the space interaction feature map;

calculating the square root of the Euclidean distance between the face feature image and the iris feature image;

dividing feature values of all positions in the space interaction feature map by the square root of the Euclidean distance between the face feature map and the iris feature map to obtain an attention feature map;

calculating natural exponent function values raised by the feature values of the positions in the attention feature map to obtain an exponent attention feature map;

passing the exponential attention feature map through the classifier to obtain a first class probability index;

passing the iris feature map through the classifier to obtain a second class probability index;

calculating the product of the first class probability index and the second class probability index as the weighting coefficient of the iris feature map; and

and weighting the iris characteristic diagram by the weighting coefficient to obtain the weighted iris characteristic diagram.