CN114429223A

CN114429223A - Heterogeneous model establishing method and device

Info

Publication number: CN114429223A
Application number: CN202210091039.3A
Authority: CN
Inventors: 卞阳; 张翔; 陈立峰; 李腾飞; 邢旭; 张伟奇
Original assignee: Shanghai Fudata Technology Co ltd
Current assignee: Shanghai Fudata Technology Co ltd
Priority date: 2022-01-26
Filing date: 2022-01-26
Publication date: 2022-05-03
Anticipated expiration: 2042-01-26
Also published as: CN114429223B

Abstract

The application provides a heterogeneous model building method and device, which relate to the field of deep learning models and comprise the following steps: acquiring information representing roles of other nodes participating in modeling; issuing algorithm information to a task service machine to determine a target modeling algorithm; performing collaborative modeling operation with a task server based on a target modeling algorithm, wherein the collaborative modeling operation comprises the following steps: calculating a loss value of the preliminary heterogeneous model based on a target modeling algorithm, determining an update gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the update gradient value; and determining whether the preliminary heterogeneous model is converged according to the weight value and the loss value, and obtaining the target heterogeneous model when the preliminary heterogeneous model is determined to be converged. By adopting the method provided by the embodiment of the application, the process of establishing the target heterogeneous model can be interconnected and communicated among all federal learning platforms in the process of federal learning, and meanwhile, the safety in collaborative modeling is improved.

Description

Heterogeneous model establishing method and device

Technical Field

The application relates to the field of deep learning, in particular to a heterogeneous model building method and device.

Background

The federated learning is a way for jointly establishing a machine learning model under the premise that a plurality of participants cannot acquire data of each other, and the heterogeneous model means that each federated learning platform realizes a machine learning algorithm by using a data structure design of the platform.

At present, due to various reasons such as heterogeneous data structures of all platforms, non-uniform federal logistic regression frameworks, various training methods, non-uniform communication protocols and the like, the training process of a heterogeneous model cannot be interconnected and communicated on all federal learning platforms.

Disclosure of Invention

Based on the above, an object of the embodiments of the present application is to provide a method and an apparatus for building a heterogeneous model, so that the built target heterogeneous model can be interconnected and intercommunicated between all federal learning platforms, and a model application scenario is expanded.

In a first aspect, an embodiment of the present application provides a heterogeneous model building method applied to a task request machine, including:

acquiring information representing roles of other nodes participating in modeling, wherein the roles comprise a task request machine and a task service machine, and the roles are determined based on a modeling scene and holding data of the modeling nodes;

issuing algorithm information to the task service machine to determine a target modeling algorithm;

performing collaborative modeling operation with the task server based on the target modeling algorithm, wherein the collaborative modeling operation comprises: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value;

and determining whether the preliminary heterogeneous model is converged according to the weight value and the loss value, and obtaining a target heterogeneous model when the preliminary heterogeneous model is determined to be converged.

In the implementation process, the heterogeneous federated logistic regression algorithm can keep consistent in blocks based on the determined target modeling algorithm in the collaborative modeling process, and the plurality of federated platforms align and allocate roles of all modeling nodes through operators, so that the federated logistic regression collaborative modeling of different structures or algorithms can be realized, the process of establishing the target heterogeneous model can be interconnected and communicated among all federated learning platforms in the federated learning process, and meanwhile, the safety in collaborative modeling is improved.

Optionally, before the co-modeling operation with the task service machine based on the target modeling algorithm, the method may include:

determining a communication key, a federal homomorphic calculation cipher key or a safety calculation fragment based on the algorithm information issued to the task server;

the performing the collaborative modeling operation with the task server based on the target modeling algorithm may include:

and performing data communication with the task server machine based on the communication key, the federal homomorphic calculation cipher key or the safety calculation fragment so as to ensure that the mathematical definition of the input and output results of the preliminary heterogeneous model constructed by the task server machine is the same during the collaborative modeling operation.

In the implementation process, interaction can be carried out based on a preset communication mode, the cryptographic module of the cryptographic module can be shared with other modeling nodes, and data sharing is achieved based on one-time communication interaction, so that the efficiency of collaborative modeling can be improved.

Optionally, the calculating a loss value of the preliminary heterogeneous model based on the target modeling algorithm may include:

receiving an inner product of the preliminary heterogeneous model sent by the task server machine based on a preset transmission direction, or sending the inner product to the task server machine, wherein the inner product is obtained by the current iteration model weight and the sample characteristic value of the preliminary heterogeneous model;

and substituting the inner product into a preset function to obtain the loss value and sending the loss value to the task service machine.

Optionally, the updating the weight values of the preliminary heterogeneous model based on the updated gradient values may include:

determining the updated gradient value according to the loss value and the sample characteristic value;

determining a weight value of the preliminary heterogeneous model after updating based on the updating gradient value and the learning rate of the preliminary heterogeneous model.

In a second aspect, an embodiment of the present application provides a method for building a heterogeneous model applied to a task service machine, including:

receiving algorithm information issued from the task request machine, and determining a target modeling algorithm;

performing collaborative modeling operation with the task request machine based on the target modeling algorithm, wherein the collaborative modeling operation comprises: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value;

In the implementation process, the heterogeneous federated logistic regression algorithm can keep consistent in partitioning by a determined target modeling algorithm in the collaborative modeling process, and a plurality of federated platforms align and distribute roles of modeling nodes through operators, so that federated logistic regression collaborative modeling of different structures or algorithms can be realized, and the established target heterogeneous model can be interconnected and intercommunicated among federated learning platforms.

determining a communication key, a federal homomorphic calculation cipher key or a safety calculation fragment based on the algorithm information issued from the task request machine;

the performing the collaborative modeling operation with the task requester based on the target modeling algorithm may include:

and performing data communication with the task request machine based on the communication key, the federal homomorphic calculation cipher key or the safety calculation fragment so as to ensure that the mathematical definition of the input and output results of the preliminary heterogeneous model constructed by the task request machine is the same during the collaborative modeling operation.

Optionally, the calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm includes:

receiving an inner product of the preliminary heterogeneous model sent by the task request machine based on a preset transmission direction, or sending the inner product to the task request machine, wherein the inner product is obtained by the current iteration model weight and the sample characteristic value of the preliminary heterogeneous model;

and receiving the loss value sent by the task request machine, wherein the loss value is obtained by substituting the inner product into a preset function by the request service machine.

Optionally, the updating the weight values of the preliminary heterogeneous model based on the updated gradient values comprises:

In a third aspect, an embodiment of the present application provides a heterogeneous model building device applied to a task request machine, including:

the system comprises a first acquisition module, a first processing module and a second processing module, wherein the first acquisition module is used for acquiring information representing roles of other nodes participating in modeling, the roles comprise a task request machine and a task service machine, and the roles are determined based on a modeling scene and holding data of the modeling nodes;

the issuing module is used for issuing algorithm information to the task service machine and determining a target modeling algorithm;

a first collaborative modeling module, configured to perform collaborative modeling operations with the task service machine based on the target modeling algorithm, where the collaborative modeling operations include: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an update gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the update gradient value;

and the first judging module is used for determining whether the preliminary heterogeneous model is converged according to the weight value and the loss value, and obtaining a target heterogeneous model when the preliminary heterogeneous model is determined to be converged.

Optionally, the heterogeneous model building apparatus may further include a first communication module, configured to determine a communication key, a federal homomorphic calculation cryptographic key, or a security calculation fragment based on the algorithm information sent to the task server.

The first collaborative modeling module may be specifically configured to perform data communication with the task server based on the communication key, the federal homomorphic computation cryptographic key, or the security computation fragment, so as to be identical to a mathematical definition of an input/output result of a preliminary heterogeneous model constructed by the task server during the collaborative modeling operation.

Optionally, the first collaborative modeling module may be further specifically configured to receive an inner product of the preliminary heterogeneous model sent by the task service machine based on a preset transmission direction, or send the inner product to the task service machine, where the inner product is obtained from a current iteration model weight and a sample feature value of the preliminary heterogeneous model; and substituting the inner product into a preset function to obtain the loss value and sending the loss value to the task service machine.

And determining the updated gradient value according to the loss value and the sample characteristic value; determining a weight value of the preliminary heterogeneous model after updating based on the updating gradient value and the learning rate of the preliminary heterogeneous model.

In a fourth aspect, an embodiment of the present application further provides an apparatus for building a heterogeneous model applied to a task service machine, including:

the second acquisition module is used for acquiring information representing roles of other nodes participating in modeling, wherein the roles comprise a task request machine and a task service machine, and the roles are determined based on modeling scenes and holding data of the modeling nodes;

the receiving module is used for receiving the algorithm information issued by the task request machine and determining a target modeling algorithm;

a second collaborative modeling module, configured to perform collaborative modeling operation with the task requester based on the target modeling algorithm, where the collaborative modeling operation includes: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value;

and the second judgment module is used for determining whether the preliminary heterogeneous model is converged according to the weight value and the loss value, and obtaining a target heterogeneous model when the preliminary heterogeneous model is determined to be converged.

In the implementation process, the heterogeneous federated logistic regression algorithm can keep consistent in blocks based on the determined target modeling algorithm in the collaborative modeling process, and the plurality of federated platforms align and allocate roles of all modeling nodes through operators, so that the federated logistic regression collaborative modeling of different structures or algorithms can be realized, and the established target heterogeneous models can be interconnected and intercommunicated among all federated learning platforms.

Optionally, the heterogeneous model building apparatus may further include a second communication module, configured to determine a communication key, a federal homomorphic calculation cryptographic key, or a security calculation fragment based on the algorithm information issued from the task requester.

The second collaborative modeling module may be specifically configured to perform data communication with the task requester based on the communication key, the federal homomorphic computation cryptographic key, or the security computation fragment, so that the mathematical definition of the input and output results of the preliminary heterogeneous model constructed by the task requester is the same during the collaborative modeling operation.

Optionally, the second collaborative modeling module may be further specifically configured to receive an inner product of the preliminary heterogeneous model sent by the task requester based on a preset transmission direction, or send the inner product to the task requester, where the inner product is obtained from a current iteration model weight and a sample feature value of the preliminary heterogeneous model; and receiving the loss value sent by the task request machine, wherein the loss value is obtained by substituting the inner product into a preset function by the request service machine.

And determining the update gradient value according to the loss value and the sample characteristic value; determining a weight value of the preliminary heterogeneous model after updating based on the updating gradient value and the learning rate of the preliminary heterogeneous model.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.

Fig. 1 is a schematic step diagram of a heterogeneous model building method applied to a task request machine according to an embodiment of the present application;

FIG. 2 is a schematic diagram illustrating a step of a task requester communicating with a task server according to an embodiment of the present disclosure;

FIG. 3 is a schematic diagram illustrating steps of collaborative modeling provided by an embodiment of the present application;

fig. 4 is a schematic step diagram of a heterogeneous model building method applied to a task server according to an embodiment of the present application;

FIG. 5 is a schematic diagram illustrating a step of a task server communicating with a task requester according to an embodiment of the present application;

FIG. 6 is a schematic diagram illustrating steps applied to collaborative modeling of a task server according to an embodiment of the present application;

fig. 7 is a schematic diagram of a heterogeneous model building apparatus applied to a task request machine according to an embodiment of the present application;

fig. 8 is a schematic diagram of a heterogeneous model building device of a task server according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. For example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. In addition, the functional modules in the embodiments of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

The applicant finds that the current model building mode based on the federal learning is divided into a longitudinal federal learning mode and a transverse federal learning mode according to different application scenes of data and users, wherein the longitudinal federal learning mode is to divide data sets according to the longitudinal direction (characteristic dimension) under the condition that the users of the two data sets overlap more and the user characteristics overlap less, and take out the part of data which is the same as the users of the two data sets but has not the same user characteristics for training. Horizontal federated learning is opposite to vertical, and both data are learned and modeled under the scene with the same characteristics but not completely the same users. Each federal learning platform uses a data structure design thereof to realize a machine learning algorithm, and because of the reasons of non-uniform federal logistic regression framework, different training methods, non-uniform communication protocols and heterogeneous data structures, each platform has different structures at the algorithm level. Therefore, heterogeneous models cannot be interconnected and communicated among all federal learning platforms, and the problem that the application scene of the models is limited exists.

Based on the above, the embodiment of the application provides a heterogeneous model establishing method, and the heterogeneous model is established by modularizing a logistic regression algorithm calculation operator and cooperatively establishing a plurality of modeling nodes based on preset module attributes, so that the established target heterogeneous model can be interconnected and intercommunicated among all federal learning platforms. Referring to fig. 1, fig. 1 is a schematic step diagram of a heterogeneous model building method applied to a task request machine according to an embodiment of the present application, where an implementation manner of the heterogeneous model building may include the following steps:

in step S11, information characterizing roles of other participating modeling nodes is obtained, the roles including a task request machine and a task service machine, and the roles are determined based on a modeling scenario and holding data of the modeling nodes.

The modeling node role can be each federal learning platform, the federal learning platform can operate on equipment with a communication function and a data processing function, and the equipment can be a computer, a server, a configurator of engineering equipment and a cloud server.

The modeling scene represents a training mode of the modeling node and an application scene of a user, a model learning mode of the modeling node is determined to be longitudinal federal learning or horizontal federal learning, and held data refers to whether the model in the modeling node has an output value Y. And judging the alignment role by acquiring the information of the roles of other modeling nodes, and determining each modeling node as a task request machine Guest or a task service machine Host. The task request machine Guest serves as a calculation leading party, a Y value is held to be responsible for calculating a loss value in a longitudinal scene, a task initiator generally serves as Guest in a transverse scene, a task service machine Host serves as a participating calculating party, and a non-Y-value party in the longitudinal scene. The task request machine and the task service machine can be electronic equipment running the federal learning platform.

In step S12, algorithm information is issued to the task service machine to determine a target modeling algorithm.

The modeling algorithm can be a secureagglgregation safety aggregation algorithm, a secuumlr safety logistic regression algorithm and other algorithms capable of realizing logistic regression, the algorithms can be pre-configured in the algorithm modules, and when the corresponding modeling algorithm is selected as the target modeling algorithm, the corresponding algorithm module can be selected to perform collaborative modeling operation based on a unified operator.

Illustratively, after information representing roles of other modeling nodes participating in modeling is acquired, each modeling node checks task categories, determines that a collaborative modeling mode is transverse modeling or longitudinal modeling, determines that each modeling node serves as a task request machine Guest or a task service machine Host according to sample content of each modeling node, such as Y value position, data volume and the like, and calls an algorithm module agreed with the role of each modeling node to perform modeling according to the assigned role.

In step S13, performing a collaborative modeling operation with the task server based on the target modeling algorithm, the collaborative modeling operation including: the method includes calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an update gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the update gradient value.

The loss value of the preliminary heterogeneous model can be calculated by a sigmoid function or a function for approximately calculating the sigmoid through the current model weight and the sample characteristic value, or through the current model vector and the sample characteristic vector. In the method for establishing the heterogeneous model provided in the embodiment of the present application, the update gradient value may be calculated according to the loss value based on a gradient descent method or a random average gradient method, and the weight value may be updated based on the update gradient value obtained by calculation.

In step S14, it is determined whether the preliminary heterogeneous model converges according to the weight value and the loss value, and when it is determined that the preliminary heterogeneous model converges, a target heterogeneous model is obtained.

For example, a weight change threshold and a loss threshold may be preset, and when the change value of the weight is lower than the weight change threshold and the loss value is lower than the loss threshold, it may be determined that the weight and the loss value do not change in a large range any more, the preliminary heterogeneous model has converged, and the training of the characterization model is completed, so as to obtain the target heterogeneous model that is completely built.

Therefore, in the embodiment of the application, the heterogeneous federated logistic regression algorithm can keep consistent blocks based on the determined target modeling algorithm in the collaborative modeling process, and the plurality of federated platforms align and allocate roles of all modeling nodes through operators, so that federated logistic regression collaborative modeling of different structures or algorithms can be realized, the process of establishing the target heterogeneous model can be interconnected and communicated among all federated learning platforms in the federated learning process, and meanwhile, the safety in collaborative modeling is improved.

In an alternative embodiment, before performing the collaborative modeling operation with the task server based on the target modeling algorithm in step S11, the embodiment of the present application provides an implementation manner of communicating with other participating modeling nodes. Referring to fig. 2, fig. 2 is a schematic diagram illustrating a step of a task requester communicating with a task server according to an embodiment of the present application, where the step of the task requester communicating with the task server may include:

in step S21, a communication key, a federal homomorphic calculation cipher key, or a security calculation fragment is determined based on the algorithm information issued to the task server.

The task request machine Guest issues an encryption mode which is agreed in advance, the encryption mode can be homomorphic encryption, noise confusion, multi-party safety calculation and the like, when the encryption mode is homomorphic encryption, an encrypted object is correspondingly sent to exchange a public key and a private key, when the encryption mode is noise confusion, a noise cancellation calculation item is correspondingly sent, and when the encryption mode is multi-party safety calculation, an encryption fragment is correspondingly sent. The federal homomorphic calculation cipher key is a key for transmitting data when homomorphic encryption is used in the federal learning process.

In step S22, data communication is performed with the task server machine based on the communication key, the federal homomorphic calculation cipher key or the security computation fragment so as to be identical to the mathematical definition of the input and output results of the preliminary heterogeneous model constructed by the task server machine in the collaborative modeling operation.

For example, the encryption algorithm may also be pre-configured in the encryption algorithm module, and when a corresponding encryption algorithm is selected for communication, data is encrypted or decrypted by the corresponding cryptographic module, and each platform may share its own cryptographic module with other platforms in an SDK manner. In addition, the platforms can communicate with each other based on a general communication protocol or based on a general cryptographic data structure.

The algorithm information includes contents which need to be defined in a data structure, such as encryption, noise addition or fragmentation data of an algorithm module before transmission, a communication encryption mode, whether output data needs to be decrypted or not, a public key of a cryptographic algorithm, a data structure of a private key ciphertext and the like.

And in the process of modeling calculation, a corresponding field conversion rule is sent between the task request machine Guest and the task service machine Host, so that the result mathematical definitions of module input and output are the same, and data sharing can be realized based on one-time communication interaction.

Therefore, the method and the device for the collaborative modeling can interact based on a preset communication mode, share the cryptographic module with other modeling nodes, realize data sharing based on one-time communication interaction, and improve the efficiency of the collaborative modeling.

In an optional embodiment, for step S13, an implementation manner of collaborative modeling is provided in the embodiment of the present application, please refer to fig. 3, where fig. 3 is a schematic diagram of steps of collaborative modeling provided in the embodiment of the present application, and the steps of collaborative modeling may include:

in step S31, an inner product of the preliminary heterogeneous model sent by the task service machine is received based on a preset transmission direction, or the inner product is sent to the task service machine, where the inner product is obtained from a current iteration model weight and a sample characteristic value of the preliminary heterogeneous model.

In step S32, the inner product is substituted into a preset function to obtain the loss value, and the loss value is sent to the task server.

The inner product calculation process needs the cooperation of the two parties for calculation, the calculated inner product can be selected to be encrypted and sent or an original text can be sent according to the safety level requirement and the difference of the algorithm framework, and finally the inner product is calculated by the task request machine to obtain a loss value.

In step S33, the update gradient value is determined according to the loss value and the sample feature value.

In step S34, a weight value after the preliminary heterogeneous model is updated is determined based on the update gradient value and the learning rate of the preliminary heterogeneous model.

For example, the algorithms for calculating the inner product, the loss value, the update gradient value, and updating the weight may be pre-configured in the corresponding calculation modules, and the corresponding calculation modules are invoked when the corresponding calculation steps are performed, for example, the inner product calculation module is configured to calculate the inner product of the current modeling node based on a formula U ═ wx, where U is the inner product of the current modeling node, w is the current model weight, and x is the sample feature value; the Loss value calculating module is used for calculating a corresponding Loss value Loss by inner product according to Sigmoid function or function for approximately calculating Sigmoid, and the updating gradient value calculating module is used for calculating the Loss value Loss based on the formula g ═ sig (wx) -y]Calculating an updated gradient value, wherein g is a gradient, sig (wx) represents a loss value, y is an output value of the current model, and x is a sample characteristic value of the model; the weight updating module is used for updating the weight based on the formula w_newCalculating the updated weight as a × g, where a is the model learning rate, g is the gradient, and w is_newIs the updated gradient value.

The updating gradient value calculating module and the loss value calculating module need to align two parties when calculating, the updating gradient value delta w can use a gradient descent method or a random average gradient method, and other methods, operators of the two modules are calculated locally, the communication event is to send a secret object of the communication event to a decryption party for decryption, wherein the secret object refers to encrypted data.

In addition, the task request machine Guest and the task service machine Host can establish a transmitting and receiving party for transmitting and receiving the communication dictionary according to roles, such as: the data direction is from Guest to Host. The interaction event name conversion dictionary in operator communication can be from a participant event name to a heterogeneous model event name: if 'b _ 1': interc _1 ',' interc _1 ': b _ 1'. And transmitting a data structure conversion function interface in operator communication.

If the intermediate communication involves inconsistent software program structures, the two parties need to exchange the translator interfaces to complete the interconnection. Each platform can share its own logistic regression communication module with other platforms.

Illustratively, taking an application scenario of homomorphic encryption three-party modeling as an example, each platform participating in collaborative modeling judges roles in the modeling process according to own data, and the roles in the application scenario include a task request machine Guest, a task service machine Host and a third party Abiter.

And (3) mutually communicating and exchanging algorithm information among all roles participating in the collaborative modeling, selecting the same algorithm from the logistic regression algorithm library, and corresponding all operator modules into the same operator.

The task request machine Guest and the task service machine Host exchange cipher module information, the same cipher module is used for generating a public key and a private key, such as paillier, and a third party arbiter decrypts the information during communication interaction.

The task request machine Guest and the task service machine Host choose to use the corresponding inner product calculation module to calculate the inner product, wherein the transmission direction of communication is that the task service machine Host sends the calculated inner product to the task request machine Guest, and the communication can be an encryption mode based on addition homomorphism (Paillier). And calculating an inner product ub of the task server Host, encrypting the ub to obtain ubs, and sending ubs of a secret state to the task request machine Guest. And the task request machine Guest calculates an inner product ua obtained according to own data and adds ub to obtain an inner product u, and calculates a model weight w and a sample characteristic value x of the heterogeneous model according to the inner product u.

The task request machine Guest and the task service machine Host use the selection Loss value calculation module to calculate the Loss value, and the task request machine Guest calculates the Loss value Loss of the heterogeneous model based on the Sigmoid function or the approximate Sigmoid function and sends the Loss value Loss to the task service machine Host.

And both the task request machine Guest and the task service machine Host select to use the updating gradient value calculation module to calculate the updating gradient value delta w according to the loss value. And the calculated updating gradient value delta w is sent to a third party Arbiter and decrypted by the third party Arbiter, the transmitted cryptographic protocol can be an addition homomorphic encryption mode in the steps and is accompanied by a random number, the third party Arbiter decrypts the cryptographic gradient value delta w to obtain the updating gradient value delta w, and returning a plaintext to both the task request machine Guest and the task service machine Host and attaching an updated gradient value delta w and a Loss value Loss of a random number, the task request machine Guest and the task service machine Host subtracting the random number from the obtained value to obtain an updated gradient value delta w and a Loss value Loss, updating the model weight w according to the updating gradient value delta w and judging whether the current model is converged according to whether the weight w and the Loss value Loss change in a large range or not, and when the model is not converged, repeating the steps from calculating the inner product to updating the model weight w until the model is converged to obtain the target heterogeneous model.

Optionally, the method provided by the application can also be applied to a scene of homomorphic encryption modeling of two parties, and in the application scene, each participant needs to judge that a role is a task request machine Guest or a task service machine Host according to data of the participant. The task request machine Guest and the task service machine Host exchange algorithm information, the same algorithm is selected from a self logistic regression algorithm library, each operator module corresponds to the same operator, the information of the cryptographic module is exchanged, and the same cryptographic module is used for generating a public key and a private key.

The task request machine Guest and the task service machine Host choose to use the corresponding inner product calculation module to calculate the inner product, wherein the communication transmission direction is that the task service machine Host sends the calculated inner product to the task request machine Guest, and the communication mode can be plaintext transmission.

And calculating an inner product ub of the task server Host by the task server Host and sending the ub to the task request machine Guest, and calculating an inner product ua of the task request machine Guest by the task request machine Guest.

The task request machine Guest and the task service machine Host use the selection Loss value calculation module to calculate the Loss value, the task request machine Guest sends the calculated Loss value Loss to the task service machine Host, and the communication mode can be an encryption mode based on addition homomorphism.

The task request machine Guest and the task service machine Host select to use the update gradient value calculation module to calculate the update gradient value delta w according to the Loss value, and respectively send the update gradient value delta w and the Loss value Loss calculated by the task request machine Guest and the task service machine Host to the other party, and the communication mode can be a mode based on addition homomorphism and accompanied by random numbers.

And subtracting the random number from the obtained numerical value by the two parties of the task request machine Guest and the task service machine Host to obtain an updated gradient value delta w and a Loss value Loss, updating the model weight w according to the updated gradient value delta w, judging whether the current model is converged according to whether the weight w and the Loss value Loss are changed in a large range, and repeating the steps from the calculation inner product to the updating of the model weight w when the model is not converged until the model is converged to obtain the target heterogeneous model.

Optionally, the method provided by the application can also be applied to a scene of transmission gradient plus noise modeling, and in the application scene, each participant needs to judge that the role is a task request machine Guest or a task service machine Host according to the data of the participant. Exchanging algorithm information between the Guest of the task request machine and the Host of the task service machine, selecting the same algorithm from a logistic regression algorithm library, corresponding each operator module into the same operator, exchanging information of the cipher modules, and calculating a noise cancellation item by using the same noise generation method module.

The task request machine Guest and the task service machine Host choose to use the corresponding inner product calculation module to calculate the inner product, wherein the communication transmission direction is that the task service machine Host sends the calculated inner product to the task request machine Guest, and the communication mode can be plaintext transmission. And calculating an inner product ub of the task server Host by the task server Host and sending the ub to the task request machine Guest, and calculating an inner product ua of the task request machine Guest by the task request machine Guest.

The task request machine Guest and the task service machine Host use the selection Loss value calculation module to calculate the Loss value, the task request machine Guest sends the calculated Loss value Loss to the task service machine Host, and the communication mode can be an encryption mode based on noise.

The task request machine Guest and the task service machine Host select to use the update gradient value calculation module to calculate the update gradient value delta w according to the Loss value, and respectively send the update gradient value delta w and the Loss value Loss calculated by the task request machine Guest and the task service machine Host to the other party, and the communication mode can be a mode based on noise and accompanied by random numbers.

And the task request machine Guest and the task service machine Host eliminate noise based on the cancellation term and subtract the random number from the obtained numerical value to obtain an updated gradient value delta w and a Loss value Loss, the model weight w is updated according to the updated gradient value delta w, whether the current model is converged is judged according to whether the weight w and the Loss value Loss are changed in a large range, and the step from calculating the inner product to updating the model weight w is repeated when the model is not converged until the model is converged to obtain the target heterogeneous model.

Optionally, the method provided by the present application may also be applied to a multi-party secure computing (MPC) scenario, in which each participant needs to determine the role as a task requester Guest or a task server Host according to the data of the other participant. Exchanging algorithm information between the Guest of the task request machine and the Host of the task service machine, selecting the same algorithm from a logistic regression algorithm library, corresponding each operator module to the same operator, exchanging information of the cipher modules, initializing the MPC module by using the same multi-party safety calculation, and fragmenting the y value, w and x.

The task server Host sends the calculated inner product to the task request machine Guest in the communication mode, wherein the communication mode can be based on fragment data structure transmission of MPC fragments. And calculating an inner product ub of the task server Host by the task server Host and sending the ub to the task request machine Guest, and calculating an inner product ua of the task request machine Guest by the task request machine Guest.

The task request machine Guest and the task service machine Host use the selection Loss value calculation module to calculate the Loss value, the task request machine Guest sends the calculated Loss value Loss to the task service machine Host, and the communication mode can be based on fragment data structure transmission of MPC fragments.

And the task request machine Guest and the task service machine Host select to use the update gradient value calculation module to calculate the update gradient value delta w according to the Loss value, and respectively send the update gradient value delta w and the Loss value Loss calculated by the task request machine Guest and the task service machine Host to the other party, wherein the communication mode can be based on fragment data structure transmission of the MPC fragments.

And the task request machine Guest and the task service machine Host obtain a Loss value Loss from a fragment state Loss value based on fragment recovery, update the model weight w according to the update gradient value delta w, judge whether the current model is converged according to whether the weight w and the Loss value Loss change in a large range, repeat the steps from calculating the inner product to updating the model weight w when the model is not converged until the model is converged, determine whether to recover the weight value w of the fragment state according to setting, and finally obtain the target heterogeneous model.

Optionally, the method provided by the application can also be applied to a scenario of two-party horizontal federal logistic regression learning, in the application scenario, the task request machine Guest is the initiator, and the other party is the task service machine Host. The task request machine Guest and the task service machine Host exchange algorithm information, the same algorithm is selected from a self logistic regression algorithm library, each operator module corresponds to the same operator, the information of the cipher modules is exchanged, and the same cipher module is used for generating a public key and a private key and generating a fusion weight random number.

The task request machine Guest and the task service machine Host choose to use the corresponding inner product calculation module to calculate the inner product, wherein the transmission direction of communication is that the task service machine Host sends the calculated inner product to the task request machine Guest, and the communication mode can be an encryption mode based on addition homomorphism and multiplies a random number.

The task request machine Guest and the task service machine Host use the selection Loss value calculation module to calculate the Loss value, the task service machine Host sends the calculated Loss value Loss to the task request machine Guest, the task request machine Guest accumulates the Loss value calculated by the task service machine Host, and the communication mode can be a plaintext transmission mode.

And the task request machine Guest and the task service machine Host select to use the updating gradient value calculation module to calculate the updating gradient value delta w according to the Loss value, encrypt the updating gradient value delta w and the Loss value Loss calculated by the task request machine Guest and the task service machine Host respectively and send the encrypted updating gradient value delta w and the encrypted Loss value Loss to the other party, and the task request machine Guest and the task service machine Host calculate the updating gradient value delta w and the Loss value Loss based on the intermediate result. The communication mode may be a mode based on additive homomorphism and accompanied by a random number.

And the task request machine Guest and the task service machine Host subtract the self random number on the basis of the calculation result obtained by decryption to obtain an updated gradient value, calculate the global updated gradient value delta w of the two parties, update the model weight w according to the global updated gradient value delta w, judge whether the current model is converged according to whether the weight w and the Loss value Loss are changed in a large range, and repeat the steps of calculating the inner product until the model weight w is updated when the model is not converged until the model is converged to obtain the target heterogeneous model.

Based on the same inventive concept, an embodiment of the present application further provides a method for building a heterogeneous model applied to a task service machine, please refer to fig. 4, where fig. 4 is a schematic diagram of steps of the method for building a heterogeneous model applied to a task service machine provided in the embodiment of the present application, and an implementation manner of building a heterogeneous model may include the following steps:

in step S41, information characterizing roles of other participating modeling nodes is obtained, the roles including a task request machine and a task service machine, and the roles are determined based on a modeling scenario and holding data of the modeling nodes.

In step S42, the algorithm information issued from the task request machine is received, and the target modeling algorithm is determined.

In step S43, performing a collaborative modeling operation with the task requester based on the target modeling algorithm, the collaborative modeling operation including: the method includes calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an update gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the update gradient value.

In step S44, it is determined whether the preliminary heterogeneous model converges according to the weight value and the loss value, and when it is determined that the preliminary heterogeneous model converges, a target heterogeneous model is obtained.

Therefore, in the embodiment of the application, the heterogeneous federated logistic regression algorithm can keep consistent in partitioning through the determined target modeling algorithm in the collaborative modeling process, and the plurality of federated platforms align and distribute roles of all modeling nodes through operators, so that federated logistic regression collaborative modeling of different structures or algorithms can be realized, and the established target heterogeneous models can be interconnected and intercommunicated among all federated learning platforms.

Optionally, before the step S41 obtains the information characterizing the roles of the other participating modeling nodes, an implementation manner of communicating with the other participating modeling nodes is provided in the embodiment of the present application, please refer to fig. 5, where fig. 5 is a schematic diagram illustrating a step of communicating between the task server and the task requester provided in the embodiment of the present application, and the step of communicating between the task server and the task requester may include:

in step S51, a communication key, a federal homomorphic calculation cryptographic key, or a security calculation fragment is determined based on the algorithm information issued from the task requester.

In step S52, data communication is performed with the task requester based on the communication key, the federal homomorphic calculation cryptographic key or the security computation fragment so as to be identical to the mathematical definition of the input and output results of the preliminary heterogeneous model constructed by the task requester in the collaborative modeling operation.

Optionally, referring to step S43, an embodiment of the present application provides an implementation manner of collaborative modeling applied to a task server, please refer to fig. 6, where fig. 6 is a schematic diagram of steps of collaborative modeling applied to a task server, and the steps of collaborative modeling of a task server may include:

in step S61, an inner product of the preliminary heterogeneous model sent by the task requester is received based on a preset transmission direction, or the inner product is sent to the task requester, where the inner product is obtained from a current iteration model weight and a sample feature value of the preliminary heterogeneous model.

In step S62, the loss value sent from the task request machine is received, and the loss value is obtained by the request service machine substituting the inner product into a preset function.

In step S63, the update gradient value is determined according to the loss value and the sample feature value.

In step S64, a weight value after the preliminary heterogeneous model is updated is determined based on the update gradient value and the learning rate of the preliminary heterogeneous model.

For a specific implementation manner of the heterogeneous model establishing method applied to the task service machine, an implementation manner of communication with the task request machine, an implementation manner of collaborative modeling, and a specific application scenario, reference may be made to the implementation manner and the application scenario applied to the description of the task request machine, and details are not described here.

Based on the same inventive concept, an embodiment of the present application further provides a heterogeneous model building apparatus 70 applied to a task request machine, please refer to fig. 7, where fig. 7 is a schematic diagram of the heterogeneous model building apparatus applied to the task request machine provided in the embodiment of the present application, and the heterogeneous model building apparatus 70 may include:

the first obtaining module 71 is configured to obtain information representing roles of other participating modeling nodes, where the roles include a task request machine and a task service machine, and the roles are determined based on a modeling scenario and holding data of the modeling nodes.

And the issuing module 72 is used for issuing algorithm information to the task service machine and determining a target modeling algorithm.

A first collaborative modeling module 73, configured to perform collaborative modeling operations with the task server based on the target modeling algorithm, where the collaborative modeling operations include: the method includes calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an update gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the update gradient value.

A first determining module 74, configured to determine whether the preliminary heterogeneous model converges according to the weight value and the loss value, and obtain a target heterogeneous model when it is determined that the preliminary heterogeneous model converges.

Optionally, the heterogeneous model building device 70 may further include a first communication module, configured to determine a communication key, a federal homomorphic calculation cryptographic key, or a security calculation fragment based on the algorithm information sent to the task server.

The first collaborative modeling module 73 may be specifically configured to perform data communication with the task server based on the communication key, the federal homomorphic computation cryptographic key, or the security computation fragment, so as to be identical to a mathematical definition of an input/output result of a preliminary heterogeneous model constructed by the task server during the collaborative modeling operation.

Optionally, the first collaborative modeling module 73 may be further specifically configured to receive an inner product of the preliminary heterogeneous model sent by the task service machine based on a preset transmission direction, or send the inner product to the task service machine, where the inner product is obtained from a current iteration model weight and a sample feature value of the preliminary heterogeneous model; and substituting the inner product into a preset function to obtain the loss value and sending the loss value to the task service machine.

Based on the same inventive concept, an embodiment of the present application further provides a heterogeneous model building apparatus 80 applied to a task service machine, please refer to fig. 8, where fig. 8 is a schematic diagram of the heterogeneous model building apparatus of the task service machine provided in the embodiment of the present application, and the heterogeneous model building apparatus 80 may include:

and the second obtaining module 81 is configured to obtain information representing roles of other participating modeling nodes, where the roles include a task request machine and a task service machine, and the roles are determined based on a modeling scenario and holding data of the modeling nodes.

And the receiving module 82 is used for receiving the algorithm information sent by the task request machine and determining a target modeling algorithm.

A second collaborative modeling module 83, configured to perform collaborative modeling operations with the task requester based on the target modeling algorithm, where the collaborative modeling operations include: the method includes calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an update gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the update gradient value.

And a second judging module 84, configured to determine whether the preliminary heterogeneous model converges according to the weight value and the loss value, and obtain a target heterogeneous model when it is determined that the preliminary heterogeneous model converges.

Optionally, the heterogeneous model building apparatus 80 may further include a second communication module, configured to determine a communication key, a federal homomorphic calculation cryptographic key, or a security calculation fragment based on the algorithm information sent from the task requester.

The second collaborative modeling module 83 may be specifically configured to perform data communication with the task requester based on the communication key, the federal homomorphic computation cryptographic key, or the security computation fragment, so as to be the same as a mathematical definition of an input/output result of a preliminary heterogeneous model constructed by the task requester during the collaborative modeling operation.

Optionally, the second collaborative modeling module 83 may be further specifically configured to receive an inner product of the preliminary heterogeneous model sent by the task requesting machine based on a preset transmission direction, or send the inner product to the task requesting machine, where the inner product is obtained from a current iteration model weight and a sample feature value of the preliminary heterogeneous model; and receiving the loss value sent by the task request machine, wherein the loss value is obtained by substituting the inner product into a preset function by the request service machine.

And determining the updated gradient value according to the loss value and the sample characteristic value; and determining the updated weight value of the preliminary heterogeneous model based on the updated gradient value and the learning rate of the preliminary heterogeneous model.

Based on the same inventive concept, an embodiment of the present application further provides an electronic device, where the electronic device includes a memory and a processor, where the memory stores program instructions, and the processor executes the steps in any one of the above implementation manners when reading and executing the program instructions.

Based on the same inventive concept, embodiments of the present application further provide a computer-readable storage medium, where computer program instructions are stored, and when the computer program instructions are read and executed by a processor, the computer program instructions perform steps in any of the above-mentioned implementation manners.

The computer-readable storage medium may be a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and other various media capable of storing program codes. The method executed by the electronic terminal defined by the process disclosed by any embodiment of the invention can be applied to the processor or realized by the processor.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.

In addition, units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

Furthermore, the functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

Alternatively, all or part may be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part.

The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.).

In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A heterogeneous model building method is applied to a task request machine and comprises the following steps:

performing collaborative modeling operation with the task server based on the target modeling algorithm; wherein the collaborative modeling operation comprises: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an update gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the update gradient value; and

2. The method of claim 1, wherein prior to the co-modeling operation with the task server based on the target modeling algorithm, the method further comprises:

the cooperative modeling operation based on the target modeling algorithm and the task server comprises the following steps:

3. The method of claim 1, wherein the calculating a loss value for a preliminary heterogeneous model based on the target modeling algorithm comprises:

receiving the inner product of the preliminary heterogeneous model sent by the task server based on a preset transmission direction, or sending the inner product to the task server; the inner product is obtained by the current iteration model weight and the sample characteristic value of the preliminary heterogeneous model;

4. The method of claim 3, wherein updating the weight values of the preliminary heterogeneous model based on the updated gradient values comprises:

determining the updated gradient value according to the loss value and the sample characteristic value; and

5. A heterogeneous model building method is applied to a task service machine and comprises the following steps:

acquiring information representing roles of other nodes participating in modeling, wherein the roles comprise a task request machine and a task service machine; wherein the role is determined based on a modeled scenario and holding data of the modeled node;

performing collaborative modeling operation with the task request machine based on the target modeling algorithm; wherein the collaborative modeling operation comprises: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an update gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the update gradient value;

6. The method of claim 5, wherein prior to the co-modeling operation with the task server based on the target modeling algorithm, the method comprises:

the cooperative modeling operation based on the target modeling algorithm and the task request machine comprises the following steps:

7. The method of claim 5, wherein the calculating a loss value for a preliminary heterogeneous model based on the target modeling algorithm comprises:

receiving an inner product of the preliminary heterogeneous model sent by the task request machine based on a preset transmission direction, or sending the inner product to the task request machine; the inner product is obtained by the current iteration model weight and the sample characteristic value of the preliminary heterogeneous model; and

8. The method of claim 7, wherein updating the weight values of the preliminary heterogeneous model based on the updated gradient values comprises:

and determining the updated weight value of the preliminary heterogeneous model based on the updated gradient value and the learning rate of the preliminary heterogeneous model.

9. A heterogeneous model building device is applied to a task request machine and comprises the following components:

the first acquisition module is used for acquiring information representing roles of other nodes participating in modeling, wherein the roles comprise a task request machine and a task service machine; wherein the role is determined based on a modeled scenario and holding data of the modeled node;

the first collaborative modeling module is used for carrying out collaborative modeling operation with the task server based on the target modeling algorithm; wherein the collaborative modeling operation comprises: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an update gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the update gradient value;

10. A heterogeneous model building device is applied to a task service machine and comprises:

the second acquisition module is used for acquiring information representing roles of other nodes participating in modeling, wherein the roles comprise a task request machine and a task service machine; wherein the role is determined based on a modeled scenario and holding data of the modeled node;

the second collaborative modeling module is used for carrying out collaborative modeling operation with the task request machine based on the target modeling algorithm; wherein the collaborative modeling operation comprises: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an update gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the update gradient value;