CN118674075A

CN118674075A - Method, device, equipment and medium for generating machine learning model

Info

Publication number: CN118674075A
Application number: CN202411156397.3A
Authority: CN
Inventors: 黄骅; 吴立辉; 孙勇; 钟雪君
Original assignee: China Electronics Technology Eastern Communication Group Co ltd
Current assignee: China Electronics Technology Eastern Communication Group Co ltd
Priority date: 2024-07-30
Filing date: 2024-08-22
Publication date: 2024-09-20

Abstract

The invention relates to the technical field of machine learning, and discloses a method, a device, equipment and a medium for generating a machine learning model, wherein the method comprises the following steps: acquiring first application scene description information corresponding to a machine learning model expected to be constructed by a user; matching a target application scene corresponding to the first application scene description information in a knowledge base, wherein the knowledge base comprises: when developing a machine learning model in different application scenes, corresponding development workflow, data set information and algorithm information of at least one available algorithm; generating a first machine learning model corresponding to each available algorithm based on the development workflow corresponding to the target application scene, the data set information and the algorithm information of at least one available algorithm, according to the invention, the development workflow and the specific information corresponding to the application scene description information input by the user are matched in the knowledge base, so that the machine learning model is generated, and the problems of high operation threshold and low training efficiency when the user develops the model can be reduced.

Description

Method, device, equipment and medium for generating machine learning model

Technical Field

The present invention relates to the field of machine learning technologies, and in particular, to a method, an apparatus, a device, and a medium for generating a machine learning model.

Background

At present, machine learning is widely applied in various fields, and various machine learning models and algorithms remarkably improve the production efficiency in various fields. For the business in the bank marketing field, the machine learning model can also obviously improve the marketing efficiency of the bank. The bank can generate a machine learning model meeting the requirements through some existing machine learning platforms so as to meet the actual business needs.

However, when a machine learning model is built through the current various machine learning platforms, due to the relatively complex construction principle of the model, service personnel often have difficulty in understanding the intrinsic operation mechanism of the model and the basis of output results, so that the service personnel have the problems of high operation threshold and low training efficiency when generating the machine learning model.

Disclosure of Invention

In view of the above, the present invention provides a method, apparatus, device and medium for generating a machine learning model, so as to solve the problems of high operation threshold and low training efficiency when the machine learning platform generates the machine learning model.

In a first aspect, the present invention provides a method for generating a machine learning model, which is applied to a machine learning platform, where the machine learning platform includes: a pre-built knowledge base, the method comprising:

Acquiring first application scene description information corresponding to a machine learning model expected to be constructed by a user;

Matching a target application scene corresponding to the first application scene description information in the knowledge base, wherein the knowledge base comprises: when developing a machine learning model in different application scenes, corresponding development workflow, data set information and algorithm information of at least one available algorithm;

And generating a first machine learning model corresponding to each available algorithm based on the development workflow corresponding to the target application scene, the data set information and the algorithm information of at least one available algorithm.

According to the method, the machine learning model is generated by acquiring the description information of the application scene corresponding to the machine learning model expected to be built by the user and then according to the development workflow and the data set information which are matched with the algorithm information in the knowledge base, and the user only needs to describe the application scene of the model expected to be built without knowing the specific principle of model construction, so that the operation threshold when the user generates the machine learning model through the machine learning platform can be greatly reduced, the model is generated according to the development workflow in the knowledge base, and the training effect of the model can be ensured.

In an alternative embodiment, after constructing the first machine learning model corresponding to each available algorithm, the method further comprises:

acquiring first test data, and testing each first machine learning model through the first test data;

and generating a corresponding evaluation report according to the test result corresponding to each first machine learning model, so that a user determines a target learning model in at least one first machine learning model according to the evaluation report.

According to the method, the machine learning models corresponding to the algorithms are tested and evaluated, so that an evaluation report is generated, the characteristics of the machine learning models are described for a user, and the user can conveniently select the machine learning model which meets the actual requirements.

In an alternative embodiment, the knowledge base further includes: descriptive information corresponding to different application scenes;

The matching the target application scene corresponding to the first application scene description information in the knowledge base comprises the following steps:

converting the first application scene description information and the description information corresponding to different application scenes in a knowledge base into a vector format through a preset language model;

And comparing the description information of the first application scene in the vector format with the description information corresponding to different application scenes in the knowledge base to determine a target application scene.

According to the embodiment, the application scene description information of the machine learning model expected to be built by the user and the description information corresponding to different application scenes in the knowledge base are converted into the vector format, so that matching is carried out according to the vectors, a target application scene corresponding to the user requirement can be more accurately matched, and the effectiveness of the subsequently built machine learning model is ensured.

In an optional implementation manner, the generating a first machine learning model corresponding to each available algorithm based on the development workflow corresponding to the target application scenario, the data set information and the algorithm information of at least one available algorithm includes:

Generating development codes of a first machine learning model corresponding to each available algorithm according to development workflow, data set information and algorithm information of at least one available algorithm corresponding to the target application scene;

And constructing initial models corresponding to the available algorithms according to the development codes of the first machine learning models corresponding to the available algorithms, and training and adjusting parameters of the initial models to obtain the first machine learning models corresponding to the available algorithms.

According to the method, the corresponding model development codes are generated through the development workflow in the knowledge base, specific algorithm information and data set information, the corresponding initial model is further constructed according to the development codes, relevant parameters are adjusted, a machine learning model is finally generated, the model is automatically constructed and trained through the machine learning platform, the using threshold of use is reduced, and the using experience of a user is improved.

In an alternative embodiment, the algorithm information in the knowledge base includes: super-parameter knowledge corresponding to the algorithm;

training and parameter adjustment are carried out on the initial model to obtain a first machine learning model corresponding to each available algorithm, and the method comprises the following steps:

Invoking training data and test data of the initial model corresponding to each available algorithm;

training the initial model through the training data, and adjusting model parameters of the initial model;

testing the trained initial models through the test data to obtain evaluation results corresponding to the trained initial models;

Acquiring super-parameter knowledge corresponding to each available algorithm;

And performing super-parameter optimization on the trained initial models based on the super-parameter knowledge corresponding to each available algorithm and the evaluation results corresponding to the trained initial models to obtain first machine learning models corresponding to each available algorithm.

According to the method, model parameters are adjusted through training data, an evaluation result corresponding to the trained model is obtained according to the test data, model super-parameters are optimized by combining specific super-parameter knowledge, a final machine learning model is obtained, and the training effect of the machine learning model can be guaranteed.

In an alternative embodiment, the development workflow includes: developing the corresponding working content of each working node in the workflow;

The generating a development code of a first machine learning model corresponding to each available algorithm according to the development workflow, the data set information and the algorithm information of at least one available algorithm corresponding to the target application scene comprises the following steps:

For each available algorithm, generating development codes corresponding to all working nodes based on algorithm information, data set information and working contents corresponding to all working nodes in a development workflow of the available algorithm;

And splicing the development codes corresponding to the working nodes to obtain the development codes of the first machine learning model corresponding to the available algorithm.

According to the method, the development codes corresponding to the working nodes are generated through the working content, the specific algorithm information and the data set information corresponding to the working nodes in the development workflow, and the codes of the working nodes are spliced, so that the development codes corresponding to the machine learning models are obtained, and the accuracy of the codes can be ensured.

In an alternative embodiment, the generating the corresponding evaluation report according to the test result corresponding to each first machine learning model includes:

Determining test results of each first machine learning model under each preset index, wherein the preset indexes comprise: accuracy, precision, recall;

And generating a corresponding evaluation report according to the test results corresponding to the preset indexes.

According to the method and the device, the corresponding evaluation report is generated by determining the test result under each index, so that the user can further know the characteristics of each machine learning model, and the user can conveniently select the machine learning model according with the actual requirements.

In a second aspect, the present invention provides an apparatus for generating a machine learning model, the apparatus comprising:

The information acquisition module is used for acquiring first application scene description information corresponding to a machine learning model expected to be constructed by a user;

The scene matching module is configured to match a target application scene corresponding to the first application scene description information in the knowledge base, where the knowledge base includes: when developing a machine learning model in different application scenes, corresponding development workflow, data set information and algorithm information of at least one available algorithm;

The model development module is used for generating a first machine learning model corresponding to each available algorithm based on the development workflow corresponding to the target application scene, the data set information and the algorithm information of at least one available algorithm.

In a third aspect, the present invention provides a computer device comprising: the memory and the processor are in communication connection, the memory stores computer instructions, and the processor executes the computer instructions, so that the machine learning model generation method of the first aspect or any corresponding implementation mode of the first aspect is executed.

In a fourth aspect, the present invention provides a computer-readable storage medium having stored thereon computer instructions for causing a computer to execute the method for generating a machine learning model according to the first aspect or any one of the embodiments corresponding thereto.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow diagram of a method of generating a machine learning model according to an embodiment of the present invention;

FIG. 2 is a flow diagram of another method of generating a machine learning model according to an embodiment of the present invention;

FIG. 3 is a flowchart illustration of a process for performing machine learning model construction by a machine learning platform, according to an embodiment of the present invention;

FIG. 4 is an exemplary diagram of a knowledge base construction flow in accordance with an embodiment of the invention;

FIG. 5 is a flowchart illustration of a machine learning platform generating model code according to an embodiment of the present invention;

FIG. 6 is a flowchart illustrating a machine learning platform performing model optimization according to the present embodiment;

FIG. 7 is a diagram of a closed-loop full-flow example of machine learning model generation in accordance with an embodiment of the present invention;

FIG. 8 is a block diagram of a machine learning model generation apparatus according to an embodiment of the present invention;

fig. 9 is a schematic diagram of a hardware structure of a computer device according to an embodiment of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Therefore, the embodiment of the invention provides a method for generating a machine learning model, which is used for generating the machine learning model by acquiring the description information of the application scene corresponding to the machine learning model expected to be constructed by a user and then matching corresponding development workflow, data set information and algorithm information in a knowledge base.

According to an embodiment of the present invention, there is provided a method embodiment of generating a machine learning model, it being noted that the steps shown in the flowcharts of the drawings may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that herein.

In this embodiment, a method for generating a machine learning model is provided, which is applied to a machine learning platform, where the machine learning platform includes: fig. 1 is a flowchart of a method for generating a machine learning model according to an embodiment of the present invention, as shown in fig. 1, the flowchart includes the steps of:

Step S101, first application scene description information corresponding to a machine learning model expected to be constructed by a user is obtained.

When a user uses a machine learning platform to construct a machine learning model meeting business requirements, a specific application scene of the machine learning model which the user wants to construct and a specific function description under the scene can be input, namely, what scene the machine learning model which the user wants to construct is applied to, and what function can be realized under the scene, and the information is corresponding application scene description information.

For example, the user may input: i want to build a machine learning model that predicts the churn of banking customers, as well as customer churn due to those reasons. Similar content can be understood as application scenario description information corresponding to the machine learning model that the user desires to build.

Step S102, matching a target application scene corresponding to the first application scene description information in a knowledge base, wherein the knowledge base comprises: when the machine learning model is developed under different application scenes, corresponding development workflow, data set information and algorithm information of at least one available algorithm.

In the machine learning platform, a knowledge base of the service field corresponding to the user is pre-combed, and the knowledge base comprises: the method is used for describing information corresponding to various application scenes which may need to construct a machine learning model in the corresponding service field, and corresponding development workflow, data set information and algorithm information when the corresponding machine learning model is developed in each application scene.

For example, the development workflow can include a plurality of specific working nodes and corresponding working contents of the working nodes; the data set information comprises description information of a data set used for developing a corresponding machine learning model, such as specific data description information of meanings, label definitions and the like corresponding to various feature columns; meanwhile, for each application scene, when a model meeting the requirement of the application scene is developed, a plurality of related algorithms may exist, so that algorithm information of at least one algorithm corresponding to the application scene is recorded in a knowledge base, and the algorithm information may include: the advantages and disadvantages of the algorithms, applicable data requirements, overparameter ranges and other information, and each algorithm can be used for constructing a corresponding machine learning model conforming to an application scene.

And matching the application scene closest to the first application scene description information in the knowledge base, and taking the application scene as a target application scene.

Step S103, based on the development workflow corresponding to the target application scene, the data set information and the algorithm information of at least one available algorithm, generating a first machine learning model corresponding to each available algorithm.

After determining the development workflow, the data set information and the algorithm information corresponding to the target application scene, according to the specific content of each working node in the development workflow and the specific description in the data set information and the algorithm information, the information can be used as a thinking chain of model development, namely COT, and the machine learning platform can execute specific operations during model development according to the specific description information in the thinking chain, for example, generate model codes, call related data to perform specific operations such as training and parameter optimization, so as to obtain machine learning models corresponding to each algorithm, and the machine learning models corresponding to each algorithm can be applied to a first application scene corresponding to the machine learning model which is expected to be constructed by a user.

According to the method for generating the machine learning model, the corresponding model development codes are generated through the development workflow in the knowledge base, the specific algorithm information and the data set information, the corresponding initial model is obtained through construction according to the development codes, relevant parameters are adjusted, the machine learning model is finally generated, the construction and training of the model are automatically carried out through the machine learning platform, the using threshold of the user is reduced, and the using experience of the user is improved.

According to an embodiment of the present invention, another embodiment of a method for generating a machine learning model is provided, and the method is applied to a machine learning platform, where the machine learning platform includes: FIG. 2 is a flowchart of another method of generating a machine learning model according to an embodiment of the present invention, as shown in FIG. 2, the flowchart including the steps of:

Step S201, acquiring first application scenario description information corresponding to a machine learning model that a user desires to construct. The specific implementation manner refers to step S101 in the embodiment shown in fig. 1, which is not described herein.

Step S202, matching a target application scene corresponding to the first application scene description information in a knowledge base, wherein the knowledge base comprises: when the machine learning model is developed under different application scenes, corresponding development workflow, data set information and algorithm information of at least one available algorithm.

Specifically, the knowledge base further comprises: descriptive information corresponding to different application scenes. The description information may be understood as information about specific descriptions of model application scenarios when constructing models of different application scenarios, for example, what functions the model needs to implement, what situations are used, what types of problems the application scenarios belong to, what targets of the model are, and so on. Taking a model application scenario of bank client streaming prediction in the field of bank marketing as an example, the information format of the corresponding scenario description information can be as follows:

model_scale, "bank customer churn prediction";

The model_scale_description is that bank customer churn prediction is a typical machine learning classification problem, and is mainly applied to the financial service industry.

In this scenario, banks act as financial institutions with the goal of preserving and improving customer satisfaction as much as possible to keep business stable and growing.

The main task of the customer churn prediction model is to identify those customers who are likely to stop using banking services for a future period of time (churn customers) and those who are likely to continue to become loyalty customers (non-churn customers).

Through the scene description information, the machine learning platform can know the specific information corresponding to each model application scene so as to perform corresponding matching, and meanwhile, when the machine learning model corresponding to the scene is generated later, the specific requirement of the model can be determined according to the specific scene description information so as to facilitate the machine learning platform to construct the model.

Specifically, in step S202, matching, in the knowledge base, a target application scenario corresponding to the first application scenario description information includes:

Step S202-1, converting the description information of the first application scene and the description information corresponding to different application scenes in the knowledge base into a vector format through a preset language model.

For the description information of the first application scene and the description information corresponding to different application scenes in the knowledge base, which belong to text information, the description information corresponding to the application scene can be converted into a vector format through some trained language models, such as BERT-base or RoBERTa. Specifically, when the knowledge base is constructed in advance, the description information of the application scene in the knowledge base can be directly saved through a vector format, so that when the description information is matched, only the description information corresponding to the first application scene is required to be subjected to vector format conversion, and time consumption is reduced.

Step S202-2, comparing the description information of the first application scene in the vector format with the description information corresponding to different application scenes in the knowledge base, and determining the target application scene.

After the application scene description information in the vector format is obtained, a target application scene corresponding to the first application scene description information can be quickly queried by using a vector distance calculation method, so that data set information, algorithm information and development workflow corresponding to the target application scene are determined in a knowledge base, and a machine learning platform can generate a corresponding machine learning model according to specific description in the information in the knowledge base through a thinking chain technology.

Step S203, generating a first machine learning model corresponding to each available algorithm based on the development workflow corresponding to the target application scenario, the data set information and the algorithm information of at least one available algorithm.

Specifically, in step S203, based on the development workflow corresponding to the target application scenario, the data set information, and algorithm information of at least one available algorithm, a first machine learning model corresponding to each available algorithm is generated, including:

Step S203-1, generating development codes of a first machine learning model corresponding to each available algorithm according to the development workflow, the data set information and the algorithm information of at least one available algorithm corresponding to the target application scene.

It can be understood that the machine learning platform has code generation capability, after determining the development workflow corresponding to the target application scenario, the data set information and the algorithm information corresponding to each available algorithm, the machine learning platform can form a thinking chain of the development process according to the description of the specific work content corresponding to each link in the development workflow and the specific description in the specific data set information and the algorithm information, and automatically generate the development code of the machine learning model corresponding to each algorithm according to the thinking chain.

Specifically, the development workflow includes: and developing the corresponding working contents of each working node in the workflow.

It will be appreciated that a development workflow is a workflow that simulates the development of a machine learning model by a real data scientist and can be understood as a complete set of data analysis and algorithm development step guidelines. The method comprises the description of specific work content corresponding to each step when the machine learning model is developed. For example, the development workflow may include content descriptions corresponding to a plurality of development work links such as data preprocessing, feature engineering, model selection, parameter tuning, model evaluation, and result interpretation.

For example, in an application scenario of predicting customer churn in a bank marketing service, when constructing a corresponding machine learning model, the information format of the working content of each working node in the corresponding development workflow may be as follows:

"model_workflow": [

{

"workflow _status": demand analysis ",

"Workflow _content": "-determine business objective: understanding what the bank wants to achieve through the predictive model, e.g., improving customer retention, reducing customer acquisition costs, etc. N-definition problem: the explicit question is to predict whether the customer will be churn or to determine which factors will lead to churn. N-data collection: knowing the characteristics and limitations of existing datasets, additional data may be needed. "

},

{

"Workflow _status": feature engineering ",

"Workflow _content": "-data cleansing: and the missing value processing and the abnormal value processing ensure the data quality. N-data conversion: classification variables are encoded (e.g., one-Hot encoding), normalized or normalized to numerical characteristics. N-feature engineering: based on business knowledge and model requirements, new features are created, such as interactive items, time series features, etc. "

},

{

"Workflow _status": model training ",

"Workflow _content" -an appropriate algorithm, such as logistic regression, random forest, decision tree, or XGBoost, is selected based on the nature of the problem (e.g., data size, number of features, non-linear relationship, etc.). N-dividing the data set: the data set is divided into a training set, a validation set and a test set. N-model training: the model is trained using a training set and parameters are adjusted to optimize performance. "

},

{

"Workflow _status": model evaluation ",

"Workflow _content" -model performance is assessed using a validation set, such as accuracy, precision, recall, F1 score, AUC-ROC curve, and the like. N-selects the appropriate evaluation index because for unbalanced data, the accuracy may not be the optimal index. "

},

{

"Workflow _status": "super-parametric optimization",

"Workflow _content" -find the optimal model parameter combination by a grid search method. "

},

]

Specifically, in step S203-1, according to the development workflow corresponding to the target application scenario, the data set information, and the algorithm information of at least one available algorithm, a development code of a first machine learning model corresponding to each available algorithm is generated, including:

Step S203-1-1, for each available algorithm, generating development codes corresponding to the working nodes based on algorithm information, data set information and working contents corresponding to the working nodes in the development workflow of the available algorithm.

And step S203-1-2, splicing the development codes corresponding to the working nodes to obtain the development codes of the first machine learning model corresponding to the available algorithm.

For step S203-1-1, it may be understood that, when performing the generation of the development code of the machine learning model corresponding to each algorithm, the machine learning platform may need to generate the development code required by the corresponding work content of each work node according to the specific work content of each work node in the development workflow, the specific description of the data set in the data set information under the target application scenario in the knowledge base, the specific description of the algorithm in the algorithm information of each algorithm, and the related information about the data set and the algorithm in the knowledge base.

Illustratively, the data set information may include: the detailed description of basic information of the data set in the data set for training the model, and information such as meaning, label definition, association analysis result between the characteristics and the labels corresponding to each characteristic column in the data set, also take a bank client stream prediction scene as an example, and the specific information format of the corresponding data set information can be as follows:

The database description predicts whether the customer will run off based on the bank customer run-off prediction dataset to assist the financial institution in developing the saving strategy. N I have bank customer churn forecast data sets, including the following three dimensions: n (1) personal information (characteristic list: age, sex, geographical location) \n (2) financial information (characteristic list: customer year, balance, salary estimate) \n (3) historical purchase record (characteristic list: number of purchased products, whether there is credit card, customer liveness) ".

"datase_column": [

{

"feature_name": "CreditScore",

"Feature_description" record number, corresponding to record (line) number, has no effect on output "

},

……

{

"feature_name": "EstimatedSalary",

"Feature_description" payroll estimates that people with lower payroll are more likely to leave the bank than people with higher payroll. "

},

{

"label_name": "Exited",

Whether the label_description is lost, whether the customer leaves the bank, and whether the label column is displayed. "

}

]

By way of example, the specific information format of the algorithm information may be as follows:

Whether a bank client runs out prediction is a typical machine learning classification task. ",

"algorithms":[

{

"Algorithm_name": "K nearest neighbor algorithm (K-Nearest Neighbors, KNN)",

"Algorithm_advantages" -theory is simple, without requiring a training process. ",

"Algorithm_ DISADVANTAGES" -calculated in the prediction is large, sensitive to noise data, and less effective. ",

"Algorithm_scale" -the algorithm is simple, suitable for small data sets, has high computational complexity, and has higher requirements on data preprocessing. ",

"Algorithm_hyper_opt" -n_neighbors: neighbor number, integer, range [3, 20] "

},

……

]

The foregoing describes only information corresponding to one algorithm by way of example, and in the actual knowledge base, for each application scenario, there may be a plurality of subordinate available algorithms, and for each algorithm, the algorithm information corresponding to each application scenario may be obtained by combing the algorithm information according to the format described above.

When the machine learning platform develops the corresponding codes according to the working content corresponding to each working node in the development workflow, the machine learning platform can form a thinking chain of the machine learning platform model development process by combining the algorithm information, the data set information and the specific description information of the application scene in the knowledge base, so as to generate the development codes corresponding to each working node for subsequent model generation.

For step S203-1-2, it may be understood that, for the machine learning model corresponding to each algorithm, after generating the development codes of each corresponding working link, the machine learning platform may splice the codes to obtain the development codes of the machine learning model corresponding to each algorithm.

Step S203-2, constructing an initial model corresponding to each available algorithm according to the development codes of the first machine learning model corresponding to each available algorithm, and training and parameter adjustment are performed on the initial model to obtain the first machine learning model corresponding to each available algorithm.

After the development codes of the first machine learning models are obtained, initial models corresponding to the available algorithms can be obtained according to the development codes, codes corresponding to specific processes such as full-link data calling, training and super-parameter optimization are arranged in the development codes corresponding to the initial models, the initial models corresponding to the algorithms are respectively trained and parameters are respectively adjusted through the codes, and finally the first machine learning models corresponding to the algorithms can be applied to the first application scene.

Specifically, the algorithm information in the knowledge base includes: and (5) super-parameter knowledge corresponding to the algorithm. For example, the type of the super-parameters when each algorithm constructs the machine learning model, and the adjustment range corresponding to the super-parameters of each type.

Specifically, in step S203-2, training and parameter tuning are performed on the initial model to obtain a first machine learning model corresponding to each available algorithm, including:

Step S203-2-1, the training data and the test data of the initial model corresponding to each available algorithm are called.

Because of the difference between the algorithms corresponding to the different initial models, there may be a certain difference between the corresponding training data, and the original data of these training data may be the same. The specific description information of the algorithm relates to the format requirement of training data when training the model corresponding to the algorithm.

Therefore, the machine learning platform carries out corresponding preprocessing on the original data according to the specific description information corresponding to each algorithm, so that the training data corresponding to different initial models may have differences. For the test data, the format of the test data is the same as that of the training data, and the content is different so as to check the specific effect of the trained machine learning model.

And step S203-2-2, training the initial model through training data, and adjusting model parameters of the initial model.

The machine learning platform can perform corresponding training according to specific development codes corresponding to the model, and adjust relevant parameters of the model in the training process.

And step S203-2-3, testing the trained initial models through test data to obtain evaluation results corresponding to the trained initial models.

After a certain round of training, the trained initial models can be tested to obtain corresponding evaluation results of the trained initial models, and specific training conditions of the models can be known according to the evaluation results.

And step S203-2-4, obtaining the super-parameter knowledge corresponding to each available algorithm.

The algorithm information in the knowledge base comprises corresponding super-parameter knowledge, and the machine learning platform can call the corresponding super-parameter knowledge to carry out subsequent super-parameter adjustment.

Step S203-2-5, performing super-parametric optimization on each trained initial model based on super-parametric knowledge corresponding to each available algorithm and evaluation results corresponding to each trained initial model, and obtaining a first machine learning model corresponding to each available algorithm.

After the evaluation results and the super-parameter knowledge corresponding to each initial model are determined, the machine learning platform can combine the specific content of the super-parameter optimization part in the development workflow to generate corresponding super-parameter optimization codes so as to automatically execute operations required by super-parameter optimization, obtain optimal super-parameter combinations corresponding to each trained initial model, combine the optimal super-parameter combinations into the trained initial models, and perform corresponding iterative training to obtain a first machine learning model corresponding to each final available algorithm.

Step S204, first test data are obtained, and each first machine learning model is tested through the first test data;

It may be understood that after the first machine learning models corresponding to the available algorithms are obtained, since the machine learning models corresponding to the different algorithms are necessarily different from each other, the different models may be tested by using test data, so as to generate an evaluation report, where the evaluation report may include information for evaluating characteristics of the models, such as advantages and disadvantages of each first machine learning model, and performances under different indexes. The user can select the first machine learning model which is most suitable for the actual application scene according to the evaluation of the advantages and disadvantages of each first machine learning model in the evaluation report.

Specifically, in step S204, a corresponding evaluation report is generated according to the test results corresponding to each first machine learning model, including:

It can be understood that when each first machine learning model is evaluated, the performance of the first machine learning model under three indexes of accuracy, precision and recall can be mainly considered, so that an evaluation report corresponding to each model can be generated according to the corresponding test performance under each index. Specifically, different weights may be set for different indexes, so as to provide the most recommended first machine learning model according to the specific performance of each index.

According to the method for generating the machine learning model, the description information of the application scene corresponding to the machine learning model expected to be constructed by the user is obtained and converted into the vector format, so that the matching effect is guaranteed, the corresponding development workflow and data set information and at least one algorithm information are matched in the knowledge base to generate at least one machine learning model, meanwhile, the evaluation report corresponding to each machine learning model is generated, the user can conveniently select the machine learning model which is most in line with the actual requirement, and because the user only needs to describe the application scene of the model expected to be constructed, the specific principle of model construction is not needed to be known, the operation threshold when the user generates the machine learning model through the machine learning platform is greatly reduced, meanwhile, the model is generated according to the development workflow in the knowledge base, and the training effect of the model is also guaranteed.

To assist in understanding the details of the above embodiments, an exemplary flowchart of a machine learning model construction performed by a machine learning platform according to an embodiment of the present invention is shown in fig. 3.

For step S1 in the method flow, an expert knowledge base of data set recommendation, algorithm model selection and development workflow is constructed. Reference may be made to fig. 4, which is an exemplary diagram of a knowledge base construction flow in accordance with an embodiment of the invention.

The built knowledge base covers rich financial field data sets, and various machine learning classical algorithms are recorded for the machine learning platform to conduct expert data set recommendation and algorithm model selection work.

Firstly, model scene description information possibly designed in a certain business field, corresponding data set information in each scene, machine learning algorithm knowledge and development workflow when a data scientist develops a machine learning model conforming to the corresponding scene are collected.

After the specific information is obtained, the specific information is subjected to corresponding carding, and a specific carding mode can refer to the example information format corresponding to the application scene description information, the data set information, the algorithm information and the development workflow in the bank client stream prediction scene in the bank marketing field, which are given in the embodiment shown in fig. 2, and will not be described here.

After the information is carded, description information corresponding to each application scene, data set information, algorithm information and development workflow under each scene are obtained, the content is stored in a knowledge base as a knowledge document and used as expert experience, meanwhile, the application scene description information of a model is vectorized to generate a model scene vector library, each model scene vector in the model scene vector library is correspondingly associated with the expert experience under the scene, namely the data set information, the algorithm information and the development workflow, so that when the application scene description information is matched, the corresponding data set information, the algorithm information and the development workflow can be obtained in the knowledge base.

Expert knowledge documents including scene model descriptions, dataset recommendations, algorithm selections, data scientist workflows are formed by structured mining of expert knowledge of datasets and algorithms, and then vector conversion is performed on the scene model description portion by means of advanced pre-trained language models (e.g., BERT-base or RoBERTa).

The user can quickly inquire the content of the model scene knowledge document by inputting a problem or a keyword and a scene model description part by using a vector distance calculation method, so that expert knowledge most relevant to the requirement is obtained. After retrieving the relevant information, the machine learning platform provides a deep interpretation and customized financial scene model solution in combination with the large model's mental chain (Chain of Thought, coT) technology.

The step S1 may be understood as a preparation work of the machine learning platform, after receiving a model building requirement of a user, the step corresponding to the step S2 is started to be executed, and in the step S2, when a knowledge base is searched based on the user requirement, an expert thinking chain is obtained, and tools are scheduled, executed and fed back.

Referring specifically to fig. 5, a flowchart illustrating a process of generating model codes by a machine learning platform according to an embodiment of the present invention matches a closest application scenario in a knowledge base according to a specific description of a model application scenario in demand information after receiving demand information of a user, and invokes development workflow, data set information, and algorithm information in the scenario.

Because the development workflow, the data set information and the algorithm information all contain specific description information, the development workflow can be combined with the data set information and the algorithm information to be used as an expert workflow thinking chain, namely COT.

Taking a bank customer stream prediction scene as an example, the machine learning platform performs specific scheduling, execution and feedback according to a thinking chain, and generates codes corresponding to a data set acquisition API, codes corresponding to a data preprocessing API, algorithm codes and evaluation codes.

The code corresponding to the data set acquisition API can be understood as a data source reading module for outputting a data set reading code block for a subsequent algorithm code by the machine learning platform through the large model information extraction capability, wherein the acquisition request parameters comprise information such as a data set keyword, a feature column name list, a tag column name and the like.

For generating codes corresponding to the data preprocessing API, it can be understood that the machine learning platform performs data preprocessing API call by the requirement of a feature engineering part in a development workflow, extracts feature engineering keywords comprising missing value processing, outlier processing, one-Hot encoding, standardization, normalization and the like, outputs feature engineering code blocks and is used for a feature engineering module of a subsequent algorithm code.

For generating the algorithm code, it can be understood that the machine learning platform uses the code generating capability of the large model according to the corresponding algorithm information in the knowledge base and the requirement of model training in development work, including machine learning algorithm functions and algorithm initial super-parameters.

For generating the evaluation code, it can be understood that the machine learning platform generates the model evaluation code according to the corresponding algorithm information in the knowledge base, the requirement of model evaluation in the development workflow, and the general knowledge of machine learning, such as a common evaluation index of machine learning classification task of accuracy, precision, recall, F score and the like, and a common evaluation index of machine learning regression task of MSE, MAE, R-squared.

The codes are spliced and fused to generate training codes, and the integrity and the performability of the training codes are ensured through the code verification capability of the large model, including grammar checking, code complement, function verification and the like.

After the training codes corresponding to the models are obtained, executing a step S3, model training, analyzing training evaluation indexes by the large model, and triggering super-parameter optimization. Detailed procedure referring to fig. 6, an exemplary diagram of a process for performing model optimization by a machine learning platform according to the present embodiment is shown.

As shown in the figure, the concurrent training of the models is performed through the training of the model training codes corresponding to the models, and after training, the models are evaluated, so that the evaluation results corresponding to the models are obtained. It should be noted that the number of models corresponds to the number of algorithms in the corresponding application scenario in the knowledge base. The machine learning platform invokes the machine learning super-parameter optimization knowledge corresponding to the target scene in the expert knowledge base, so that corresponding super-parameter optimization prompt words are generated according to specific training evaluation results of each model, scheduling and feedback related to super-parameter optimization are executed based on the above-mentioned thinking chain, super-parameter optimization is performed by generating codes related to super-parameter optimization, and the super-parameter optimized machine learning model is evaluated to obtain an evaluation report for a user to select an optimal machine learning model.

Specifically, when performing the super-parameter optimization, the optimizing interface of the super-parameter grid search of the machine learning algorithm supported by various platforms can be prefabricated, according to the preset machine learning algorithm and the parameters and the parameter type given by the algorithm type selecting and super-parameter optimizing part in the step S1, the parameter grid searching range is scheduled by the large model intelligent agent in the machine learning platform, the input field information extraction is performed, the super-parameter optimization of the algorithm is realized, and the super-parameter combination conclusion is output.

Finally, in step S4, the large model generates a model training report. Aiming at the whole scene model development process, according to the requirements of data scientist workflows in the step S1, the method comprises the steps of demand analysis, feature engineering, model training, model evaluation, super-parameter optimization and other nodes, and the execution conditions of all the nodes in the step S2 and the step S3, training report generation prompt words are summarized and formed, and the analysis summarizing capability of a large model is used to generate an overall training workflow report.

The invention aims to provide an enhanced closed-loop automatic learning system integrating a thinking chain and a knowledge base, a user inputs a financial scene machine learning model training requirement by using natural language, a CoT is obtained from an expert knowledge base by using a RAG retrieval technology, model training, model evaluation and super-parameter optimization are realized by using the planning capability of a large model, and a model training report is finally output to the user.

For the exemplary embodiment shown in fig. 3, the overall flow may be shown in fig. 7, which is a closed-loop full-flow exemplary diagram generated by a machine learning model according to an embodiment of the present invention.

As shown in the figure, a user inputs a model requirement, namely a specific application scene of the model, and the machine learning platform matches corresponding development workflow, data set information and algorithm information in an expert knowledge base through RAG retrieval. And taking the matched development workflow, data set information and algorithm information as a thinking chain COT in the model development process, carrying out model training and super-parameter optimization according to the thinking chain, finally training and evaluating the machine learning model, generating a final model training report and sending the final model training report to a user so as to enable the user to determine the optimal machine learning model.

According to the embodiment of the invention, the knowledge base comprising the data set, the algorithm knowledge and the workflow thinking chain of the data scientist is established in the machine learning platform, so that the automatic generation and optimization of the intelligent agent are realized. The method not only improves the development efficiency, but also ensures the accuracy and the practicability of the model, and fully utilizes the expert experience of a data scientist. The workflow thinking chain CoT (Continual On-TASK LEARNING) of the data scientist and the knowledge base retrieval enhancement technology RAG (RETRIEVAL-Augmented Generation) are effectively fused. The CoT allows the large model to continue decision making, execution, feedback in a specific task, while the RAG is adept at information retrieval. This combination allows the system to improve answer quality and personalization through deep understanding and expert recommendation while locating information quickly. And when the system is used for model development, a closed-loop automatic flow is adopted for development, and a real-time model optimization strategy and evaluation feedback are provided in the development process. The real-time feedback mechanism enables the system to continuously optimize performance, and improves flexibility and accuracy of the whole information processing process.

The invention can be oriented to the intelligent marketing field of banks, and provides marketing decision support tools for banks by combining large language model intelligent agent technology, thereby overcoming the limitation of the current mainstream closed-loop automatic learning system. The current closed-loop automatic learning system mainly focuses on low-code application, has higher requirement on basic knowledge of user machine learning, and therefore cannot realize a low-technology access threshold in a real sense. By introducing an automatic machine learning technology and a large-model intelligent agent technology, the aim is to reduce the admission barrier, eliminate the strict limitation on the professional knowledge of users, enable the users to seamlessly accept wider user groups, realize the financial application of artificial intelligent energized production through man-machine interaction no matter how the artificial intelligent technology background of the users is.

The present embodiment also provides a device for generating a machine learning model, which is used to implement the foregoing embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

The present embodiment provides a generating device of a machine learning model, as shown in fig. 8, including:

The information obtaining module 401 is configured to obtain first application scenario description information corresponding to a machine learning model that a user desires to construct.

The scene matching module 402 is configured to match a target application scene corresponding to the first application scene description information in a knowledge base, where the knowledge base includes: when the machine learning model is developed under different application scenes, corresponding development workflow, data set information and algorithm information of at least one available algorithm.

The model development module 403 is configured to generate a first machine learning model corresponding to each available algorithm based on the development workflow corresponding to the target application scenario, the data set information, and algorithm information of at least one available algorithm.

In some alternative embodiments, the model development module 403 is further configured to obtain first test data after constructing the first machine learning models corresponding to the available algorithms, and test each first machine learning model through the first test data;

In some alternative embodiments, the knowledge base further includes: descriptive information corresponding to different application scenes;

The scene matching module 402, when matching a target application scene corresponding to the first application scene description information in the knowledge base, includes:

Converting the first application scene description information and the description information corresponding to different application scenes in the knowledge base into a vector format through a preset language model;

In an alternative embodiment, the model development module 403, when generating the first machine learning model corresponding to each available algorithm based on the development workflow corresponding to the target application scenario, the data set information, and the algorithm information of at least one available algorithm, includes:

generating development codes of a first machine learning model corresponding to each available algorithm according to development workflow, data set information and algorithm information of at least one available algorithm corresponding to a target application scene;

And constructing initial models corresponding to the available algorithms according to development codes of the first machine learning models corresponding to the available algorithms, and training and adjusting parameters of the initial models to obtain the first machine learning models corresponding to the available algorithms.

The model development module 403, when training and parameter tuning the initial model to obtain a first machine learning model corresponding to each available algorithm, includes:

Training the initial model through training data, and adjusting model parameters of the initial model;

testing the trained initial models through test data to obtain evaluation results corresponding to the trained initial models;

Acquiring super-parameter knowledge corresponding to each available algorithm;

and performing super-parametric optimization on each trained initial model based on the super-parametric knowledge corresponding to each available algorithm and the evaluation result corresponding to each trained initial model to obtain a first machine learning model corresponding to each available algorithm.

The model development module 403, when generating a development code of the first machine learning model corresponding to each available algorithm according to the development workflow, the data set information and the algorithm information of at least one available algorithm corresponding to the target application scenario, includes:

In an alternative embodiment, the model development module 403, when generating the corresponding evaluation report according to the test result corresponding to each first machine learning model, includes:

Further functional descriptions of the above respective modules and units are the same as those of the above corresponding embodiments, and are not repeated here.

The machine learning model generating device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application SPECIFIC INTEGRATED Circuit) Circuit, a processor and a memory that execute one or more software or firmware programs, and/or other devices that can provide the above functions.

The embodiment of the invention also provides computer equipment, which is provided with the device for generating the machine learning model shown in the figure 8.

Referring to fig. 9, fig. 9 is a schematic structural diagram of a computer device according to an alternative embodiment of the present invention, as shown in fig. 9, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are communicatively coupled to each other using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the computer device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In some alternative embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple computer devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). One processor 10 is illustrated in fig. 9.

The processor 10 may be a central processor, a network processor, or a combination thereof. The processor 10 may further include a hardware chip, among others. The hardware chip may be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general-purpose array logic, or any combination thereof.

Wherein the memory 20 stores instructions executable by the at least one processor 10 to cause the at least one processor 10 to perform a method for implementing the embodiments described above.

The memory 20 may include a storage program area that may store an operating system, at least one application program required for functions, and a storage data area; the storage data area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, memory 20 may optionally include memory located remotely from processor 10, which may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk, or solid state disk; the memory 20 may also comprise a combination of the above types of memories.

The computer device further comprises input means 30 and output means 40. The processor 10, memory 20, input device 30, and output device 40 may be connected by a bus or other means, for example by a bus connection in fig. 9.

The input device 30 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the computer apparatus, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointer stick, one or more mouse buttons, a trackball, a joystick, and the like. The output means 40 may include a display device, auxiliary lighting means (e.g., LEDs), tactile feedback means (e.g., vibration motors), and the like. Such display devices include, but are not limited to, liquid crystal displays, light emitting diodes, displays and plasma displays. In some alternative implementations, the display device may be a touch screen.

The embodiments of the present invention also provide a computer readable storage medium, and the method according to the embodiments of the present invention described above may be implemented in hardware, firmware, or as a computer code which may be recorded on a storage medium, or as original stored in a remote storage medium or a non-transitory machine readable storage medium downloaded through a network and to be stored in a local storage medium, so that the method described herein may be stored on such software process on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, a solid state disk or the like; further, the storage medium may also comprise a combination of memories of the kind described above. It will be appreciated that a computer, processor, microprocessor controller or programmable hardware includes a storage element that can store or receive software or computer code that, when accessed and executed by the computer, processor or hardware, implements the methods illustrated by the above embodiments.

Although embodiments of the present invention have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope of the invention as defined by the appended claims.

Claims

1. A method for generating a machine learning model, which is applied to a machine learning platform, wherein the machine learning platform comprises: a pre-built knowledge base, the method comprising:

2. The method of claim 1, wherein after constructing the first machine learning model corresponding to each available algorithm, the method further comprises:

generating a corresponding evaluation report according to the test result corresponding to each first machine learning model, so that a user determines a target learning model in at least one first machine learning model according to the evaluation report;

the generating a corresponding evaluation report according to the test results corresponding to each first machine learning model includes:

3. The method of claim 1, wherein the knowledge base further comprises: descriptive information corresponding to different application scenes;

4. The method of claim 1, wherein generating the first machine learning model corresponding to each available algorithm based on the development workflow corresponding to the target application scenario, the dataset information, and the algorithm information of the at least one available algorithm comprises:

5. The method of claim 4, wherein the algorithm information in the knowledge base comprises: super-parameter knowledge corresponding to the algorithm;

Acquiring super-parameter knowledge corresponding to each available algorithm;

6. The method of claim 4, wherein the development workflow comprises: developing the corresponding working content of each working node in the workflow;

7. The method according to claim 5, wherein performing the super-parametric optimization on the trained initial models based on the super-parametric knowledge corresponding to the available algorithms and the evaluation results corresponding to the trained initial models to obtain the first machine learning models corresponding to the available algorithms comprises:

the machine learning platform generates corresponding super-parameter optimization codes based on super-parameter knowledge corresponding to each available algorithm and an evaluation result corresponding to each trained initial model;

Obtaining optimal super-parameter combinations corresponding to each trained initial model according to the super-parameter optimization codes;

and performing iterative training based on the optimal super-parameter combination to obtain a first machine learning model corresponding to each available algorithm.

8. A generating device of a machine learning model, applied to a machine learning platform, the machine learning platform comprising: a pre-built knowledge base, said apparatus comprising:

9. A computer device, comprising:

A memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the method of generating a machine learning model according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer instructions for causing a computer to execute the method of generating a machine learning model according to any one of claims 1 to 7.