CN110704135B

CN110704135B - Competition data processing system and method based on virtual environment

Info

Publication number: CN110704135B
Application number: CN201910915336.3A
Authority: CN
Inventors: 杨菲; 李嘉懿; 贺同路; 郭学栋; 徐晓龙; 任永亮
Original assignee: Beijing Intelligent Workshop Technology Co ltd
Current assignee: Beijing Intelligent Workshop Technology Co ltd
Priority date: 2019-09-26
Filing date: 2019-09-26
Publication date: 2020-12-08
Anticipated expiration: 2039-09-26
Also published as: CN110704135A

Abstract

The invention provides a competition data processing system and method based on a virtual environment. By the method, a user only needs to submit codes for realizing a training process and a prediction process to a physical server virtually established by the competition platform, and does not need to download a data set and train and submit a prediction result on own hardware resources, and all the steps of training, predicting, evaluating and the like are finished on a GPU (graphics processing unit) of a virtual environment established by a cloud server appointed by the competition platform, so that the competition data processing process is more standardized. According to the method designed by the invention, the training, predicting, evaluating and other processes in the machine learning competition can be automatically realized on the cloud server, and a user only needs to submit codes required by the training process. The method simplifies the operation of the user and saves time.

Description

Competition data processing system and method based on virtual environment

Technical Field

The invention belongs to the technical field of competition data processing, and particularly relates to a competition data processing system and method based on a virtual environment.

Background

With the continuous development of artificial intelligence theory and computer technology, artificial intelligence technology has been continuously applied to various fields in production and life, and especially machine learning and deep learning technology has made a great breakthrough in the fields of computer vision, natural language processing, voice processing and the like. As people gradually recognize the strong potential of the artificial intelligence technology, at present, many platforms open up artificial intelligence algorithm competitions in multiple fields at home and abroad so as to encourage algorithm engineers and enthusiasts to develop excellent algorithms to solve the problems in the actual scene.

In order to select the advantages and disadvantages of various intelligent algorithms which cannot be executed by the same compound life body or are provided for complex or large-scale tasks, the tasks with practical scenes and application meanings need to be given in advance, and then candidates provide respective algorithms for problem solving and performance testing. Typically, this process may be performed in a race, although a race is not the only way, e.g., performance evaluations may also be performed.

The common processes of the existing competition are as follows: firstly, a competition platform issues a label data set, the data set comprises a training set with known labels and a test set with unknown labels, a user sets up a machine learning model according to the task type and the characteristics of data after downloading the data set, and the training set is used for training the model. After the model is trained, the model is used for predicting the label of the data in the test set, finally, the predicted result is uploaded to a platform, and the platform scores the model according to the indexes such as accuracy, submission time and the like.

The above-mentioned contest code evaluation flow suffers from a number of drawbacks. First, in the prior art, the competitor needs to download the data set from the platform, and the size of the data set may be very large, which may occupy many network and time resources and cause unnecessary troubles to the competitor. Secondly, after the user acquires the data set, the user needs to train the machine learning model by using the hardware resources of the user, and the speed difference of different hardware training models can be very large, so that the unfairness of different competitors in the aspect of submission time is increased. Finally, for the prediction results submitted by the competitors, part of the competitors may artificially modify the results of the model prediction to achieve higher accuracy, which is a cheating behavior inevitably existing in the existing evaluation method and is difficult to find by the competition platform.

In contrast, the invention patent application of publication No. CN109508293A (title: a highly concurrent algorithm competition program online evaluation system, classification No. G06F11/36) proposes an online evaluation system in the field of computer science teaching experiments, in which a web server sends an evaluation task to an evaluation gateway module, and the evaluation task includes a source code submitted by a user and evaluation point configuration information of a question. After the validity of the evaluation task is checked, the evaluation task is added into a task queue to wait for program compilation. The task queue is provided with a priority weight, and information such as arrival time, estimated evaluation time, user priority and the like is comprehensively considered to give priority. The task with high priority is processed by the testing data storage module by the testing machine module, the input and output file is obtained from the testing data server, the content of the input file is provided to the user program through standard input, and after the operation, the user program is output to the standard output and is obtained by the testing machine. The control module monitors and clocks the sandbox container during program run time. If the program can normally run and output within the specified time, the comparison module compares the output result with the standard output answer in the test data storage module to judge whether the program is correct or not. If the program in the sandbox makes an illegal system call or runs overtime, the control module can forcibly stop running. The result generation module sends the result of the comparison module and the information such as the running time, the memory and the like provided by the control module back to the evaluation gateway, the evaluation gateway collects the information and sends the information back to the website server, and the evaluation result is displayed to the user. The system only tests aiming at a theoretical program and does not have any practical problem solving process;

expanding retrieval in a non-patent library to find that some algorithm-class competition evaluation systems exist, the main method is as follows:

downloading a relevant data set to a local client side in a specified competition, finishing data preprocessing work at the local client side, configuring a relevant development environment and installing deep learning frame dependence at the local client side, finishing model training work at the local client side by utilizing GPU (graphic processing unit) computing resources, inputting a corresponding data file according to a model and generating a predicted result file, wherein the general format of the predicted result file is a CSV (file format) file. And uploading a result file generated by the local client to the competition platform, and performing comparative analysis, data processing and the like according to a standard preset by the cloud.

However, the inventor further finds that the above method is only a general introduction, and has no specific technical solution, and the problems of version incompatibility, learning framework installation incompatibility, and the like are often encountered in the local client installation development environment and deep learning framework dependence, and the environment preparation work before the algorithm is realized consumes a long time and energy;

in addition, the local model training is especially directed at the large-scale neural network structure training, and a CPU in an ordinary local client cannot meet the requirement of model calculation. And configuring the GPU hardware devices of the local clients consumes significant costs.

Disclosure of Invention

Based on the defects in the prior art, the invention provides a competition data processing method based on a virtual environment. By the method, a user only needs to submit codes for realizing a training process and a prediction process to a physical server for creating the virtual environment of the competition platform, and does not need to download a data set and perform the steps of training and submitting a prediction result on own hardware resources, and all the steps of training, predicting, evaluating and the like are completed on a GPU (graphics processing unit) of the virtual environment created by a cloud server appointed by the competition platform, so that the competition data processing process is more standardized. According to the method designed by the invention, the training, predicting, evaluating and other processes in the machine learning competition can be automatically realized on the cloud server, and a user only needs to submit codes required by the training process. The method simplifies the operation of the user and saves time.

The competition data processing method based on the virtual environment is mainly divided into three processes of code submission by a user, model training and model evaluation, and all the processes are automatically completed in the virtual environment of the cloud server. After the user submits the training code, firstly, whether the user submits the code under the project for the first time or not and whether the repetition degree of the code submitted by the user at this time and the code submitted for the first time is too high or not are judged. And if the code repetition degrees of the two times are not too high, automatically creating a training task for the training, and adding the task into a waiting queue. When the task reaches the top of the waiting queue, the training process is automatically started. In the training process, a virtual environment is first created for the training task, and the dependent items required by the user code are downloaded in the environment. The training code is executed if the user code meets the specification. And if the training code is successfully executed, saving the trained model result to the specified path, and automatically executing the evaluation step. In the evaluation step, the test set data and tags are first downloaded into the virtual environment. If the user code does not have cheating behaviors, the difference between the label of the test set data and the real label is predicted by using the model trained in the training process, and the model is scored according to a certain algorithm. And then scoring the training task according to factors such as model scoring and submission time of the training task. And if the repetition degree of the code submitted by the user at this time and the first submitted code is too high, skipping the training and evaluation process, and directly returning the training result and the score of the code submitted by the user for the first time.

In a first aspect of the present invention, there is provided a virtual environment-based competition data processing method, comprising the steps of:

creating a virtual environment for executing user-written training code;

a user submits algorithm competition data corresponding to a competition task, wherein the algorithm competition data comprises a code sequence file for realizing the algorithm;

code virtualization processing, namely uploading the code sequence file to a physical server supporting the virtualization environment through the virtualization environment;

the physical server obtains the upload request and obtains the ID of the currently submitting user,

as a first innovation point of the invention, a virtual environment is created through a corresponding physical server, a training code is executed in the virtual environment, and a passage code sequence is handed over on the physical server, so that the defects in the prior art can be effectively avoided;

judging whether the user submits the code of the competition task for the first time;

if the code is submitted for the first time, the code is stored on a physical server;

otherwise, comparing the user code with the code submitted by the user for the competition task at the previous time;

judging whether the similarity between the code and the code submitted by the user for the competition task at the previous time is more than 95%;

if not, generating a training task for the user request, and adding the current task into a task queue waiting for training;

judging whether the current task is at the top of a waiting queue, wherein the waiting queue moves forwards after the GPU of the server finishes processing one task each time;

and if the current task is not at the top of the waiting queue, continuing waiting until the current task moves to the top of the queue.

Judging whether an unfinished training task exists on the GPU according to the ID of the user;

if so, continuing to wait until the task currently trained by the user is processed;

the training process for the current task is started.

After the user submits the training code and the training task reaches the top of the waiting queue, the training process is started, which specifically includes:

s201: the training process begins.

S202: allocating idle GPU resources for the training tasks;

s203: creating a virtual environment for the training task;

s204: acquiring a dependent item required by code operation from a training code written by a user;

s205: downloading and installing the dependent items mentioned in the previous step in a virtual environment;

s206: automatically judging whether the user code meets the standard required by the competition platform;

s207: if the code is not in accordance with the standard, throwing an exception that the code is not in accordance with the standard, and terminating the training process;

s208: if the user code meets the specification, executing the code;

s209: judging whether the code throws an exception or not in the execution process;

s210: if the exception is thrown in the execution, the exception of code execution error is continuously thrown, and the training process is terminated;

s211: after training is finished, storing the trained model to a specified path on a cloud server;

s212: the training process is ended.

Wherein, the model evaluation step is started after the training process is finished, and the method specifically comprises the following steps:

s301: starting an evaluation process;

s302: downloading competition test set data and corresponding labels thereof to a virtual environment from a server designated path according to the competition id;

s303: judging whether cheating behaviors exist in the user codes or not;

s304: if the user code has cheating behavior, throwing out code cheating exception and terminating the evaluation process;

s305: if the code has no cheating behavior, reading the training model saved in the step S211 from the appointed path of the server;

s306: calling a prediction method in the user code, inputting the test set data into the trained model read in the last step, and outputting a predicted label of each piece of data;

s307: judging whether the number of the predicted values output by the model is equal to the number of the labels of the test set or not;

s308: if the number of the predicted values output by the model is not equal to the number of the labels, throwing an assessment violation exception, and terminating the assessment process;

s309: and scoring the model according to the predicted value and the value of the label.

S310: and scoring the training task according to factors such as the scoring of the model in the last step, the task submitting time, the positions of the model scoring in the ranking list and the like.

Further, the competition data processing method based on the virtual environment further includes:

a701: successfully training cloud GPU model and outputting accuracy score predicted by model

A702: judging whether the training codes are identical to the codes of the project samples or not

A703: if the codes are identical to the sample codes, the training results are automatically returned to the user, and the works are invalid

A704: determining whether the model score exceeds a baseline standard score

A705: if the model score does not exceed the baseline standard score, the result is automatically released without manual review

A706: manually reviewing the comparison code if the model exceeds the baseline standard score

A707: if the manual review of the model fails, the reason and the notice of the failure of the review are sent to the user

A708: ranking the approved users according to the score scores

A709: calculating the amount of bonus amount obtained by scoring the model

A710: sending audit result notification through email and WeChat service platform

A711: bonus money can be presented through website real name authentication.

The competition data processing method based on the virtual environment further comprises the step of viewing the log in real time, and specifically comprises the following steps:

a514: judging whether the model training is successful or not

A515: an error result is returned through the e-mail and the mobile terminal social platform, so that the training error log is checked through the mobile terminal, and the modified code is resubmitted;

a516: and returning a success result through the E-mail and the mobile terminal social platform, and completing the notification of submitting cloud training, starting training, finishing training and evaluating the score of the model through the mobile terminal in the process.

If the submitted code is in error, the state of failure is returned under the corresponding state and the training is stopped, and the model training is finished, and the final model score is verified according to the evaluation file.

In a second aspect of the present invention, a virtual environment-based competition data processing system is provided, configured to implement the foregoing virtual environment-based competition data processing method, where the system further includes: and the result display module is used for comparing and displaying the related data of the local model terminal configuration module and the cloud feedback configuration module.

The result display module at least comprises: the device comprises a loss rate display module, an accuracy rate display module and a difference rate comparison module.

The competition data processing system based on the virtual environment further comprises a log analysis module, wherein the log analysis module records data event changes of the local model terminal configuration module and the cloud feedback configuration module, and the data event changes comprise training times, feedback times, downloading times and uploading times.

The above methods of the present invention may be implemented in the form of computer program instructions and may be integrated into a mobile terminal for operation, including the form in which the computer program instructions are stored in a computer-readable storage medium.

Accordingly, in a third aspect of the present invention, there is provided a computer readable storage medium having stored thereon computer readable instructions, which are executable by a processor and a memory, for implementing the method.

Further advantages of the present invention will be further apparent from the detailed description of the preferred embodiments in conjunction with the drawings.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.

FIG. 1 is a flow chart of the present application for user submission of code

FIG. 2 is a flow chart of the training process of the present application

FIG. 3 is a flowchart of the model evaluation steps that begin after the training process ends according to the present application

FIG. 4 is a diagram of a training log

FIG. 5 is a training flow diagram of cloud feedback

FIG. 6 is a review and feedback flow diagram

DETAILED DESCRIPTION OF EMBODIMENT (S) OF INVENTION

In order that the above objects, features and advantages of the present invention can be more clearly understood, the present invention will be further described with reference to the accompanying drawings and examples. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.

Referring next to FIG. 1, this embodiment discloses a flow diagram for user submission of code.

In fig. 1, the specific steps are as follows:

s101: the commit code process begins.

S102: the user designs a proper algorithm and writes codes for realizing the algorithm to complete the competition task according to the competition task and the characteristics of the data.

S103: and the user submits the written code file to the server. The tool for submitting the code can be an application program, a webpage platform and the like.

S104: the server obtains a request submitted by a user.

S105: the user's ID is obtained.

S106: it is determined whether the user is the first code to submit the item.

S107: and if the code is submitted for the first time, storing the code on the server.

S108: if not, the user code is compared with the code that the user submitted for the first time in the project.

S109: and judging whether the similarity between the code and the code submitted by the user for the first time in the project is higher than 95% by using a certain algorithm.

S110: and if the similarity of the codes of the two times is not higher than 95%, generating a training task for the request of the user, and adding the current task into a task queue waiting for training.

S111: and judging whether the current task is positioned at the top end of the waiting queue, wherein the waiting queue moves forwards after the GPU of the server finishes processing one task each time.

S112: and if the current task is not at the top of the waiting queue, continuing waiting until the current task moves to the top of the queue.

S113: and judging whether the user has an unfinished training task on the GPU currently according to the ID of the user.

S114: and if the user currently has the task which is being trained, the task request continues to wait until the task which is currently being trained by the user is processed.

S115: the training process for the current task is started.

S116: the commit code process ends.

Referring to fig. 2, after the user submits the training code and the training task reaches the top of the waiting queue, the training process starts, and fig. 2 shows a flowchart of the training process, which includes the following specific steps:

s201: the training process begins.

S202: and idle GPU resources are allocated for the training task, and the allocated resources can be a single GPU or a plurality of GPUs working in parallel.

S203: a virtual environment is created for the training task. The purpose of this is to ensure that the environment in which the user trains the code to run does not conflict with other code running environments. The virtual environment may be created by Docker or in other ways.

S204: and acquiring the dependency items required by the code operation from the training code written by the user.

S205: and downloading and installing the dependent items mentioned in the previous step in the virtual environment.

S206: and automatically judging whether the user code meets the standard required by the competition platform or not by using a certain algorithm.

S207: if the code is not in compliance with the specification, an exception is thrown that the code is out of compliance with the specification and the training process is terminated.

S208: if the user code meets the specification, the code is executed.

S209: and judging whether the code throws an exception or not in the execution process.

S210: if the exception is thrown in the execution, the exception of code execution error is continuously thrown, and the training process is terminated.

S211: and after the training is finished, storing the trained model to a specified path on a cloud server.

S212: the training process is ended.

After the training process is finished, the model evaluation step is started, and the evaluation process is as shown in fig. 3, and the specific steps are as follows:

s301: the evaluation process is started.

S302: and downloading the competition test set data and the corresponding labels thereof to the virtual environment from the server according to the competition id.

S303: and judging whether cheating behaviors exist in the user codes or not.

S304: and if the user code has cheating behaviors, throwing out a code cheating exception and terminating the evaluation process.

S305: if the code has no cheating action, the training model saved in step S211 is read from the server-specified path.

S306: and calling a prediction method in the user code, inputting the test set data into the trained model read in the last step, and outputting the predicted label of each piece of data.

S307: and judging whether the number of the predicted values output by the model is equal to the number of the labels of the test set.

S308: and if the number of the predicted values output by the model is not equal to the number of the labels, throwing an evaluation violation exception, and terminating the evaluation process.

S309: and scoring the model according to a certain algorithm according to the predicted value and the value of the label.

FIG. 4 shows the steps of the training log. Referring to fig. 4, the specific application steps of the training log include:

the training of each time shows the results: loss (test loss rate), acc (test accuracy rate), val _ loss (verification loss rate), val _ acc (verification accuracy rate) and best accurateof the training best accuracy;

in the current project directory, a local command line executes a/flyai train (script command), codes can be submitted to a cloud for GPU training, and training parameters of BATCH (data volume acquired by each training) and EPOCHS (number of training cycles) can be set in the local command line;

after successful submission, a website link with Train ID (training task number) is displayed in a local command line, copied and opened in any browser, and a model training log displayed in real time can be seen;

the specific application steps of the training log are S401 to S410.

Fig. 5 is a training flow diagram of cloud feedback.

If the submitted code is in error, a failure state is returned under a corresponding state and the training is stopped, and the model score is finally verified according to the evaluation file after the model training is finished;

the training notification can push messages by two channels, namely an E-mail channel and a WeChat service number public platform;

binding the WeChat when registering the account number, and feeding back operation information in time and pushing a message notice of successful submission when a WeChat third-party access service is used for realizing that the training is submitted to the cloud operation locally;

all the operation prompt notifications are respectively as follows: submitting cloud training, starting cloud GPU training, failing/succeeding in training, feeding back scores, passing/failing model audit and updating notifications in a ranking list;

a501: entry into project package catalogue at local Terminal or third party IDE tool Terminal

A502: determining the system environment to perform the operation

A503: if the local environment is macOS/Linux (operating system), chmod + x./flyai (script command) needs to be executed for operation authorization

A504: if the local environment is Windows, no operation is performed, and the next step is skipped

A505: execute initialization command/flyyi init (script command)

A506: judging whether to execute for the first time in the local command line/flyyi init (script command)

A507: the command is executed for the first time, the deep learning development environment is automatically installed, and the two-dimensional code request WeChat code scanning login website account is displayed

A508: per fyyi test (script command) into local calling

A509: determine if the local command line is the first execution/flyai test (script command)

A510: the first execution,/flyai test (script command) will request to download 100 test data sets of the project from the cloud to the local end, and generate a folder named "data" at the local end, which contains two subfolders: "input", "output".

A511: the method realizes automatic detection of python environment dependence used by the project and automatic downloading, and performs 10 times of local model training by using the local CPU equipment

A512: when a local command is entered into a project directory, execution of a/flyyi train (script command) can submit code to the cloud for training, while BATCH (amount of data acquired per training) and EPOCHS (number of training cycles) can be set in the local command line

A513: if the submission is successful, a website link with a TrainID (training task number) is displayed on a local command line, and the content of a training log is checked in real time by opening the link, wherein the training log contains a plurality of training states.

A514: judging whether the model training is successful or not

A515: error results are returned through the e-mail and the mobile terminal social platform, training error logs are checked through the mobile terminal, and the modified codes are resubmitted

A516: successful results are returned through the e-mails and the mobile terminal social platform, and the mobile terminal completes the notification of submitting cloud training, starting training, completing training and evaluating model scores in the process

FIG. 6 is a flow chart of auditing and feedback, which essentially includes steps S601-S607.

After S601-S607 are executed, the method further includes the following steps:

A704: determining whether the model score exceeds a baseline standard score

A708: ranking the approved users according to the score scores

A709: calculating the amount of bonus amount obtained by scoring the model

A711: bonus money can be presented through website real name authentication.

And submitting the codes to a cloud GPU for automatic training and evaluation through a convenient script command, displaying a training log in real time and feeding back a training result in time.

In a word, all processes are automatically completed on the cloud server. After the user submits the training code, firstly, whether the user submits the code under the project for the first time or not and whether the repetition degree of the code submitted by the user at this time and the code submitted for the first time is too high or not are judged. And if the code repetition degrees of the two times are not too high, automatically creating a training task for the training, and adding the task into a waiting queue. When the task reaches the top of the waiting queue, the training process is automatically started. In the training process, a virtual environment is first created for the training task, and the dependent items required by the user code are downloaded in the environment. The training code is executed if the user code meets the specification. And if the training code is successfully executed, saving the trained model result to the specified path, and automatically executing the evaluation step. In the evaluation step, the test set data and tags are first downloaded into the virtual environment. If the user code does not have cheating behaviors, the difference between the label of the test set data and the real label is predicted by using the model trained in the training process, and the model is scored according to a certain algorithm. And then scoring the training task according to factors such as model scoring and submission time of the training task. And if the repetition degree of the code submitted by the user at this time and the first submitted code is too high, skipping the training and evaluation process, and directly returning the training result and the score of the code submitted by the user for the first time.

While preferred embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous modifications, changes, and substitutions will now occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims

1. A competition data processing method based on a virtual environment comprises the following steps:

creating a virtual environment for executing user-written training code;

the physical server acquires the uploading request and the ID of the current submitting user, and judges whether the user submits the code of the competition task for the first time;

if the current task is not at the top of the waiting queue, continuing waiting until the current task moves to the top of the queue;

the training process for the current task is started.

2. The method of claim 1, wherein starting the training process after the user submits the training code and the training task reaches the top of the waiting queue, specifically comprising:

s201: the training process begins;

s202: allocating idle GPU resources for the training tasks;

s203: creating a virtual environment for the training task;

s208: if the user code meets the specification, executing the code;

s212: the training process is ended.

3. The method according to claim 2, wherein the step of model evaluation is initiated after the training process is completed, and specifically comprises:

s301: starting an evaluation process;

s303: judging whether cheating behaviors exist in the user codes or not;

s309: scoring the model according to the predicted value and the value of the label;

4. The method of any of claims 1-3, further comprising:

A704: determining whether the model score exceeds a baseline standard score

A708: ranking the approved users according to the score scores

A709: calculating the amount of bonus amount obtained by scoring the model

A711: bonus money can be presented through website real name authentication.

5. The method of claim 1, further comprising the step of viewing the log in real time, specifically comprising:

a514: judging whether the model training is successful or not

6. The method of claim 5, wherein if the submitted code is in error, the submitted code returns to a failed state in a corresponding state and stops training, and the model training is completed, and a final model score is verified according to the evaluation document.

7. A virtual environment based contest data processing system for implementing the method of any one of claims 1-6, said system further comprising: and the result display module is used for comparing and displaying the related data of the local model terminal configuration module and the cloud feedback configuration module.

8. The contest data processing system of claim 7, said result display module including at least: the device comprises a loss rate display module, an accuracy rate display module and a difference rate comparison module.

9. The competition data processing system of claim 7, further comprising a log analysis module that records data event changes of the local model terminal configuration module and the cloud feedback configuration module, including training times, feedback times, download times, and upload times.

10. A computer readable storage medium having stored thereon computer executable instructions, the instructions being executable with a processor and a memory for implementing the method of any one of claims 1-6.