[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

WO2024007849A1 - 面向智能计算的分布式训练容器调度 - Google Patents

面向智能计算的分布式训练容器调度 Download PDF

Info

Publication number
WO2024007849A1
WO2024007849A1 PCT/CN2023/101093 CN2023101093W WO2024007849A1 WO 2024007849 A1 WO2024007849 A1 WO 2024007849A1 CN 2023101093 W CN2023101093 W CN 2023101093W WO 2024007849 A1 WO2024007849 A1 WO 2024007849A1
Authority
WO
WIPO (PCT)
Prior art keywords
container
node
target
computing node
model
Prior art date
Application number
PCT/CN2023/101093
Other languages
English (en)
French (fr)
Inventor
李勇
程稳
吴运翔
陈�光
朱世强
曾令仿
Original Assignee
之江实验室
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 之江实验室 filed Critical 之江实验室
Priority to JP2023547139A priority Critical patent/JP2024529206A/ja
Publication of WO2024007849A1 publication Critical patent/WO2024007849A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45562—Creating, deleting, cloning virtual machine instances
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the present disclosure relates to the field of computer technology, and in particular, to a method, device, storage medium and electronic device for intelligent computing-oriented distributed training container scheduling.
  • a common way during model training is that the server deploys the sub-models split based on the model to one or more containers.
  • Multiple containers can share the computing power of computing nodes (such as GPUs) on the physical machine. resources for model training.
  • the computing resources of each computing node may change dynamically, and multiple containers share a physical machine. The performance of the container may be affected by other containers, which will reduce the efficiency of distributed training. .
  • the present disclosure provides a method, device, storage medium and electronic device for intelligent computing-oriented distributed training container scheduling to partially solve the above-mentioned problems existing in the prior art.
  • the present disclosure provides a distributed training container scheduling method for intelligent computing, including:
  • At least one computing node for deploying the multiple sub-models is determined, and multiple containers are created on the at least one computing node to respectively deploy the multiple sub-models to the multiple inside the container;
  • the computing node According to the load data of the at least one computing node and the corresponding operation time of the multiple containers, determine the computing node that needs to adjust the container distribution as the target node;
  • the training task of the target model is executed.
  • split the target model to obtain the multiple sub-models including:
  • the network layers included in the target model are split to obtain the multiple sub-models.
  • each container in the plurality of containers determine the operation time of the sub-model deployed in the container when executing the training task of the sub-model, including:
  • the start time of the training task for executing the sub-model deployed in the container contained in the training statistics and the end time to determine the operation time of the sub-model when executing the training task of the sub-model deployed in the container;
  • the training statistical information stored in the shared storage system is determined based on the target log generated by each computing node in the at least one computing node when executing the model training task, and the target log is based on a preset Specified keywords are filtered from the logs generated by each computing node in the at least one computing node.
  • the training statistical information is accumulated to a specified number, it is written to the shared storage system and retrieved from The at least one computing node is deleted.
  • determine the computing nodes that need to adjust container distribution based on the load data of the at least one computing node and the corresponding computing time of the multiple containers including:
  • the computing node that needs to adjust the container distribution is determined.
  • determine the computing nodes that need to adjust the container distribution including:
  • the first node and the second node are determined as computing nodes that need to adjust container distribution.
  • determine the computing nodes that need to adjust the container distribution including:
  • the computing node determines the computing node to deploy the new container to be created based on the load data of the at least one computing node , as the computing node that needs to adjust the container distribution;
  • the distribution of containers in the target node is adjusted, including:
  • determine the computing node on which the new container to be created is deployed as the computing node that needs to adjust the container distribution including:
  • Sort other computing nodes except the computing node where the target container is deployed in order of load data of the at least one computing node from small to large to obtain a second sorting result;
  • the adjacently sorted computing nodes will be The node with the lower load among the two computing nodes is used as the computing node to deploy the new container to be created. Otherwise, continue to determine whether the load difference between the two adjacently sorted computing nodes is within the preset range. Until all computing nodes in the second sorting result are traversed or the computing node on which the new container to be created is deployed is determined.
  • the method also includes:
  • the computing node on which the new container to be created is deployed is determined from other computing nodes except the associated node.
  • the computing nodes distributed throughout the container include:
  • the computing node where the target container is deployed will be used as the computing node that needs to adjust the container distribution, where the sub-model deployed in the specified container is not the same as the computing node where the container distribution needs to be adjusted.
  • the submodels deployed in the above target container are the same;
  • the distribution of containers in the target node is adjusted, including:
  • adjusting the distribution of containers in the target node with the similarity of the operation durations corresponding to the multiple containers as an adjustment target includes:
  • the computing node corresponding to the target container is used as the first target node
  • the computing node with a lighter load is determined from the any two computing nodes as the second target node.
  • the present disclosure provides a device for distributed training container scheduling for intelligent computing, including:
  • the first acquisition module is used to acquire sample data and target models
  • a splitting module used to split the target model to obtain multiple sub-models, wherein each of the multiple sub-models includes part of the network layers in the target model;
  • a first determining module configured to determine at least one computing node for deploying the multiple sub-models according to the multiple sub-models, and create multiple containers on the at least one computing node to store the multiple sub-models Deploy to the multiple containers respectively;
  • a first training module configured to use the sample data to perform a model training task to train the multiple sub-models deployed in the multiple containers;
  • the second acquisition module is used to acquire the load data of the at least one computing node when executing the model training task, and for each container in the plurality of containers, determine when executing the training task of the sub-model deployed in the container.
  • the operation time of the sub-model is used as the operation time corresponding to the container;
  • the second determination module is used to determine the computing node that needs to adjust the container distribution as the target node based on the load data of the at least one computing node and the operation time corresponding to the plurality of containers;
  • An adjustment module configured to adjust the distribution of containers in the target node with the operation duration corresponding to the plurality of containers being close to each other as an adjustment target;
  • the second training module is used to perform the training task of the target model based on each computing node after adjusting the container distribution.
  • the splitting module is specifically used to determine the operation time of the target model when performing a model training task; split the network layers included in the target model according to the operation time of the target model, to obtain the multiple sub-models.
  • the second acquisition module is specifically configured to determine the training statistical information corresponding to the container from a preset shared storage system; execute the sub-model deployed in the container according to the training statistical information contained in the training statistical information.
  • the start time and end time of the training task determine the operation time of the sub-model when executing the training task of the sub-model deployed in the container;
  • the training statistical information stored in the shared storage system is determined based on the target log generated by each computing node in the at least one computing node when executing the model training task, and the target log is based on a preset Specified keywords are filtered from the logs generated by each computing node in the at least one computing node.
  • the training statistical information is accumulated to a specified number, it is written to the shared storage system and retrieved from The at least one computing node is deleted.
  • the second determination module is specifically configured to sort the containers in descending order of the operation durations corresponding to the containers in each computing node to obtain the first sorting result;
  • the container located in front of the ranking in the sorting result is used as the target container; according to the load data of the target container and each computing node, Determine the computing nodes that need to adjust container distribution.
  • the second determination module is specifically configured to sort the plurality of containers in descending order of corresponding operation durations of the plurality of containers to obtain the first sorting result;
  • a container located in front of the preset ranking in a sorting result is used as a target container; based on the load data of the target container and the at least one computing node, the computing node that needs to adjust the container distribution is determined.
  • the second determination module is specifically configured to, if it is determined that the difference between the operation time corresponding to the target container and the operation time corresponding to other containers exceeds a second set threshold, based on the at least one computing node Load data determines the computing node where the new container to be created is deployed, as the computing node where the container distribution needs to be adjusted;
  • the adjustment module is specifically used to create a new container in the target node with the operation time corresponding to the multiple containers being close to each other as the adjustment target, and perform the model data of the sub-model deployed in the target container. Copy to deploy the copied submodel in the new container.
  • the second determination module is specifically configured to sort other computing nodes except the computing node where the target container is deployed in order of load data of the at least one computing node from small to large, to obtain the first Second sorting result; according to the second sorting result, determine whether the load difference between the two adjacently sorted computing nodes is within the preset range; for any two adjacently sorted calculations in the other computing nodes node, if it is determined that the load difference between the two adjacently sorted computing nodes does not fall within the preset range, then the node with the lower load among the two adjacently sorted computing nodes will be used as the node to be deployed.
  • the computing node of the new container created otherwise, continue to determine whether the load difference between the two adjacently sorted computing nodes is within the preset range until all computing nodes in the second sorting result are traversed or Until the computing node on which the new container to be created is deployed is determined.
  • the second determination module is also configured to, if it is determined that the load difference between two adjacently sorted computing nodes in the second sorting result is within the preset range, determine and The sub-model corresponding to the new container to be created has a sub-model with a network layer dependency relationship, as an associated sub-model; determine the computing node where the associated sub-model is deployed, as an associated node; test the associated node and remove the The network delay between other computing nodes other than the associated node; according to the network delay obtained by the test, the computing node to deploy the new container to be created is determined from other computing nodes other than the associated node.
  • the second determination module is specifically configured to determine the computing node on which the target container is deployed; if it is determined that the specified container is also deployed on the computing node where the target container is deployed, the computing node of the target container will be deployed. Node, as the computing node that needs to adjust the container distribution, wherein the sub-model deployed in the specified container is the same as the sub-model deployed in the target container;
  • the adjustment module is specifically configured to delete the target container or the specified container in the computing node where the target container is deployed, with the operation duration corresponding to the multiple containers being close to each other as an adjustment target.
  • the adjustment module is specifically configured to set the adjustment target for the containers in the target node by taking the corresponding operation durations of the plurality of containers to be close and the load of the at least one computing node to be close to each other as the adjustment target. distribution is adjusted.
  • the present disclosure provides a computer-readable storage medium that stores a computer program.
  • the computer program is executed by a processor, the above-mentioned distributed training container scheduling method for intelligent computing is implemented.
  • the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor.
  • the processor executes the program, the above-mentioned distributed training container scheduling for intelligent computing is implemented. Methods.
  • the distributed training container scheduling method for intelligent computing splits the target model to obtain each sub-model; according to each sub-model, each computing node used to deploy each sub-model is determined, and each computing node is Create each container on the node to deploy each sub-model into each container respectively; use sample data to perform model training tasks to train the deployed sub-models in each container; based on the load data of each computing node and the operations corresponding to each container time duration, determine the computing node that needs to adjust the container distribution as the target node; adjust the distribution of each container in the target node with the similarity of the computing time corresponding to the containers in each computing node where the sub-model is deployed as the adjustment target, Continue to perform the training task of the target model.
  • the target model is first split into multiple sub-models, and then each computing node used to deploy each sub-model is determined, and each container is created on each computing node to Each sub-model is deployed into each container respectively to complete the training task through each computing node.
  • the present disclosure monitors the load data of each computing node, and adjusts the operation time corresponding to the containers in each computing node where the sub-model is deployed to be close, and adjusts the distribution of each container in each computing node. Dynamic adjustment is beneficial to load balancing among various computing nodes and further improves the efficiency of model training.
  • Figure 1 is a schematic flowchart of a distributed training container scheduling method for intelligent computing provided by an embodiment of the present disclosure
  • Figure 2 is a schematic diagram of system relationships provided by an embodiment of the present disclosure
  • Figure 3 is a schematic diagram of container adjustment provided by an embodiment of the present disclosure.
  • Figure 4 is a schematic diagram of a device structure for distributed training container scheduling for intelligent computing provided by an embodiment of the present disclosure
  • FIG. 5 is a schematic structural diagram of an electronic device corresponding to FIG. 1 provided by an embodiment of the present disclosure.
  • Figure 1 is a schematic flowchart of a distributed training container scheduling method for intelligent computing provided in the present disclosure, which includes the following steps:
  • S102 Split the target model to obtain multiple sub-models, where each sub-model includes part of the network layers in the target model.
  • this disclosure Provides a distributed training container scheduling method for intelligent computing, using multiple machines and multiple cards to conduct distributed training of the model.
  • the execution subject of the present disclosure may be a system, an electronic device such as a laptop or a desktop computer, or a system for performing model training tasks (the system may be composed of a device cluster composed of multiple terminal devices).
  • the system may be composed of a device cluster composed of multiple terminal devices.
  • the system is used as the execution subject to describe the method of distributed training container scheduling for intelligent computing provided by the present disclosure.
  • the system can obtain sample data and a target model, and then split the target model to obtain multiple sub-models, where each sub-model contains part of the network layers in the target model.
  • the system can determine the operation time of the target model when performing the model training task as the operation time of the target model. Based on the determined operating time of the target model, the system can split the different network layers included in the target model with the goal of making the operating time of each sub-model similar when executing the model training task.
  • the system can split it according to the operation time of the target model to obtain two split sub-models, so that one of the sub-models contains the first 10 network layers of the target model. , the other sub-model contains the last 20 network layers of the target model. Then, when the system performs the training tasks of these two sub-models, the operation time of the two sub-models is close, that is, the operation time of the two sub-models is the same. The difference falls within the preset range.
  • the system can also directly split the target model according to the number of network layers contained in the target model. Assuming that there are 30 network layers in the target model, the system can evenly divide the number of network layers in the target model, so that after splitting into two sub-models, one sub-model contains the first 15 network layers of the target model, and the other sub-model contains the first 15 network layers of the target model. The model contains the last 15 network layers of the target model. This disclosure does not limit the method of model splitting.
  • S104 According to the multiple sub-models, determine at least one computing node for deploying the multiple sub-models, and create multiple containers on the at least one computing node to respectively deploy the multiple sub-models to the within multiple containers.
  • the system can determine the computing nodes used to deploy multiple sub-models, and perform Create multiple containers on the node to deploy multiple sub-models into the multiple containers respectively.
  • Each physical machine has 2 computing nodes (such as GPU). After splitting the target model into 20 sub-models, the system can be created on each computing node. 20 containers to deploy the 20 split sub-models into 20 containers respectively.
  • S106 Use the sample data to perform a model training task to train multiple sub-models deployed in the multiple containers.
  • the system can perform model training tasks using sample data to train deployed sub-models within each container.
  • the system can use a log collection framework to collect relevant data during the training process of each sub-model.
  • the relevant data includes all data generated by each sub-model during the training process and is used to reflect The calculation and operation of each sub-model on the container.
  • the system can use log printing to collect relevant data.
  • the system can print the start and end of model calculation, the start and end of memory access, and other time points as training statistical information to the log.
  • the system can add container address information, thread numbers and other information that can uniquely identify the training thread to the log content. At the same time, the system can also add and other information to the log content. Keywords used to distinguish log content, such as container-adaptive-adjust.
  • the system can continuously scan newly generated logs. If the log starting from a time point is scanned, the system can record the execution time of the sub-model (such as calculation, memory access, etc.) during the training process. and thread number and other unique identification information, and then continue scanning until the log at the end of the time point is scanned, and then calculate the execution time of this training process.
  • the execution time of the sub-model such as calculation, memory access, etc.
  • the system can filter out the target logs generated during model training based on keywords, then determine the start time and end time of the execution of the sub-model based on the target log, and record the execution time and thread number of the sub-model in the target log. Information such as this is sent to the shared storage system for storage as the training statistical information corresponding to the sub-model's corresponding container.
  • each computing node will obtain the training statistical information of each sub-model through the filtered target log. If the number of training statistics does not exceed the preset threshold, the system can continue to scan logs until the number of training statistics exceeds the preset threshold. At this time, the system will send the training statistics in batches to the preset shared storage system.
  • each computing node after each computing node sends the training statistical information to the shared storage system in batches, the system can delete the training statistical information retained in each computing node, and then continue to record the training statistical information corresponding to each container until the distribution Training is over.
  • the system can also preset a batch sending time. If the last time the system sent training statistics to the shared storage system in batches exceeds the preset sending time, the system will send the training statistics to the shared storage system in batches. For example, if the preset sending time is 15 minutes, during the training process of each sub-model, the system can send training statistical information in batches to the shared storage system every fifteen minutes.
  • each training statistical information stored in the shared storage system is determined based on the target log generated by each computing node when executing the model training task.
  • the target log is obtained from each computing node based on the preset designated keywords.
  • the training statistical information filtered out from the generated logs is written to the shared storage system and deleted from each computing node after it accumulates to a specified number or reaches a preset time.
  • S108 Obtain the load data of the at least one computing node when executing the model training task, and for each container, determine the operation time of the sub-model when executing the training task of the sub-model deployed in the container, as the corresponding the operation time.
  • the system can obtain the load data of each computing node when executing the model training task, and for each container, determine the operation time of the sub-model when executing the training task of the sub-model deployed in the container, as the operation time corresponding to the container. .
  • the subsequent system can analyze the running status of the containers based on the corresponding computing time of each container and the load data of each computing node, and then adjust the container distribution.
  • the system can read the training statistical information corresponding to each container from a preset shared storage system. For each container, the system can determine the time when the training task of the sub-model deployed in the container is executed based on the start time and end time of the training task of the sub-model deployed in the container contained in the training statistics. The running time of the model.
  • the system continuously writes training statistical information to the shared storage system.
  • the system reads the training statistics corresponding to each container from the shared storage system and analyzes the status of the container, it can obtain the training statistics generated by each sub-model in the previous round of training. information.
  • the training iteration order corresponding to the data used by the system to analyze the running status of the container is one behind the training iteration order of the current model training.
  • the training iteration order of the current model training is i
  • the training iteration order corresponding to the training statistical information read by the system from the shared storage system is i-1.
  • the system can also store the training iteration sequence as one of the keywords in the shared storage system, so that the training statistical information corresponding to the same iteration sequence is continuously stored.
  • S112 Adjust the distribution of containers in the target node with the operation durations corresponding to the multiple containers being close to each other as the adjustment target.
  • the system After determining the load data of each computing node and the corresponding computing time of each container, the system can determine the computing node that needs to adjust the container distribution as the target node.
  • the system can sort the containers according to the order of the operation time corresponding to the multiple containers from large to small, obtain the first sorting result, and rank one or more of the first sorting results before the preset ranking.
  • container as one or more target containers.
  • the system uses the computing time corresponding to each container to reflect the running status of each container. Subsequently, the computing nodes that need to adjust the container distribution can be determined based on the computing time corresponding to each container.
  • the system can obtain the first sorting result in descending order of the operation time corresponding to each container in each computing node, and use the first five containers in the sorting result as targets. container.
  • system can also determine the target container through other methods.
  • the system can obtain the load information of each computing node, and based on the load information of each computing node, determine the computing node with the lowest GPU utilization among each computing node. If the GPU utilization of the computing node is lower than the preset threshold, the computing node The container with the highest I/O load in the node is used as the target container.
  • the system can determine the computing nodes that need to adjust the container distribution based on the load data of the target container and each computing node.
  • the system can determine the computing node on which the target container is deployed. As the first node, if it is determined based on the load data of the first node that the load of the first node is higher than the first set threshold, then the system determines the computing node for deploying the target container from other computing nodes. Deploy the computing nodes of some containers in the first node as the second node.
  • the system can determine the first node and the second node as the computing nodes that need to adjust the container distribution, and set the computing time corresponding to the containers in each computing node where the sub-model is deployed to be close as the adjustment target, and set the target in the first node to The container is migrated to the second node.
  • the first set threshold may be preset, or may be an average load of other computing nodes except the first node.
  • the system determines that the load value of the first node is 20 based on the load data of the first node (the load value is used to characterize the level of the load, and the load value is positively correlated with the load). If the first set threshold is 10, or this The average load of each computing node is 10. At this time, the system needs to determine the computing node used to deploy some containers in the first node from other computing nodes except the first node as the second node.
  • the system can first determine the target container with the highest I/O load on the first node, and then determine the computing node with the lowest I/O load based on the load data of other computing nodes except the first node. This node serves as the second node. At this time, the target container in the first node is migrated to the second node to adjust the distribution of containers in each computing node.
  • the computing nodes that need to adjust the container distribution can also be determined through other methods.
  • the system can determine the computing node to deploy the new container to be created based on the load data of each computing node, as Determine the computing nodes that need to adjust container distribution.
  • the second set threshold may be preset, or may be an average value corresponding to the calculation time corresponding to each container.
  • the system determines the operation time corresponding to the target container and the operation time corresponding to other containers. The difference between the times exceeds the second threshold set by the system (such as 5 minutes). At this time, the system can determine the computing node to deploy the new container to be created based on the load data of each computing node, as a basis for determining that the container needs to be adjusted. Distributed computing nodes serve as target nodes.
  • the system can sort the computing nodes other than the computing nodes where the target container is deployed in order of the load data of the computing nodes from small to large, obtain the second sorting result, and judge the adjacent neighbors in sequence according to the second sorting result. Whether the load difference between the two sorted compute nodes is within a preset range.
  • the system can separate the two adjacently sorted computing nodes.
  • the node with the lower load among the computing nodes is used as the computing node to deploy the new container to be created. Otherwise, continue to determine whether the load difference between the two adjacently sorted computing nodes is within the preset range until the traversal until all computing nodes in the second sorting result are completed or the computing node on which the new container to be created is deployed is determined.
  • the load data of each computing node can be characterized by GPU utilization, CPU utilization, memory usage, and bandwidth of the storage device.
  • the system can first sort the computing nodes except the computing nodes where the target container is deployed according to the GPU utilization data of each computing node in ascending order.
  • the adjacent computing nodes will be The node with lower GPU utilization among the two sorted computing nodes is used as the computing node to deploy the new container to be created. Otherwise, continue to determine whether the GPU utilization difference between the next two adjacent sorted computing nodes is located Within the preset range, until the compute node on which the new container to be created is deployed is determined.
  • the system can order the CPU utilization data of each computing node from small to large. , re-sort other computing nodes except the computing nodes where the target container is deployed, and re-obtain the second sorting result.
  • the two adjacently sorted computing nodes will be The node with lower CPU utilization among the computing nodes is used as the computing node to deploy the new container to be created.
  • the system can sequentially compare the GPU utilization, CPU utilization, memory usage, storage device bandwidth and other data sizes of the computing nodes until the computing node to deploy the new container to be created is determined.
  • the system can also use other methods to determine the computing node. Deploy the compute node on which the new container to be created is deployed.
  • the system may determine that a network layer exists in the sub-model corresponding to the new container to be created.
  • Submodels of dependencies as associated submodels. For example, if the output of one submodel is the input of another submodel, then the two submodels can be used as associated submodels.
  • the system can determine the computing node where the associated sub-model is deployed as an associated node.
  • the system can test the network delay between the associated node and other computing nodes other than the associated node, and then determine the new deployment to be created from the other computing nodes other than the associated node based on the network delay obtained by the test.
  • the compute node of the container can be determined.
  • the system can use the computing node with the smallest network delay between the associated nodes as the computing node to deploy the new container to be created.
  • the system can determine the average network delay between the associated node and other computing nodes, and use other computing nodes whose network delay time with the associated node is lower than the average value as computing nodes for deploying new containers to be created.
  • the system can deploy the computing node of the sub-model to adjust the target with similar computing time for the container, create a new container in the target node, and deploy the container in the target container. Copy the model data of the submodel to deploy the copied submodel in a new container.
  • system can also determine the target node through other methods.
  • the system can first determine the computing node where the target container is deployed. If it is determined that the computing node where the target container is deployed also has the specified container deployed, the system can use the computing node where the target container is deployed as the container that needs to be adjusted. Distributed compute nodes where the submodel deployed in the specified container is the same as the submodel deployed in the target container.
  • the system can adjust the target by setting the computing time corresponding to the container in the computing node where the sub-model is deployed to be close, and delete the target container or specified container in the computing node where the target container is deployed.
  • the system can adjust the operation time corresponding to the containers in the computing nodes where the sub-model is deployed to be close. , delete the container on the physical node, and retain only one container with the same model parameters of the sub-model deployed on the physical node.
  • the system always uses the computing time corresponding to the container in the computing node where the sub-model is deployed to be close, and the computing node where the container is deployed is The adjustment target is to have similar loads, and the distribution of containers in the target node is adjusted.
  • the system After the system adjusts the distribution of each container in the target node, based on each computing node after adjusting the container distribution, the system can continue to use sample data to perform the training task of the target model.
  • the system can perform a breakpoint save operation on all current containers to save the training information of the current training iteration sequence.
  • the system can obtain the previously saved training information, then start the training threads for the sub-models in all containers, and continue to train each sub-model. It is worth noting that the intermediate training variables of the submodel in the newly created container can be copied from other containers with the same model data as the submodel.
  • Figure 2 is a schematic diagram of system relationships provided by the present disclosure.
  • each computing node will continuously write training statistical information in batches to the shared storage system.
  • the analyzer Before adjusting the distribution of containers contained in each computing node, the analyzer can read the training statistics from the shared storage system to obtain the load data of each computing node when executing the model training task, and for each computing node.
  • Container determine the operation time of the sub-model when executing the training task of the sub-model deployed in the container, as the operation time corresponding to the container.
  • the scheduler can adjust the distribution of containers in each computing node.
  • Figure 3 is a schematic diagram of container adjustment provided by the present disclosure.
  • the scheduler can adjust the distribution of containers in the target node based on the adjustment target that the computing time corresponding to the containers in each computing node where the sub-model is deployed is close.
  • the specific adjustment method has been described in detail in steps S110 to S112.
  • each computing node can continue to perform the training task of the target model based on the adjusted container distribution.
  • the target model is first split into multiple sub-models, and then each computing node used to deploy each sub-model is determined, and each container is created on each computing node to Each sub-model is deployed into each container respectively to complete the training task through each computing node.
  • the present disclosure monitors the load data of each computing node, and adjusts the operation time corresponding to the containers in each computing node where the sub-model is deployed to be close, and adjusts the distribution of each container in each computing node. Dynamic adjustment is beneficial to load balancing among various computing nodes and further improves the efficiency of model training.
  • the present disclosure also provides a corresponding distributed training container scheduling device for intelligent computing, as shown in Figure 4.
  • Figure 4 is a schematic diagram of a distributed training container scheduling device for intelligent computing provided by the present disclosure, including:
  • the first acquisition module 400 is used to acquire sample data and target models
  • Splitting module 402 is used to split the target model to obtain multiple sub-models, wherein each sub-model of the multiple sub-models includes part of the network layer in the target model;
  • the first determining module 404 is configured to determine at least one computing node for deploying the multiple sub-models according to the multiple sub-models, and create multiple containers on the at least one computing node to combine the multiple sub-models. The models are deployed into the multiple containers respectively;
  • the first training module 406 is configured to use the sample data to perform a model training task to train the multiple sub-models deployed in the multiple containers;
  • the second acquisition module 408 is used to acquire the load data of the at least one computing node when executing a model training task, and for each container in the plurality of containers, determine the training task of executing the sub-model deployed in the container. is the operation time of the sub-model, as the operation time corresponding to the container;
  • the second determination module 410 is used to determine the computing node that needs to adjust the container distribution as the target node based on the load data of the at least one computing node and the operation duration corresponding to the multiple containers;
  • the adjustment module 412 is used to adjust the distribution of containers in the target node with the operation duration corresponding to the multiple containers being close to each other as the adjustment target;
  • the second training module 414 is used to execute the training task of the target model based on each computing node after adjusting the container distribution.
  • the splitting module 402 is specifically configured to determine the operation time of the target model when performing a model training task; split the network layers included in the target model according to the operation time of the target model. , to obtain the multiple sub-models.
  • the second acquisition module 408 is specifically configured to determine the training statistical information corresponding to the container from a preset shared storage system; execute the sub-model deployed in the container according to the sub-model contained in the training statistical information.
  • the start time and end time of the training task determine the operation time of the sub-model when executing the training task of the sub-model deployed in the container;
  • the training statistical information stored in the shared storage system is determined based on the target log generated by each computing node in the at least one computing node when executing the model training task, and the target log is based on a preset Specified keywords are filtered from the logs generated by each computing node in the at least one computing node.
  • the training statistical information is accumulated to a specified number, it is written to the shared storage system and retrieved from The at least one computing node is deleted.
  • the second determination module 410 is specifically configured to sort the plurality of containers in descending order of the corresponding operation durations of the plurality of containers to obtain the first sorting result;
  • the container located in front of the preset ranking in the first sorting result is used as the target container; based on the load data of the target container and the at least one computing node, the computing node that needs to adjust the container distribution is determined.
  • the second determination module 410 is specifically configured to determine the computing node on which the target container is deployed as the first node; if it is determined based on the load data of the first node that the load of the first node is higher than the Once the threshold is set, the computing node used to deploy some of the containers in the first node is determined from other computing nodes as the second node; the first node and the second node are determined as containers that need to be adjusted. Distributed computing nodes.
  • the second determination module 410 is specifically configured to: if it is determined that the difference between the operation time corresponding to the target container and the operation time corresponding to other containers exceeds a second set threshold, based on the at least one computing node Based on the load data, determine the computing node where the new container to be created is deployed, as the computing node where the container distribution needs to be adjusted;
  • the adjustment module 412 is specifically used to create a new container in the target node with the operation durations corresponding to the multiple containers being close to each other as the adjustment target, and adjust the model data of the sub-model deployed in the target container. Make a copy to deploy the copied submodel in the new container.
  • the second determination module 410 is specifically configured to sort other computing nodes except the computing node where the target container is deployed in order of load data of the at least one computing node from small to large, to obtain The second sorting result; according to the second sorting result, determine whether the load difference between the two adjacently sorted computing nodes is within the preset range; for any two adjacently sorted computing nodes in the other computing nodes The computing node, if it is determined that the load difference between the two adjacently sorted computing nodes does not fall within the preset range, then the node with the lower load among the two adjacently sorted computing nodes will be used as the deployment The computing node of the new container to be created. Otherwise, continue to determine whether the load difference between the two adjacently sorted computing nodes is within the preset range until all computing nodes in the second sorting result are traversed. Or until the computing node where the new container to be created is deployed is determined.
  • the second determination module 410 is further configured to, if it is determined that the load difference between two adjacently sorted computing nodes in the second sorting result is within the preset range, determine The sub-model that has a network layer dependency relationship with the sub-model corresponding to the new container to be created is used as an associated sub-model; the computing node where the associated sub-model is deployed is determined as an associated node; and the relationship between the associated node and all other sub-models is tested. other than the associated nodes mentioned above Calculate the network delay between the nodes; according to the network delay obtained by the test, determine the computing node to deploy the new container to be created from other computing nodes except the associated node.
  • the second determination module 410 is specifically configured to determine the computing node on which the target container is deployed; if it is determined that the specified container is also deployed on the computing node on which the target container is deployed, then the computing node on which the target container is deployed is deployed.
  • Computing node as the computing node that needs to adjust the container distribution, wherein the sub-model deployed in the specified container is the same as the sub-model deployed in the target container;
  • the adjustment module 412 is specifically configured to delete the target container or the designated container in the computing node where the target container is deployed, with the operation duration corresponding to the multiple containers being close to each other as the adjustment target.
  • the adjustment module 412 is specifically configured to adjust the calculation time of the multiple containers corresponding to each other and the load of the at least one computing node to be close to each other as the adjustment target. The distribution of containers is adjusted.
  • the present disclosure also provides a computer-readable storage medium, which stores a computer program.
  • the computer program can be used to execute a method for intelligent computing-oriented distributed training container scheduling provided in FIG. 1 .
  • the present disclosure also provides a schematic structural diagram corresponding to the electronic device shown in FIG. 1 shown in FIG. 5 .
  • the electronic device includes a processor, internal bus, network interface, memory and non-volatile memory, and of course may also include other hardware required for business.
  • the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the distributed training container scheduling method for intelligent computing described in Figure 1 above.
  • PLD Programmable Logic Device
  • FPGA Field Programmable Gate Array
  • HDL Hardware Description Language
  • ABEL Advanced Boolean Expression Language
  • AHDL Altera Hardware Description Language
  • Confluence a specific programming language
  • CUPL Cornell University Programming Language
  • HDCal a Hardware Description Language
  • JHDL Java Hardware Description Language
  • Lava Lava
  • Lola MyHDL
  • PALASM RHDL
  • VHDL Very-High-Speed Integrated Circuit Hardware Description Language
  • the controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (eg, software or firmware) executable by the (micro)processor. , logic gates, switches, Application Specific Integrated Circuit (ASIC), programmable logic controllers and embedded microcontrollers.
  • controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, For Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the memory's control logic.
  • the controller in addition to implementing the controller in the form of pure computer-readable program code, the controller can be completely programmed with logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded logic by logically programming the method steps. Microcontroller, etc. to achieve the same function. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the means for implementing various functions can be considered as structures within hardware components as well as software modules implementing the methods.
  • a typical implementation device is a computer.
  • the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device Or a combination of any of these devices.
  • embodiments of the present disclosure may be provided as methods, systems, or computer program products. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
  • computer-usable storage media including, but not limited to, disk storage, CD-ROM, optical storage, etc.
  • These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions
  • the device implements the functions specified in a process or processes of the flowchart and/or a block or blocks of the block diagram.
  • These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device.
  • Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.
  • a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
  • processors CPUs
  • input/output interfaces network interfaces
  • memory volatile and non-volatile memory
  • Memory may include non-permanent storage in computer-readable media, random access memory (RAM) and/or non-volatile memory in the form of read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
  • RAM random access memory
  • ROM read-only memory
  • flash RAM flash random access memory
  • Computer-readable media includes both persistent and non-volatile, removable and non-removable media that can be implemented by any method or technology for storage of information.
  • Information may be computer-readable instructions, data structures, modules of programs, or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), and read-only memory.
  • PRAM phase change memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • RAM random access memory
  • read-only memory read-only memory
  • ROM read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • flash memory or other memory technology
  • compact disc read-only memory CD-ROM
  • DVD digital versatile disc
  • Magnetic tape cassettes tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium can be used to store information that can be accessed by a computing device.
  • computer-readable media does not include transitory media, such as modulated data signals and carrier waves.
  • embodiments of the present disclosure may be provided as methods, systems, or computer program products. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
  • computer-usable storage media including, but not limited to, disk storage, CD-ROM, optical storage, etc.
  • program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types.
  • the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network.
  • program modules may be located in both local and remote computer storage media including storage devices.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

一种面向智能计算的分布式训练容器调度的方法，包括：将目标模型进行拆分，以得到多个子模型；根据多个子模型，确定用于部署多个子模型的至少一个计算节点，并在至少一个计算节点上创建多个容器，以将多个子模型分别部署到多个容器内；采用样本数据执行模型训练任务，以训练多个容器内的部署的多个子模型；根据至少一个计算节点的负载数据以及多个容器分别对应的运算时长，确定出需要调整容器分布的计算节点，作为目标节点；以多个容器分别对应的运算时长相接近为调整目标，对目标节点中的容器的分布进行调整；基于调整容器分布后的各计算节点，执行目标模型的训练任务。

Description

面向智能计算的分布式训练容器调度 技术领域
本公开涉及计算机技术领域，尤其涉及一种面向智能计算的分布式训练容器调度的方法、装置、存储介质及电子设备。
背景技术
随着人工智能的发展，机器学习的应用领域从广度向深度发展，这对模型训练和应用都提出了更高的要求。随着模型尺寸及训练数据量的大幅增加，为了提高模型的训练效率，基于容器的分布式训练被越来越广泛地应用。
具体地，在模型训练时常见的一种方式是：服务器将基于模型拆分后的子模型部署到一个或多个容器上，多个容器可以共用物理机上的计算节点(如GPU)的算力资源以进行模型训练。但在模型训练过程中，各个计算节点的算力资源可能存在动态变化，且多个容器共享一台物理机，容器的性能情况可能会受到其他容器的影响，这样反而会降低分布式训练的效率。
因此，如何在训练模型时对计算节点上部署的容器的分布进行动态调整，使得计算节点内训练子模型时间相接近且减少计算节点之间负载不均衡的现象，则是个亟待解决的问题。
发明内容
本公开提供一种面向智能计算的分布式训练容器调度的方法、装置、存储介质及电子设备，以部分的解决现有技术存在的上述问题。
本公开采用下述技术方案：
本公开提供了一种面向智能计算的分布式训练容器调度的方法，包括：
获取样本数据以及目标模型；
将所述目标模型进行拆分，以得到多个子模型，其中，所述多个子模型中每个子模型包含有所述目标模型中的部分网络层；
根据所述多个子模型，确定用于部署所述多个子模型的至少一个计算节点，并在所述至少一个计算节点上创建多个容器，以将所述多个子模型分别部署到所述多个容器内；
采用所述样本数据执行模型训练任务，以训练所述多个容器内的部署的所述多个子模型；
获取所述至少一个计算节点在执行模型训练任务时的负载数据，以及针对所述多个容器中每个容器，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长，作为该容器对应的运算时长；
根据所述至少一个计算节点的负载数据以及所述多个容器分别对应的运算时长，确定出需要调整容器分布的计算节点，作为目标节点；
以所述多个容器分别对应的运算时长相接近为调整目标，对所述目标节点中的容器的分布进行调整；
基于调整容器分布后的各计算节点，执行所述目标模型的训练任务。
可选地，将所述目标模型进行拆分，以得到所述多个子模型，包括：
确定在执行模型训练任务时所述目标模型的运算时长；
根据所述目标模型的运算时长，将所述目标模型中包含的网络层进行拆分，以得到所述多个子模型。
可选地，针对所述多个容器中每个容器，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长，包括：
从预设的共享存储系统中确定出该容器所对应的训练统计信息；
根据所述训练统计信息中包含的执行该容器内部署的子模型的训练任务的起始时间 以及结束时间，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长；
其中，所述共享存储系统中存储的训练统计信息是基于所述至少一个计算节点中每个计算节点在执行模型训练任务时所产生的目标日志确定出的，所述目标日志是根据预设的指定关键词，从所述至少一个计算节点中每个计算节点所产生的日志中过滤出来的，所述训练统计信息是在累计到指定数量后，写入到所述共享存储系统中，并从所述至少一个计算节点中删除。
可选地，根据所述至少一个计算节点的负载数据以及所述多个容器分别对应的运算时长，确定出需要调整容器分布的计算节点，包括：
按照所述多个容器分别对应的运算时长从大到小的顺序，对所述多个容器进行排序，得到第一排序结果；
将所述第一排序结果中位于预设排位前的容器，作为目标容器；
根据所述目标容器以及所述至少一个计算节点的负载数据，确定出需要调整容器分布的计算节点。
可选地，根据所述目标容器以及所述至少一个计算节点的负载数据，确定出需要调整容器分布的计算节点，包括：
确定部署所述目标容器的计算节点，作为第一节点；
若根据所述第一节点的负载数据确定所述第一节点的负载高于第一设定阈值，则从其他计算节点中确定出用于部署所述第一节点中部分容器的计算节点，作为第二节点；
将所述第一节点和所述第二节点确定为需要调整容器分布的计算节点。
可选地，根据所述目标容器以及所述至少一个计算节点的负载数据，确定出需要调整容器分布的计算节点，包括：
若确定所述目标容器对应的运算时长与其他容器对应的运算时间之间差值超过第二设定阈值，则根据所述至少一个计算节点的负载数据，确定部署待创建的新容器的计算节点，作为需要调整容器分布的计算节点；
以所述多个容器分别对应的运算时长相接近为调整目标，对所述目标节点中的容器的分布进行调整，包括：
以所述多个容器分别对应的运算时长相接近为调整目标，在所述目标节点中创建新容器，并对所述目标容器中部署的子模型的模型数据进行复制，以将复制得到的子模型部署在所述新容器中。
可选地，根据所述至少一个计算节点的负载数据，确定部署待创建的新容器的计算节点，作为需要调整容器分布的计算节点，包括：
按照所述至少一个计算节点的负载数据从小到大的顺序，将除部署有所述目标容器的计算节点以外的其他计算节点进行排序，得到第二排序结果；
按照所述第二排序结果，依次判断相邻排序的两个计算节点之间的负载差值是否位于预设范围内；
针对所述其他计算节点中任意相邻排序的两个计算节点，若确定该相邻排序的两个计算节点之间的负载差值未落入所述预设范围内，则将该相邻排序的两个计算节点中负载较低的节点，作为部署待创建的新容器的计算节点，否则，继续判断下一相邻排序的两个计算节点之间的负载差值是否位于预设范围内，直至遍历完所述第二排序结果中的所有计算节点或确定出部署待创建的新容器的计算节点为止。
可选地，所述方法还包括：
若确定所述第二排序结果中每个相邻排序的两个计算节点之间的负载差值位于所述预设范围内，则确定与所述待创建的新容器对应的子模型存在网络层依赖关系的子模型，作为关联子模型；
确定部署有所述关联子模型的计算节点，作为关联节点；
测试所述关联节点与除所述关联节点以外的其他计算节点之间的网络延迟；
根据测试得到的网络延迟，从除所述关联节点以外的其他计算节点中确定出部署待创建的新容器的计算节点。
可选地，根据所述目标容器以及所述至少一个计算节点的负载数据，确定出需要调 整容器分布的计算节点，包括：
确定部署所述目标容器的计算节点；
若确定部署所述目标容器的计算节点中还部署有指定容器，则将部署所述目标容器的计算节点，作为需要调整容器分布的计算节点，其中，所述指定容器中部署的子模型与所述目标容器中部署的子模型相同；
以所述多个容器分别对应的运算时长相接近为调整目标，对所述目标节点中的容器的分布进行调整，包括：
以所述多个容器分别对应的运算时长相接近为调整目标，将部署所述目标容器的计算节点中的所述目标容器或所述指定容器删除。
可选地，以所述多个容器分别对应的运算时长相接近为调整目标，对所述目标节点中的容器的分布进行调整，包括：
以所述多个容器分别对应的运算时长相接近，且所述至少一个计算节点分别的负载相接近为调整目标，对所述目标节点中的容器的分布进行调整；
若所述目标容器对应的运算时长大于其他容器对应的运算时长的容器数量超过第一阈值，则将所述目标容器对应的计算节点作为第一目标节点；
若所述各计算节点中任意两个计算节点的负载数据的差值大于第二阈值，则从所述任意两个计算节点确定出负载较轻的计算节点，作为第二目标节点。
本公开提供了一种面向智能计算的分布式训练容器调度的装置，包括：
第一获取模块，用于获取样本数据以及目标模型；
拆分模块，用于将所述目标模型进行拆分，以得到多个子模型，其中，所述多个子模型中每个子模型包含有所述目标模型中的部分网络层；
第一确定模块，用于根据所述多个子模型，确定用于部署所述多个子模型的至少一个计算节点，并在所述至少一个计算节点上创建多个容器，以将所述多个子模型分别部署到所述多个容器内；
第一训练模块，用于采用所述样本数据执行模型训练任务，以训练所述多个容器内的部署的所述多个子模型；
第二获取模块，用于获取所述至少一个计算节点在执行模型训练任务时的负载数据，以及针对所述多个容器中每个容器，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长，作为该容器对应的运算时长；
第二确定模块，用于根据所述至少一个计算节点的负载数据以及所述多个容器分别对应的运算时长，确定出需要调整容器分布的计算节点，作为目标节点；
调整模块，用于以所述多个容器分别对应的运算时长相接近为调整目标，对所述目标节点中的容器的分布进行调整；
第二训练模块，用于基于调整容器分布后的各计算节点，执行所述目标模型的训练任务。
可选地，所述拆分模块具体用于确定在执行模型训练任务时所述目标模型的运算时长；根据所述目标模型的运算时长，将所述目标模型中包含的网络层进行拆分，以得到所述多个子模型。
可选地，所述第二获取模块具体用于从预设的共享存储系统中确定出该容器所对应的训练统计信息；根据所述训练统计信息中包含的执行该容器内部署的子模型的训练任务的起始时间以及结束时间，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长；
其中，所述共享存储系统中存储的训练统计信息是基于所述至少一个计算节点中每个计算节点在执行模型训练任务时所产生的目标日志确定出的，所述目标日志是根据预设的指定关键词，从所述至少一个计算节点中每个计算节点所产生的日志中过滤出来的，所述训练统计信息是在累计到指定数量后，写入到所述共享存储系统中，并从所述至少一个计算节点中删除。
可选地，所述第二确定模块具体用于按照所述各计算节点中各容器对应的运算时长从大到小的顺序，对各容器进行排序，得到第一排序结果；将所述第一排序结果中位于设排位前的容器，作为目标容器；根据所述目标容器以及所述各计算节点的负载数据， 确定出需要调整容器分布的计算节点。
可选地，所述第二确定模块具体用于按照所述多个容器分别对应的运算时长从大到小的顺序，对所述多个容器进行排序，得到第一排序结果；将所述第一排序结果中位于预设排位前的容器，作为目标容器；根据所述目标容器以及所述至少一个计算节点的负载数据，确定出需要调整容器分布的计算节点。
可选地，所述第二确定模块具体用于若确定所述目标容器对应的运算时长与其他容器对应的运算时间之间差值超过第二设定阈值，则根据所述至少一个计算节点的负载数据，确定部署待创建的新容器的计算节点，作为需要调整容器分布的计算节点；
所述调整模块，具体用于以所述多个容器分别对应的运算时长相接近为调整目标，在所述目标节点中创建新容器，并对所述目标容器中部署的子模型的模型数据进行复制，以将复制得到的子模型部署在所述新容器中。
可选地，所述第二确定模块具体用于按照所述至少一个计算节点的负载数据从小到大的顺序，将除部署有所述目标容器的计算节点以外的其他计算节点进行排序，得到第二排序结果；按照所述第二排序结果，依次判断相邻排序的两个计算节点之间的负载差值是否位于预设范围内；针对所述其他计算节点中任意相邻排序的两个计算节点，若确定该相邻排序的两个计算节点之间的负载差值未落入所述预设范围内，则将该相邻排序的两个计算节点中负载较低的节点，作为部署待创建的新容器的计算节点，否则，继续判断下一相邻排序的两个计算节点之间的负载差值是否位于预设范围内，直至遍历完所述第二排序结果中的所有计算节点或确定出部署待创建的新容器的计算节点为止。
可选地，所述第二确定模块还用于，若确定所述第二排序结果中每个相邻排序的两个计算节点之间的负载差值位于所述预设范围内，则确定与所述待创建的新容器对应的子模型存在网络层依赖关系的子模型，作为关联子模型；确定部署有所述关联子模型的计算节点，作为关联节点；测试所述关联节点与除所述关联节点以外的其他计算节点之间的网络延迟；根据测试得到的网络延迟，从除所述关联节点以外的其他计算节点中确定出部署待创建的新容器的计算节点。
可选地，所述第二确定模块具体用于，确定部署所述目标容器的计算节点；若确定部署所述目标容器的计算节点中还部署有指定容器，则将部署所述目标容器的计算节点，作为需要调整容器分布的计算节点，其中，所述指定容器中部署的子模型与所述目标容器中部署的子模型相同；
所述调整模块，具体用于以所述多个容器分别对应的运算时长相接近为调整目标，将部署所述目标容器的计算节点中的所述目标容器或所述指定容器删除。
可选地，所述调整模块具体用于，以所述多个容器分别对应的运算时长相接近，且所述至少一个计算节点分别的负载相接近为调整目标，对所述目标节点中的容器的分布进行调整。
本公开提供了一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序被处理器执行时实现上述面向智能计算的分布式训练容器调度的方法。
本公开提供了一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现上述面向智能计算的分布式训练容器调度的方法。
本公开采用的上述至少一个技术方案能够达到以下有益效果：
本公开提供的面向智能计算的分布式训练容器调度的方法，将目标模型进行拆分，以得到各子模型；根据各个子模型，确定用于部署各个子模型的各计算节点，并在各计算节点上创建各容器，以将各子模型分别部署到各容器内；采用样本数据执行模型训练任务，以训练各容器内的部署的子模型；根据各计算节点的负载数据以及各容器对应的运算时长，确定出需要调整容器分布的计算节点，作为目标节点；以部署有子模型的各计算节点中的容器所对应的运算时长相接近为调整目标，对目标节点中各容器的分布进行调整，继续执行目标模型的训练任务。
从上述方法中可以看出，在执行模型训练任务时，先将目标模型拆分为多个子模型，再确定用于部署各个子模型的各计算节点，并在各计算节点上创建各容器，以将各子模型分别部署到所述各容器内，以通过各计算节点完成训练任务。在模型训练过程中，本公开监测各个计算节点的负载数据，以部署有子模型的各计算节点中的容器所对应的运算时长相接近为调整目标，对所述各计算节点中各容器的分布进行动态调整，有利于各个计算节点之间负载均衡，进一步地提高了模型训练的效率。
附图说明
此处所说明的附图用来提供对本公开的进一步理解，构成本公开的一部分，本公开的示意性实施例及其说明用于解释本公开，并不构成对本公开的不当限定。
图1为本公开实施例提供的一种面向智能计算的分布式训练容器调度的方法的流程示意图；
图2为本公开实施例提供的一种系统关系示意图；
图3为本公开实施例提供的容器调整示意图；
图4为本公开实施例提供的一种面向智能计算的分布式训练容器调度的装置结构的示意图；
图5为本公开实施例提供的一种对应于图1的电子设备的结构示意图。
具体实施方式
为使本公开的目的、技术方案和优点更加清楚，下面将结合本公开具体实施例及相应的附图对本公开技术方案进行清楚、完整地描述。所描述的实施例仅是本公开一部分实施例，而不是全部的实施例。基于本公开中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本公开保护的范围。
以下结合附图，详细说明本公开各实施例提供的技术方案。
图1为本公开中提供的一种面向智能计算的分布式训练容器调度的方法的流程示意图，包括以下步骤：
S100：获取样本数据以及目标模型。
S102：将所述目标模型进行拆分，以得到多个子模型，其中，每个子模型包含有所述目标模型中的部分网络层。
随着模型尺寸及训练数据量的大幅增加，大规模模型可能无法完整部署在单台物理机上，单个GPU卡的显存容量也无法满足大规模模型训练的要求，为了提高模型的训练效率，本公开提供一种面向智能计算的分布式训练容器调度的方法，采用多机多卡的方式对模型进行分布式训练。
本公开的执行主体可以是系统，也可以是诸如笔记本电脑，台式电脑等电子设备，抑或是用于执行模型训练任务的系统(该系统可由多个终端设备构成的设备集群构成)。为了便于说明，下面仅以系统为执行主体，对本公开提供的面向智能计算的分布式训练容器调度的方法进行说明。
系统可以获取样本数据以及目标模型，再将目标模型进行拆分，以得到多个子模型，其中，每个子模型包含有目标模型中的部分网络层。
在本公开中，系统对目标模型进行拆分的方法有多种。
具体地，系统可以确定在执行模型训练任务时目标模型的运算时长，作为目标模型的运算时长。根据确定出的目标模型的运算时长，系统可以以在执行模型的训练任务时各子模型的运算时长相接近为拆分目标，对目标模型中包含的不同网络层进行拆分。
例如，假设目标模型中包含的网络层有30层，系统可以根据目标模型的运算时长进行拆分，得到拆分后的两个子模型，使其中一个子模型中包含目标模型的前10层网络层，另外一个子模型中包含目标模型的后20层网络层，那么，系统在执行这两个子模型的训练任务时两个子模型的运算时长是相接近的，即，这两个子模型的运算时长的差值是落入预设范围内的。
当然，系统也可以直接根据目标模型中包含的网络层数量，对目标模型进行拆分。假设目标模型中包含的网络层有30层，系统可以均分目标模型的网络层数量，使得拆分后的两个子模型，其中一个子模型中包含目标模型的前15层网络层，另外一个子模型中包含目标模型的后15层网络层。本公开不对模型拆分的方式进行限制。
S104：根据所述多个子模型，确定用于部署所述多个子模型的至少一个计算节点，并在所述至少一个计算节点上创建多个容器，以将所述多个子模型分别部署到所述多个容器内。
根据多个子模型，系统可以确定用于部署多个子模型的计算节点，并在这些计算节 点上创建多个容器，以将多个子模型分别部署到所述多个容器内。
例如，假设目前由五台物理机共同完成模型训练任务，每台物理机上有2个计算节点(如GPU)，在将目标模型拆分成20个子模型后，系统可以在各个计算节点上共创建20个容器，以将拆分得到的20个子模型分别部署到20个容器内。
S106：采用所述样本数据执行模型训练任务，以训练所述多个容器内的部署的多个子模型。
系统可以采用样本数据执行模型训练任务，以训练各容器内的部署的子模型。
具体地，在对各个子模型进行模型训练时，系统可以采用日志采集框架来收集各个子模型训练过程中的相关数据，相关数据包括各子模型在训练过程中所产生的所有数据，用于反映各子模型在容器上的计算和运行情况。
具体地，系统可以采用日志打印的方式收集相关数据。在各子模型进行训练时，系统可以将模型计算开始和结束、内存访问开始和结束等时间点作为训练统计信息打印到日志。
为了从相关数据中筛选出训练统计信息，在打印日志时，系统可以在日志内容中加入容器地址信息、线程号等可以唯一标识训练线程的信息，同时，系统还可以在日志内容中添加和其他日志内容区别的关键字，比如container-adaptive-adjust。
在对各子模型进行模型训练时，系统可以不断扫描新生成的日志，若扫描到时间点开始的日志，系统可以记录这次训练过程中子模型的执行(比如计算、访存等)的时间和线程号等唯一标识信息，然后继续扫描，直到扫描到时间点结束的日志，再计算这次训练过程的执行时间。
系统可以根据关键字过滤出用于模型训练时所产生的目标日志，进而根据目标日志确定子模型的执行的开始时间和结束时间，并将目标日志中记录的子模型的执行的时间和线程号等信息，作为子模型对应容器所对应的训练统计信息，发送到共享存储系统中保存。
具体地，对于执行模型训练任务的各计算节点来说，各计算节点会通过过滤出的目标日志，来得到各子模型的训练统计信息。若训练统计信息数量没有超过预设的阈值，系统可以继续扫描日志，直到训练统计信息数量超过预设的阈值，此时，系统再将训练统计信息批量发送到预设的共享存储系统中。
值得说明的是，在各计算节点将训练统计信息批量发送到共享存储系统中后，系统可以将各计算节点中保有的训练统计信息删除，再继续记录各个容器对应的训练统计信息，直到分布式训练结束。
当然，系统也可以预设一个批量发送时间，若系统上一次将训练统计信息批量发送到共享存储系统的时间超过预设的发送时长，则将训练统计信息批量发送到共享存储系统中。例如，若预设的发送时长为15分钟，则在各子模型训练过程中，系统可以每隔十五分钟将训练统计信息批量发送到共享存储系统中。
也就是说，共享存储系统中存储的各训练统计信息是基于各计算节点在执行模型训练任务时所产生的目标日志确定出的，目标日志是根据预设的指定关键词，从各计算节点所产生的日志中过滤出来的，训练统计信息是在累计到指定数量后或达到预设时间后，写入到共享存储系统中，并从各计算节点中删除。
S108：获取所述至少一个计算节点在执行模型训练任务时的负载数据，以及针对每个容器，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长，作为该容器对应的运算时长。
系统可以获取各计算节点在执行模型训练任务时的负载数据，以及针对每个容器，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长，作为该容器对应的运算时长。
后续系统可以根据各个容器对应的运算时长以及各个计算节点的负载数据对容器的运行状态进行分析，进而调整容器分布。
具体地，系统可以从预设的共享存储系统中读取出每个容器所对应的训练统计信息。针对每个容器，系统可以根据训练统计信息中包含的执行该容器内部署的子模型的训练任务的起始时间以及结束时间，来确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长。
在模型训练的过程中，系统不断地往共享存储系统中写入训练统计信息，为了 减少对分布式训练的影响，在系统从共享存储系统中读取每个容器所对应的训练统计信息并对容器的状态进行分析时，可以获取各个子模型上一轮训练时所产生的训练统计信息。
也就是说，系统对容器运行状态进行分析时所采用的数据对应的训练迭代次序比当前的模型训练的训练迭代次序落后一个。例如，假设当前模型训练的训练迭代次序是i，那么系统从共享存储系统中读取的训练统计信息对应的训练迭代次序是i-1。
需要说明的是，为了提高共享存储系统的性能，系统可以将训练迭代次序也作为关键字之一存储到共享存储系统中，使得同一个迭代次序对应的训练统计信息连续存储。
S110：根据所述至少一个计算节点的负载数据以及所述多个容器对应的运算时长，确定出需要调整容器分布的计算节点，作为目标节点。
S112：以所述多个容器分别对应的运算时长相接近为调整目标，对所述目标节点中的容器的分布进行调整。
在确定各计算节点的负载数据以及各容器对应的运算时长后，系统可以确定出需要调整容器分布的计算节点，作为目标节点。
具体地，系统可以按照多个容器分别对应的运算时长从大到小的顺序，对容器进行排序，得到第一排序结果，并将第一排序结果中位于预设排位前的一个或多个容器，作为一个或多个目标容器。系统用各容器对应的运算时长来反映各容器的运行状态，后续可以基于各容器对应的运算时长确定出需要调整容器分布的计算节点。
例如，假设预设排位的具体数值为5，系统可以按照各计算节点中各容器对应的运算时长从大到小的顺序，得到第一排序结果，并将排序结果中前五个容器作为目标容器。
当然，系统也可以通过其他方式确定出目标容器。
例如，系统可以获取各计算节点的负载信息，基于各计算节点的负载信息，确定各计算节点中GPU利用率最低的计算节点，若该计算节点GPU利用率低于预设阈值，则将该计算节点中I/O负载最高的容器，作为目标容器。
在确定目标容器后，系统可以根据目标容器以及各计算节点的负载数据，确定出需要调整容器分布的计算节点。
具体地，系统可以确定部署目标容器的计算节点，作为第一节点，若根据第一节点的负载数据确定第一节点的负载高于第一设定阈值，则从其他计算节点中确定出用于部署第一节点中部分容器的计算节点，作为第二节点。
系统可以将第一节点和第二节点确定为需要调整容器分布的计算节点，以部署有子模型的各计算节点中的容器所对应的运算时长相接近为调整目标，将第一节点中的目标容器迁移到第二节点中。
其中，第一设定阈值可以是预先设定的，也可以是除第一节点外的其他计算节点负载的平均值。
例如，系统根据第一节点的负载数据确定出第一节点的负载值为20(负载值用于表征负载的高低，负载值与负载呈正相关关系)，若第一设定阈值为10，或者此时各计算节点负载的平均值为10，此时，系统需要从除第一节点外的其他计算节点中，确定出用于部署第一节点中部分容器的计算节点，作为第二节点。
具体地，系统可以先确定出第一节点中I/O负载最高的一个目标容器，再根据除第一节点外的其他计算节点的负载数据，确定出I/O负载最低的一个计算节点，将该节点作为第二节点，此时，将第一节点中的目标容器迁移到第二节点中，来调整各个计算节点中容器的分布。
当然，在本公开中，还可以通过其他方式来确定出需要调整容器分布的计算节点。
若系统确定目标容器对应的运算时长与其他容器对应的运算时间之间差值超过第二设定阈值，那么系统可以根据各计算节点的负载数据，确定部署待创建的新容器的计算节点，作为确定出需要调整容器分布的计算节点。
其中，第二设定阈值可以是预先设定的，也可以是各个容器对应的运算时长对应的平均值。
例如，若确定出的目标容器的数量为1，该目标容器对应的运算时长为20min，除该目标容器其他容器对应的运算时长为10min，系统确定目标容器对应的运算时长与其他容器对应的运算时间之间差值超过系统设定的第二设定阈值(如5min)，此时，系统可以根据各计算节点的负载数据，确定部署待创建的新容器的计算节点，作为确定出需要调整容器分布的计算节点，作为目标节点。
在这种情况下，系统用于确定出目标节点的方法有很多。
具体地，系统可以按照计算节点的负载数据从小到大的顺序，将除部署有目标容器的计算节点以外的其他计算节点进行排序，得到第二排序结果，按照第二排序结果，依次判断相邻排序的两个计算节点之间的负载差值是否位于预设范围内。
针对上述其他计算节点中任意相邻排序的两个计算节点，若确定该相邻排序的两个计算节点之间的负载差值未落入预设范围内，系统可以将该相邻排序的两个计算节点中负载较低的节点，作为部署待创建的新容器的计算节点，否则，继续判断下一相邻排序的两个计算节点之间的负载差值是否位于预设范围内，直至遍历完第二排序结果中的所有计算节点或确定出部署待创建的新容器的计算节点为止。
其中，各计算节点的负载数据可以由GPU利用率、CPU利用率、内存使用率、存储设备的带宽来表征。
例如，系统可以先按照各计算节点的GPU利用率数据从小到大的顺序，将除部署有目标容器的计算节点以外的其他计算节点进行排序。
针对所述其他计算节点中任意相邻排序的两个计算节点，若确定该相邻排序的两个计算节点之间的GPU利用率的差值未落入预设范围内，则将该相邻排序的两个计算节点中GPU利用率较低的节点，作为部署待创建的新容器的计算节点，否则，继续判断下一相邻排序的两个计算节点之间的GPU利用率差值是否位于预设范围内，直到确定出部署待创建的新容器的计算节点为止。
若在第二排序结果中每个排序相邻的两个计算节点的GPU利用率的差值落入预设范围内，此时，系统可以按照各计算节点的CPU利用率数据从小到大的顺序，将除部署有目标容器的计算节点以外的其他计算节点重新进行排序，重新得到第二排序结果。
此时，针对任意相邻排序的两个计算节点，若确定该相邻排序的两个计算节点之间的CPU利用率的差值未落入预设范围内，则将该相邻排序的两个计算节点中CPU利用率较低的节点，作为部署待创建的新容器的计算节点。
以此类推，系统可以通过依次比较计算节点的GPU利用率、CPU利用率、内存使用率、存储设备的带宽等数据大小，直到确定出部署待创建的新容器的计算节点。
若系统依次比较计算节点的GPU利用率、CPU利用率、内存使用率、存储设备的带宽等数据大小后，仍然没有确定出部署待创建的新容器的计算节点，系统还可以采用其他方法确定出部署待创建的新容器的计算节点。
具体地，若系统确定第二排序结果中每个相邻排序的两个计算节点之间的负载差值位于预设范围内，则系统可以确定与待创建的新容器对应的子模型存在网络层依赖关系的子模型，作为关联子模型。例如，一个子模型的输出结果为另外一个子模型的输入，那么这两个子模型可以作为关联子模型。
同时，系统可以确定部署有关联子模型的计算节点，作为关联节点。系统可以测试关联节点与除所述关联节点以外的其他计算节点之间的网络延迟大小，再根据测试得到的网络延迟，从除所述关联节点以外的其他计算节点中确定出部署待创建的新容器的计算节点。
例如，系统可以将与关联节点之间的网络延迟最小的计算节点，作为部署待创建的新容器的计算节点。或者，系统可以确定关联节点与其他计算节点之间网络延迟的平均值，将与关联节点的网络延迟时间低于该平均值的其他计算节点，作为部署待创建的新容器的计算节点。
在确定出待创建的新容器的计算节点后，系统可以部署有子模型的计算节点中的容器所对应的运算时长相接近为调整目标，在目标节点中创建新容器，并对目标容器中部署的子模型的模型数据进行复制，以将复制得到的子模型部署在新容器中。
除此之外，系统还可以通过其他方式确定出目标节点。
具体地，系统可以先确定部署目标容器的计算节点，若确定部署目标容器的计算节点中还部署有指定容器，系统可以将部署目标容器的计算节点，作为需要调整容器 分布的计算节点，其中，指定容器中部署的子模型与目标容器中部署的子模型相同。
在确定出目标节点后，系统可以以部署有子模型的计算节点中的容器所对应的运算时长相接近为调整目标，将部署目标容器的计算节点中的目标容器或指定容器删除。
也就是说，若子模型模型参数相同的多个容器同时部署在一个物理节点(如相同物理机)上，系统可以以部署有子模型的计算节点中的容器所对应的运算时长相接近为调整目标，删除掉该物理节点上的容器，仅保留该物理节点上部署的子模型的模型参数相同的一个容器。
值得说明的是，本公开中对目标节点中各容器的分布进行调整时，系统都是以部署有子模型的计算节点中的容器所对应的运算时长相接近，且部署有容器的计算节点的负载相接近为调整目标，对目标节点中各容器的分布进行调整。
S114：基于调整容器分布后的各计算节点，执行所述目标模型的训练任务。
在系统对目标节点中各容器的分布进行调整后，基于调整容器分布后的各计算节点，系统可以继续采用样本数据，执行目标模型的训练任务。
值得说明的是，在对目标节点中的容器的分布进行调整之前，系统可以在当前所有容器上执行断点保存操作，将当前训练迭代次序的训练信息保存。
基于调整容器分布后的各计算节点，通过断点加载操作，系统可以获取到将之前保存的训练信息，再启动对所有容器中子模型的训练线程，继续对各个子模型进行训练。值得说明的是，新创建的容器中子模型的中间训练变量可以从与该子模型模型数据相同的其他容器中拷贝。
需要进行说明的是，本公开阐述的上述内容，是仅以系统为执行主体，对面向智能计算的分布式训练容器调度的方法进行说明的。但实际上，该系统可以由多个计算节点、分析器以及调度器构成。
图2为本公开提供一种系统关系示意图。
如图2所示，各个计算节点在对各个子模型进行分布式训练的过程中，各个计算节点会不断向共享存储系统中批量写入训练统计信息。
在对各个计算节点中包含的容器的分布进行调整之前，分析器可以从共享存储系统中读取到训练统计信息，以此来获取各计算节点在执行模型训练任务时的负载数据，以及针对每个容器，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长，作为该容器对应的运算时长。
分析器在根据各计算节点的负载数据以及各容器对应的运算时长，确定出需要调整容器分布的计算节点后，调度器可以调整各计算节点中容器的分布。
图3为本公开提供的容器调整示意图。
如图3所示，调度器可以以部署有子模型的各计算节点中的容器所对应的运算时长相接近为调整目标，对目标节点中各容器的分布进行调整。具体调整方法在S110至S112步骤中已进行了详细说明。
在调度器对容器分布进行调整后，基于调整容器分布后的各计算节点，各计算节点可以继续执行目标模型的训练任务。
从上述方法中可以看出，在执行模型训练任务时，先将目标模型拆分为多个子模型，再确定用于部署各个子模型的各计算节点，并在各计算节点上创建各容器，以将各子模型分别部署到所述各容器内，以通过各计算节点完成训练任务。在模型训练过程中，本公开监测各个计算节点的负载数据，以部署有子模型的各计算节点中的容器所对应的运算时长相接近为调整目标，对所述各计算节点中各容器的分布进行动态调整，有利于各个计算节点之间负载均衡，进一步地提高了模型训练的效率。
以上为本公开的一个或多个实施的方法，基于同样的思路，本公开还提供了相应的面向智能计算的分布式训练容器调度的装置，如图4所示。
图4为本公开提供的一种的面向智能计算的分布式训练容器调度的装置的示意图，包括：
第一获取模块400，用于获取样本数据以及目标模型；
拆分模块402，用于将所述目标模型进行拆分，以得到多个子模型，其中，所述多个子模型中每个子模型包含有所述目标模型中的部分网络层；
第一确定模块404，用于根据所述多个子模型，确定用于部署所述多个子模型的至少一个计算节点，并在所述至少一个计算节点上创建多个容器，以将所述多个子模型分别部署到所述多个容器内；
第一训练模块406，用于采用所述样本数据执行模型训练任务，以训练所述多个容器内的部署的所述多个子模型；
第二获取模块408，用于获取所述至少一个计算节点在执行模型训练任务时的负载数据，以及针对所述多个容器中每个容器，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长，作为该容器对应的运算时长；
第二确定模块410，用于根据所述至少一个计算节点的负载数据以及所述多个容器分别对应的运算时长，确定出需要调整容器分布的计算节点，作为目标节点；
调整模块412，用于以所述多个容器分别对应的运算时长相接近为调整目标，对所述目标节点中的容器的分布进行调整；
第二训练模块414，用于基于调整容器分布后的各计算节点，执行所述目标模型的训练任务。
可选地，所述拆分模块402具体用于确定在执行模型训练任务时所述目标模型的运算时长；根据所述目标模型的运算时长，将所述目标模型中包含的网络层进行拆分，以得到所述多个子模型。
可选地，所述第二获取模块408具体用于从预设的共享存储系统中确定出该容器所对应的训练统计信息；根据所述训练统计信息中包含的执行该容器内部署的子模型的训练任务的起始时间以及结束时间，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长；
其中，所述共享存储系统中存储的训练统计信息是基于所述至少一个计算节点中每个计算节点在执行模型训练任务时所产生的目标日志确定出的，所述目标日志是根据预设的指定关键词，从所述至少一个计算节点中每个计算节点所产生的日志中过滤出来的，所述训练统计信息是在累计到指定数量后，写入到所述共享存储系统中，并从所述至少一个计算节点中删除。
可选地，所述第二确定模块410具体用于按照所述多个容器分别对应的运算时长从大到小的顺序，对所述多个容器进行排序，得到第一排序结果；将所述第一排序结果中位于预设排位前的容器，作为目标容器；根据所述目标容器以及所述至少一个计算节点的负载数据，确定出需要调整容器分布的计算节点。
可选地，所述第二确定模块410具体用于确定部署所述目标容器的计算节点，作为第一节点；若根据所述第一节点的负载数据确定所述第一节点的负载高于第一设定阈值，则从其他计算节点中确定出用于部署所述第一节点中部分容器的计算节点，作为第二节点；将所述第一节点和所述第二节点确定为需要调整容器分布的计算节点。
可选地，所述第二确定模块410具体用于若确定所述目标容器对应的运算时长与其他容器对应的运算时间之间差值超过第二设定阈值，则根据所述至少一个计算节点的负载数据，确定部署待创建的新容器的计算节点，作为需要调整容器分布的计算节点；
所述调整模块412，具体用于以所述多个容器分别对应的运算时长相接近为调整目标，在所述目标节点中创建新容器，并对所述目标容器中部署的子模型的模型数据进行复制，以将复制得到的子模型部署在所述新容器中。
可选地，所述第二确定模块410具体用于按照所述至少一个计算节点的负载数据从小到大的顺序，将除部署有所述目标容器的计算节点以外的其他计算节点进行排序，得到第二排序结果；按照所述第二排序结果，依次判断相邻排序的两个计算节点之间的负载差值是否位于预设范围内；针对所述其他计算节点中任意相邻排序的两个计算节点，若确定该相邻排序的两个计算节点之间的负载差值未落入所述预设范围内，则将该相邻排序的两个计算节点中负载较低的节点，作为部署待创建的新容器的计算节点，否则，继续判断下一相邻排序的两个计算节点之间的负载差值是否位于预设范围内，直至遍历完所述第二排序结果中的所有计算节点或确定出部署待创建的新容器的计算节点为止。
可选地，所述第二确定模块410还用于，若确定所述第二排序结果中每个相邻排序的两个计算节点之间的负载差值位于所述预设范围内，则确定与所述待创建的新容器对应的子模型存在网络层依赖关系的子模型，作为关联子模型；确定部署有所述关联子模型的计算节点，作为关联节点；测试所述关联节点与除所述关联节点以外的其他计 算节点之间的网络延迟；根据测试得到的网络延迟，从除所述关联节点以外的其他计算节点中确定出部署待创建的新容器的计算节点。
可选地，所述第二确定模块410具体用于，确定部署所述目标容器的计算节点；若确定部署所述目标容器的计算节点中还部署有指定容器，则将部署所述目标容器的计算节点，作为需要调整容器分布的计算节点，其中，所述指定容器中部署的子模型与所述目标容器中部署的子模型相同；
所述调整模块412，具体用于以所述多个容器分别对应的运算时长相接近为调整目标，将部署所述目标容器的计算节点中的所述目标容器或所述指定容器删除。
可选地，所述调整模块412具体用于，以所述多个容器分别对应的运算时长相接近，且所述至少一个计算节点分别的负载相接近为调整目标，对所述目标节点中的容器的分布进行调整。
本公开还提供了一种计算机可读存储介质，该存储介质存储有计算机程序，计算机程序可用于执行上述图1提供的一种面向智能计算的分布式训练容器调度的方法。
本公开还提供了图5所示的一种对应于图1的电子设备的示意结构图。如图5所示，在硬件层面，该电子设备包括处理器、内部总线、网络接口、内存以及非易失性存储器，当然还可能包括其他业务所需要的硬件。处理器从非易失性存储器中读取对应的计算机程序到内存中然后运行，以实现上述图1所述的面向智能计算的分布式训练容器调度的方法。
当然，除了软件实现方式之外，本公开并不排除其他实现方式，比如逻辑器件抑或软硬件结合的方式等等，也就是说以下处理流程的执行主体并不限定于各个逻辑单元，也可以是硬件或逻辑器件。
在20世纪90年代，对于一个技术的改进可以很明显地区分是硬件上的改进(例如，对二极管、晶体管、开关等电路结构的改进)还是软件上的改进(对于方法流程的改进)。然而，随着技术的发展，当今的很多方法流程的改进已经可以视为硬件电路结构的直接改进。设计人员几乎都通过将改进的方法流程编程到硬件电路中来得到相应的硬件电路结构。因此，不能说一个方法流程的改进就不能用硬件实体模块来实现。例如，可编程逻辑器件(Programmable Logic Device，PLD)(例如现场可编程门阵列(Field Programmable Gate Array，FPGA))就是这样一种集成电路，其逻辑功能由用户对器件编程来确定。由设计人员自行编程来把一个数字系统“集成”在一片PLD上，而不需要请芯片制造厂商来设计和制作专用的集成电路芯片。而且，如今，取代手工地制作集成电路芯片，这种编程也多半改用“逻辑编译器(logic compiler)”软件来实现，它与程序开发撰写时所用的软件编译器相类似，而要编译之前的原始代码也得用特定的编程语言来撰写，此称之为硬件描述语言(Hardware Description Language，HDL)，而HDL也并非仅有一种，而是有许多种，如ABEL(Advanced Boolean Expression Language)、AHDL(Altera Hardware Description Language)、Confluence、CUPL(Cornell University ProgrammingLanguage)、HDCal、JHDL(Java Hardware Description Language)、Lava、Lola、MyHDL、PALASM、RHDL(Ruby Hardware Description Language)等，目前最普遍使用的是VHDL(Very-High-Speed Integrated Circuit Hardware Description Language)与Verilog。本领域技术人员也应该清楚，只需要将方法流程用上述几种硬件描述语言稍作逻辑编程并编程到集成电路中，就可以很容易得到实现该逻辑方法流程的硬件电路。
控制器可以按任何适当的方式实现，例如，控制器可以采取例如微处理器或处理器以及存储可由该(微)处理器执行的计算机可读程序代码(例如软件或固件)的计算机可读介质、逻辑门、开关、专用集成电路(Application Specific Integrated Circuit，ASIC)、可编程逻辑控制器和嵌入微控制器的形式，控制器的例子包括但不限于以下微控制器：ARC 625D、Atmel AT91SAM、Microchip PIC18F26K20以及Silicone Labs C8051F320，存储器控制器还可以被实现为存储器的控制逻辑的一部分。本领域技术人员也知道，除了以纯计算机可读程序代码方式实现控制器以外，完全可以通过将方法步骤进行逻辑编程来使得控制器以逻辑门、开关、专用集成电路、可编程逻辑控制器和嵌入微控制器等的形式来实现相同功能。因此这种控制器可以被认为是一种硬件部件，而对其内包括的用于实现各种功能的装置也可以视为硬件部件内的结构。或者甚至，可以将用于实现各种功能的装置视为既可以是实现方法的软件模块又可以是硬件部件内的结构。
上述实施例阐明的系统、装置、模块或单元，具体可以由计算机芯片或实体实现，或者由具有某种功能的产品来实现。一种典型的实现设备为计算机。具体的，计算机例如可以为个人计算机、膝上型计算机、蜂窝电话、相机电话、智能电话、个人数字助理、媒体播放器、导航设备、电子邮件设备、游戏控制台、平板计算机、可穿戴设备 或者这些设备中的任何设备的组合。
为了描述的方便，描述以上装置时以功能分为各种单元分别描述。当然，在实施本公开时可以把各单元的功能在同一个或多个软件和/或硬件中实现。
本领域内的技术人员应明白，本公开的实施例可提供为方法、系统、或计算机程序产品。因此，本公开可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本公开可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本公开是参照根据本公开实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
在一个典型的配置中，计算设备包括一个或多个处理器(CPU)、输入/输出接口、网络接口和内存。
内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。内存是计算机可读介质的示例。
计算机可读介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁带磁磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。按照本文中的界定，计算机可读介质不包括暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
还需要说明的是，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、商品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、商品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个......”限定的要素，并不排除在包括所述要素的过程、方法、商品或者设备中还存在另外的相同要素。
本领域技术人员应明白，本公开的实施例可提供为方法、系统或计算机程序产品。因此，本公开可采用完全硬件实施例、完全软件实施例或结合软件和硬件方面的实施例的形式。而且，本公开可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本公开可以在由计算机执行的计算机可执行指令的一般上下文中描述，例如程序模块。一般地，程序模块包括执行特定任务或实现特定抽象数据类型的例程、程序、对象、组件、数据结构等等。也可以在分布式计算环境中实践本公开，在这些分布式计算环境中，由通过通信网络而被连接的远程处理设备来执行任务。在分布式计算环境中，程序模块可以位于包括存储设备在内的本地和远程计算机存储介质中。
本公开中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于系 统实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
以上所述仅为本公开的实施例而已，并不用于限制本公开。对于本领域技术人员来说，本公开可以有各种更改和变化。凡在本公开的精神和原理之内所作的任何修改、等同替换、改进等，均应包含在本公开的权利要求范围之内。

Claims (22)

  1. 一种面向智能计算的分布式训练容器调度的方法，其特征在于，包括：
    获取样本数据以及目标模型；
    将所述目标模型进行拆分，以得到多个子模型，其中，所述多个子模型中每个子模型包含有所述目标模型中的部分网络层；
    根据所述多个子模型，确定用于部署所述多个子模型的至少一个计算节点，并在所述至少一个计算节点上创建多个容器，以将所述多个子模型分别部署到所述多个容器内；
    采用所述样本数据执行模型训练任务，以训练所述多个容器内的部署的所述多个子模型；
    获取所述至少一个计算节点在执行模型训练任务时的负载数据；
    针对所述多个容器中每个容器，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长，作为该容器对应的运算时长；
    根据所述至少一个计算节点的负载数据以及所述多个容器分别对应的运算时长，确定出需要调整容器分布的计算节点，作为目标节点；
    以所述多个容器分别对应的运算时长相接近为调整目标，对所述目标节点中的容器的分布进行调整；
    基于调整容器分布后的各计算节点，执行所述目标模型的训练任务。
  2. 如权利要求1所述的方法，其特征在于，将所述目标模型进行拆分，以得到所述多个子模型，包括：
    确定在执行模型训练任务时所述目标模型的运算时长；
    根据所述目标模型的运算时长，将所述目标模型中包含的网络层进行拆分，以得到所述多个子模型。
  3. 如权利要求1所述的方法，其特征在于，针对所述多个容器中每个容器，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长，包括：
    从预设的共享存储系统中确定出该容器所对应的训练统计信息；
    根据所述训练统计信息中包含的执行该容器内部署的子模型的训练任务的起始时间以及结束时间，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长；
    其中，所述共享存储系统中存储的训练统计信息是基于所述至少一个计算节点中每个计算节点在执行模型训练任务时所产生的目标日志确定出的，所述目标日志是根据预设的指定关键词，从所述至少一个计算节点中每个计算节点所产生的日志中过滤出来的，所述训练统计信息是在累计到指定数量后，写入到所述共享存储系统中，并从所述至少一个计算节点中删除。
  4. 如权利要求1所述的方法，其特征在于，根据所述至少一个计算节点的负载数据以及所述多个容器分别对应的运算时长，确定出需要调整容器分布的计算节点，包括：
    按照所述多个容器分别对应的运算时长从大到小的顺序，对所述多个容器进行排序，得到第一排序结果；
    将所述第一排序结果中位于预设排位前的容器，作为目标容器；
    根据所述目标容器以及所述至少一个计算节点的负载数据，确定出需要调整容器分布的计算节点。
  5. 如权利要求4所述的方法，其特征在于，根据所述目标容器以及所述至少一个计算节点的负载数据，确定出需要调整容器分布的计算节点，包括：
    确定部署所述目标容器的计算节点，作为第一节点；
    若根据所述第一节点的负载数据确定所述第一节点的负载高于第一设定阈值，则从其他计算节点中确定出用于部署所述第一节点中部分容器的计算节点，作为第二节点；
    将所述第一节点和所述第二节点确定为需要调整容器分布的计算节点。
  6. 如权利要求4所述的方法，其特征在于，根据所述目标容器以及所述至少一个计算节点的负载数据，确定出需要调整容器分布的计算节点，包括：
    若确定所述目标容器对应的运算时长与其他容器对应的运算时间之间差值超过第二设定阈值，则根据所述至少一个计算节点的负载数据，确定部署待创建的新容器的计算节点，作为需要调整容器分布的计算节点；
    以所述多个容器分别对应的运算时长相接近为调整目标，对所述目标节点中的容器的分布进行调整，包括：
    以所述多个容器分别对应的运算时长相接近为调整目标，在所述目标节点中创建新容器，并对所述目标容器中部署的子模型的模型数据进行复制，以将复制得到的子模型部署在所述新容器中。
  7. 如权利要求6所述的方法，其特征在于，根据所述至少一个计算节点的负载数据，确定部署待创建的新容器的计算节点，作为需要调整容器分布的计算节点，包括：
    按照所述至少一个计算节点的负载数据从小到大的顺序，将除部署有所述目标容器的计算节点以外的其他计算节点进行排序，得到第二排序结果；
    按照所述第二排序结果，依次判断相邻排序的两个计算节点之间的负载差值是否位 于预设范围内；
    针对所述其他计算节点中任意相邻排序的两个计算节点，若确定该相邻排序的两个计算节点之间的负载差值未落入所述预设范围内，则将该相邻排序的两个计算节点中负载较低的节点，作为部署待创建的新容器的计算节点，否则，继续判断下一相邻排序的两个计算节点之间的负载差值是否位于预设范围内，直至遍历完所述第二排序结果中的所有计算节点或确定出部署待创建的新容器的计算节点为止。
  8. 如权利要求7所述的方法，其特征在于，所述方法还包括：
    若确定所述第二排序结果中每个相邻排序的两个计算节点之间的负载差值位于所述预设范围内，则确定与所述待创建的新容器对应的子模型存在网络层依赖关系的子模型，作为关联子模型；
    确定部署有所述关联子模型的计算节点，作为关联节点；
    测试所述关联节点与除所述关联节点以外的其他计算节点之间的网络延迟；
    根据测试得到的网络延迟，从除所述关联节点以外的其他计算节点中确定出部署待创建的新容器的计算节点。
  9. 如权利要求4所述的方法，其特征在于，根据所述目标容器以及所述至少一个计算节点的负载数据，确定出需要调整容器分布的计算节点，包括：
    确定部署所述目标容器的计算节点；
    若确定部署所述目标容器的计算节点中还部署有指定容器，则将部署所述目标容器的计算节点，作为需要调整容器分布的计算节点，其中，所述指定容器中部署的子模型与所述目标容器中部署的子模型相同；
    以所述多个容器分别对应的运算时长相接近为调整目标，对所述目标节点中的容器的分布进行调整，包括：
    以所述多个容器分别对应的运算时长相接近为调整目标，将部署所述目标容器的计算节点中的所述目标容器或所述指定容器删除。
  10. 如权利要求1～9任一项所述的方法，其特征在于，以所述多个容器分别对应的运算时长相接近为调整目标，对所述目标节点中的容器的分布进行调整，包括：
    以所述多个容器分别对应的运算时长相接近，且所述至少一个计算节点分别的负载相接近为调整目标，对所述目标节点中的容器的分布进行调整。
  11. 一种面向智能计算的分布式训练容器调度的装置，其特征在于，包括：
    第一获取模块，用于获取样本数据以及目标模型；
    拆分模块，用于将所述目标模型进行拆分，以得到多个子模型，其中，所述多个子模型中每个子模型包含有所述目标模型中的部分网络层；
    第一确定模块，用于根据所述多个子模型，确定用于部署所述多个子模型的至少一个计算节点，并在所述至少一个计算节点上创建多个容器，以将所述多个子模型分别部署到所述多个容器内；
    第一训练模块，用于采用所述样本数据执行模型训练任务，以训练所述多个容器内的部署的所述多个子模型；
    第二获取模块，用于获取所述至少一个计算节点在执行模型训练任务时的负载数据，以及针对所述多个容器中每个容器，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长，作为该容器对应的运算时长；
    第二确定模块，用于根据所述至少一个计算节点的负载数据以及所述多个容器分别对应的运算时长，确定出需要调整容器分布的计算节点，作为目标节点；
    调整模块，用于以所述多个容器分别对应的运算时长相接近为调整目标，对所述目标节点中的容器的分布进行调整；
    第二训练模块，用于基于调整容器分布后的各计算节点，执行所述目标模型的训练任务。
  12. 如权利要求11所述的装置，其特征在于，所述拆分模块具体用于确定在执行模型训练任务时所述目标模型的运算时长；根据所述目标模型的运算时长，将所述目标模型中包含的网络层进行拆分，以得到所述多个子模型。
  13. 如权利要求11所述的装置，其特征在于，所述第二获取模块具体用于从预设的共享存储系统中确定出该容器所对应的训练统计信息；根据所述训练统计信息中包含的执行该容器内部署的子模型的训练任务的起始时间以及结束时间，确定在执行该容器内部署的子模型的训练任务时该子模型的运算时长；
    其中，所述共享存储系统中存储的训练统计信息是基于所述至少一个计算节点中每个计算节点在执行模型训练任务时所产生的目标日志确定出的，所述目标日志是根据预设的指定关键词，从所述至少一个计算节点中每个计算节点所产生的日志中过滤出来的，所述训练统计信息是在累计到指定数量后，写入到所述共享存储系统中，并从所述至少一个计算节点中删除。
  14. 如权利要求11所述的装置，其特征在于，所述第二确定模块具体用于按照所述多个容器分别对应的运算时长从大到小的顺序，对所述多个容器进行排序，得到第一 排序结果；将所述第一排序结果中位于预设排位前的容器，作为目标容器；根据所述目标容器以及所述至少一个计算节点的负载数据，确定出需要调整容器分布的计算节点。
  15. 如权利要求14所述的装置，其特征在于，所述第二确定模块具体用于确定部署所述目标容器的计算节点，作为第一节点；若根据所述第一节点的负载数据确定所述第一节点的负载高于第一设定阈值，则从其他计算节点中确定出用于部署所述第一节点中部分容器的计算节点，作为第二节点；将所述第一节点和所述第二节点确定为需要调整容器分布的计算节点。
  16. 如权利要求14所述的装置，其特征在于，所述第二确定模块具体用于若确定所述目标容器对应的运算时长与其他容器对应的运算时间之间差值超过第二设定阈值，则根据所述至少一个计算节点的负载数据，确定部署待创建的新容器的计算节点，作为需要调整容器分布的计算节点；
    所述调整模块，具体用于以所述多个容器分别对应的运算时长相接近为调整目标，在所述目标节点中创建新容器，并对所述目标容器中部署的子模型的模型数据进行复制，以将复制得到的子模型部署在所述新容器中。
  17. 如权利要求16所述的装置，其特征在于，所述第二确定模块具体用于按照所述至少一个计算节点的负载数据从小到大的顺序，将除部署有所述目标容器的计算节点以外的其他计算节点进行排序，得到第二排序结果；按照所述第二排序结果，依次判断相邻排序的两个计算节点之间的负载差值是否位于预设范围内；针对所述其他计算节点中任意相邻排序的两个计算节点，若确定该相邻排序的两个计算节点之间的负载差值未落入所述预设范围内，则将该相邻排序的两个计算节点中负载较低的节点，作为部署待创建的新容器的计算节点，否则，继续判断下一相邻排序的两个计算节点之间的负载差值是否位于预设范围内，直至遍历完所述第二排序结果中的所有计算节点或确定出部署待创建的新容器的计算节点为止。
  18. 如权利要求17所述的装置，其特征在于，所述第二确定模块还用于，若确定所述第二排序结果中每个相邻排序的两个计算节点之间的负载差值位于所述预设范围内，则确定与所述待创建的新容器对应的子模型存在网络层依赖关系的子模型，作为关联子模型；确定部署有所述关联子模型的计算节点，作为关联节点；测试所述关联节点与除所述关联节点以外的其他计算节点之间的网络延迟；根据测试得到的网络延迟，从除所述关联节点以外的其他计算节点中确定出部署待创建的新容器的计算节点。
  19. 如权利要求14所述的装置，其特征在于，所述第二确定模块具体用于，确定部署所述目标容器的计算节点；若确定部署所述目标容器的计算节点中还部署有指定容器，则将部署所述目标容器的计算节点，作为需要调整容器分布的计算节点，其中，所述指定容器中部署的子模型与所述目标容器中部署的子模型相同；
    所述调整模块，具体用于以所述多个容器分别对应的运算时长相接近为调整目标，将部署所述目标容器的计算节点中的所述目标容器或所述指定容器删除。
  20. 如权利要求11～19任一项所述的装置，其特征在于，所述调整模块具体用于，以所述多个容器分别对应的运算时长相接近，且所述至少一个计算节点分别的负载相接近为调整目标，对所述目标节点中的容器的分布进行调整。
  21. 一种计算机可读存储介质，其特征在于，所述存储介质存储有计算机程序，所述计算机程序被处理器执行时实现上述权利要求1～10任一项所述的方法。
  22. 一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，其特征在于，所述处理器执行所述程序时实现上述权利要求1～10任一项所述的方法。
PCT/CN2023/101093 2023-04-26 2023-06-19 面向智能计算的分布式训练容器调度 WO2024007849A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2023547139A JP2024529206A (ja) 2023-04-26 2023-06-19 インテリジェントコンピューティング向けのコンテナスケジューリングのための分散トレーニング

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310461389.9 2023-04-26
CN202310461389.9A CN116167463B (zh) 2023-04-26 2023-04-26 一种面向智能计算的分布式模型训练容器调度方法及装置

Publications (1)

Publication Number Publication Date
WO2024007849A1 true WO2024007849A1 (zh) 2024-01-11

Family

ID=86414952

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/101093 WO2024007849A1 (zh) 2023-04-26 2023-06-19 面向智能计算的分布式训练容器调度

Country Status (3)

Country Link
JP (1) JP2024529206A (zh)
CN (1) CN116167463B (zh)
WO (1) WO2024007849A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117724823A (zh) * 2024-02-07 2024-03-19 之江实验室 基于声明式语义的多模型工作流描述的任务执行方法
CN118378726A (zh) * 2024-06-25 2024-07-23 之江实验室 一种模型训练系统、方法、存储介质及电子设备
CN118567791A (zh) * 2024-07-31 2024-08-30 之江实验室 一种缓存方法、装置、存储介质以及电子设备

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116167463B (zh) * 2023-04-26 2023-07-07 之江实验室 一种面向智能计算的分布式模型训练容器调度方法及装置
CN116382599B (zh) * 2023-06-07 2023-08-29 之江实验室 一种面向分布式集群的任务执行方法、装置、介质及设备
CN116755941B (zh) * 2023-08-21 2024-01-09 之江实验室 一种节点故障感知的分布式模型训练的方法及装置
CN117035123B (zh) * 2023-10-09 2024-01-09 之江实验室 一种并行训练中的节点通信方法、存储介质、设备

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113110914A (zh) * 2021-03-02 2021-07-13 西安电子科技大学 一种基于微服务架构的物联网平台构建方法
CN114091536A (zh) * 2021-11-19 2022-02-25 上海梦象智能科技有限公司 一种基于变分自编码器的负荷分解方法
US20220207716A1 (en) * 2019-06-18 2022-06-30 Tetra Laval Holdings & Finance S.A. Detection of deviations in packaging containers for liquid food
CN114787833A (zh) * 2019-09-23 2022-07-22 普雷萨根私人有限公司 分散式人工智能(ai)/机器学习训练系统
CN115248728A (zh) * 2022-09-21 2022-10-28 之江实验室 面向智能计算的分布式训练任务调度方法、系统和装置
CN115827253A (zh) * 2023-02-06 2023-03-21 青软创新科技集团股份有限公司 一种芯片资源算力分配方法、装置、设备及存储介质
CN116167463A (zh) * 2023-04-26 2023-05-26 之江实验室 一种模型训练的方法、装置、存储介质及电子设备

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109559734B (zh) * 2018-12-18 2022-02-18 百度在线网络技术（北京）有限公司 声学模型训练的加速方法和装置
CN111563584B (zh) * 2019-02-14 2022-12-09 上海寒武纪信息科技有限公司 一种神经网络模型的拆分方法及相关产品
CN111723900B (zh) * 2019-03-18 2023-10-20 北京灵汐科技有限公司 一种基于众核处理器的神经网络的映射方法及计算设备
CN110413391B (zh) * 2019-07-24 2022-02-25 上海交通大学 基于容器集群的深度学习任务服务质量保证方法和系统
CN111752713B (zh) * 2020-06-28 2022-08-05 浪潮电子信息产业股份有限公司 模型并行训练任务负载均衡方法、装置、设备及存储介质
CN112308205A (zh) * 2020-06-28 2021-02-02 北京沃东天骏信息技术有限公司 基于预训练模型的模型改进方法及装置
CN113011483B (zh) * 2021-03-11 2022-08-05 北京三快在线科技有限公司 一种模型训练和业务处理的方法及装置
CN113220457B (zh) * 2021-05-24 2024-03-22 深圳市智芯华玺信息技术有限公司 模型部署方法、模型部署装置、终端设备及可读存储介质
CN113723443A (zh) * 2021-07-12 2021-11-30 鹏城实验室 一种视觉大模型分布式训练方法及系统
CN113992525B (zh) * 2021-10-12 2024-06-21 支付宝(杭州)信息技术有限公司 一种应用的容器数量调节方法及装置
CN114780225B (zh) * 2022-06-14 2022-09-23 支付宝(杭州)信息技术有限公司 一种分布式模型训练系统、方法及装置
CN116011587A (zh) * 2022-12-30 2023-04-25 支付宝(杭州)信息技术有限公司 一种模型训练方法、装置、存储介质及电子设备

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220207716A1 (en) * 2019-06-18 2022-06-30 Tetra Laval Holdings & Finance S.A. Detection of deviations in packaging containers for liquid food
CN114787833A (zh) * 2019-09-23 2022-07-22 普雷萨根私人有限公司 分散式人工智能(ai)/机器学习训练系统
CN113110914A (zh) * 2021-03-02 2021-07-13 西安电子科技大学 一种基于微服务架构的物联网平台构建方法
CN114091536A (zh) * 2021-11-19 2022-02-25 上海梦象智能科技有限公司 一种基于变分自编码器的负荷分解方法
CN115248728A (zh) * 2022-09-21 2022-10-28 之江实验室 面向智能计算的分布式训练任务调度方法、系统和装置
CN115827253A (zh) * 2023-02-06 2023-03-21 青软创新科技集团股份有限公司 一种芯片资源算力分配方法、装置、设备及存储介质
CN116167463A (zh) * 2023-04-26 2023-05-26 之江实验室 一种模型训练的方法、装置、存储介质及电子设备

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117724823A (zh) * 2024-02-07 2024-03-19 之江实验室 基于声明式语义的多模型工作流描述的任务执行方法
CN118378726A (zh) * 2024-06-25 2024-07-23 之江实验室 一种模型训练系统、方法、存储介质及电子设备
CN118567791A (zh) * 2024-07-31 2024-08-30 之江实验室 一种缓存方法、装置、存储介质以及电子设备

Also Published As

Publication number Publication date
CN116167463A (zh) 2023-05-26
CN116167463B (zh) 2023-07-07
JP2024529206A (ja) 2024-08-06

Similar Documents

Publication Publication Date Title
WO2024007849A1 (zh) 面向智能计算的分布式训练容器调度
WO2020000944A1 (zh) 基于抢占式调度的资源共享使用方法、系统及设备
CN113918351B (zh) 深度学习框架与ai加速卡片内分布式训练适配方法和装置
CN111324427B (zh) 一种基于dsp的任务调度方法及装置
WO2016205978A1 (en) Techniques for virtual machine migration
CN107273200B (zh) 一种针对异构存储的任务调度方法
CN109656779A (zh) 内存监控方法、装置、终端和存储介质
CN109992366B (zh) 任务调度方法及调度装置
CN107562532B (zh) 一种预测设备集群的硬件资源利用率的方法及装置
CN110389842A (zh) 一种动态资源分配方法、装置、存储介质及设备
CN104182278A (zh) 一种判定计算机硬件资源繁忙程度的方法和装置
CN111381961A (zh) 一种处理定时任务的方法、装置与电子设备
CN107402863B (zh) 一种用于通过日志系统处理业务系统的日志的方法与设备
WO2024187737A1 (zh) 一种数据处理的方法、装置、存储介质及电子设备
CN113010312A (zh) 一种超参数调优方法、装置及存储介质
US20220076102A1 (en) Method and apparatus for managing neural network models
CN109165201B (zh) 日志的归并方法及终端设备
CN111176637A (zh) 基于缓存抢占延迟约束下aadl模型的可调度性分析方法
US9659041B2 (en) Model for capturing audit trail data with reduced probability of loss of critical data
CN110765082B (zh) Hadoop文件处理方法、装置、存储介质及服务器
Kalnauz et al. Productivity estimation of serverless computing
Benini et al. Multi-stage Benders decomposition for optimizing multicore architectures
CN113360527B (zh) 缓冲区尺寸处理方法、装置、计算机设备
CN110008382A (zh) 一种确定TopN数据的方法、系统及设备
CN110059712A (zh) 异常数据的检测方法及装置

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2023547139

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23834623

Country of ref document: EP

Kind code of ref document: A1