CN107045456A - A kind of resource allocation methods and explorer - Google Patents

Abstract

Embodiments of the present invention provide a resource allocation method and a resource manager, which are used to improve resource utilization, and/or, to improve execution efficiency of user jobs. The method includes: receiving a job submitted by a client device, and decomposing the job into a plurality of tasks, wherein each task in the plurality of tasks is configured with a corresponding resource requirement; and estimating the running time of each task ; According to the resource demand and running time corresponding to each task, combined with the preset scheduling strategy, determine the first allocation configuration of the multiple tasks, and the first allocation configuration is used to indicate that the multiple tasks are in multiple computing The distribution of the operable computing nodes in the nodes, the scheduling strategy includes at least one of the resource utilization priority strategy and the efficiency priority strategy; the plurality of tasks can be assigned to the plurality of tasks according to the first allocation configuration run on a compute node. The invention is applicable to the field of high-performance clusters.

Description

Resource allocation method and resource manager

technical field

The invention relates to the field of high-performance clusters, in particular to a resource allocation method and a resource manager.

Background technique

The rapid development of the Internet has produced a large amount of user data, and distributed processing is a standard method for processing large-scale data sets. Its typical mode is to decompose a user job (English: Job) into a series of tasks (English: Task) that can be distributed and run, and schedule these tasks to appropriate nodes (English: Scheduler) through the scheduler (English: Scheduler) node) to operate on. After the task operation is completed, the operation results of the task are collected and organized to form the final result output of the job.

The scheduler is the coupling point between cluster resources and user jobs. The quality of the scheduling strategy directly affects the resource utilization of the entire cluster and the execution efficiency of user jobs. The scheduling strategy of the widely used Hadoop system is shown in Figure 1. Among them, Hadoop queues the tasks with resource requirements according to a certain strategy, such as the dominant resource fairness (English full name: dominantresource fairness, English abbreviation: DRF) strategy), and each node reports the amount of resources on the node through the heartbeat, and triggers the allocation mechanism. If the amount of resources on the node meets the requirements of the first Task, the scheduler places the Task on the node. However, this scheduling strategy only considers the fairness of resources, which is relatively simple, and cannot flexibly select the resource utilization priority strategy and efficiency priority strategy for resource allocation according to different scenarios, so that it cannot make the utilization of cluster resources higher. And/or, the execution efficiency of the user job is high.

Contents of the invention

Embodiments of the present invention provide a resource allocation method and a resource manager, which are used to flexibly select a resource utilization priority strategy and an efficiency priority strategy for resource allocation, thereby improving resource utilization, and/or improving user job execution efficiency.

In order to achieve the above purpose, the embodiments of the present invention provide the following technical solutions:

In a first aspect, a resource allocation method in a distributed computing system is provided, the distributed computing system includes multiple computing nodes, the method includes: receiving a job submitted by a client device, and decomposing the job into multiple tasks, Wherein, each task in the plurality of tasks is configured with a corresponding resource demand; the running time of each task is estimated; according to the resource demand and running time corresponding to each task, combined with a preset scheduling strategy, determine The first allocation configuration of the plurality of tasks, the first allocation configuration is used to indicate the distribution of the plurality of tasks on the runnable computing nodes among the plurality of computing nodes, the scheduling strategy includes resource utilization priority strategy and At least one of the efficiency priority strategies: assigning the plurality of tasks to computing nodes that can run the plurality of tasks according to the first allocation configuration.

Based on the resource allocation method provided by the embodiment of the present invention, in the resource allocation method, after receiving the job submitted by the client device and decomposing the job into a plurality of tasks with corresponding resource requirement configurations, each task is further estimated The running time of each task, and according to the resource requirements and running time of each task, combined with the preset scheduling strategy, determine the first allocation configuration of the multiple tasks, and then allocate the multiple tasks according to the first allocation configuration to the runnable compute nodes for the multiple tasks. Wherein, the first allocation configuration is used to indicate the distribution of the multiple tasks on the computing nodes that can run the multiple tasks, and the scheduling strategy includes at least one of resource utilization priority strategy and efficiency priority strategy. That is to say, the scheme takes into account the running time of each task, and when scheduling tasks with fixed space requirements (that is, the resource requirements of the task) and time requirements (that is, the time of the task) to the corresponding nodes, it can According to the corresponding scheduling policy, the resource utilization priority strategy and the efficiency priority strategy are flexibly selected for resource allocation, so that an allocation configuration with higher resource utilization rate and/or higher efficiency is finally adopted. On the one hand, since the allocation configuration with high resource utilization can be adopted, that is, the task combination with high resource utilization of those nodes can be scheduled to the nodes through the scheduling strategy, so this allocation scheme can effectively reduce the resource utilization in the prior art. The problem of fragmentation, thereby improving the resource utilization of the cluster. On the other hand, since a more efficient allocation configuration can be adopted, that is, the task combination with the shortest execution time of the job can be scheduled to the node through the scheduling strategy, so compared with the existing technology, this allocation scheme can significantly shorten the job Execution time, improve job execution efficiency. To sum up, the resource allocation method provided by the embodiment of the present invention can flexibly select the resource utilization rate priority policy and the efficiency priority policy for resource allocation according to the corresponding scheduling policy, thereby improving the resource utilization rate, and/or, improving the efficiency of user jobs. effectiveness.

With reference to the first aspect, in the first possible implementation manner of the first aspect, if the scheduling policy is a resource utilization priority policy, the first allocation configuration is specifically such that each of the multiple task-running computing nodes The allocation configuration that maximizes the single-node resource utilization of computing nodes.

With reference to the first aspect, in the second possible implementation manner of the first aspect, if the scheduling strategy is an efficiency-first strategy, the first allocation configuration is specifically the allocation configuration that enables the fastest overall job execution speed.

With reference to the first aspect or the first possible implementation of the first aspect or the second possible implementation of the first aspect, in the third possible implementation of the first aspect, the estimation of the running time of each task, Specifically, it may include: for each task, process according to the following operations for the first task: match the hard information of the first task with the hard information of the historical tasks in the sample library; The hard information matches the historical running times of historical tasks to estimate the running time of the first task.

Specifically, the hard information in this embodiment of the present invention may specifically include job type, execution user and other information.

It should be noted that the embodiment of the present invention is only an example of a specific implementation of estimating the running time of a task. Of course, the running time of a task may also be estimated in other ways, for example, by pre-running the task. That is, by pre-running a small number of job instances to obtain an accurate estimate of the full runtime. Also, the run times of subsequent tasks of the same job will be more accurately estimated by reference to the run times of tasks that have already run. The embodiment of the present invention does not limit the specific implementation manner of estimating the running time of the task.

In combination with any possible implementation of the first aspect to the third possible implementation of the first aspect, in the fourth possible implementation of the first aspect, according to the resource demand and running Time, before determining the first allocation configuration of the plurality of tasks in combination with the preset scheduling strategy, further includes: classifying the plurality of tasks according to resource types to obtain at least one type of task;

The determining the first allocation configuration of the plurality of tasks according to the resource demand and running time corresponding to each task in combination with a preset scheduling strategy specifically includes: for each type of the at least one type of task Tasks are processed according to the following operations for the first type of tasks: according to the resource demand and running time corresponding to each task in the first type of tasks, combined with the preset scheduling strategy, determine the sub-allocation configuration of the first type of tasks , the sub-allocation configuration is used to indicate the distribution of the first type of tasks on the runnable computing nodes among the plurality of computing nodes; the combination of sub-allocation configurations of each type of task in the at least one type of tasks is determined as A first allocation configuration of the plurality of tasks.

Since the resource allocation method provided by the embodiment of the present invention can first classify multiple tasks according to the types of resources, and then perform resource allocation for each type of task separately, that is to say, it can simultaneously consider the requirements for heterogeneous clusters and jobs with special resource requirements. Therefore, it has wider applicability and better comprehensive performance.

Optionally, take into account that the running time estimates will usually have some deviation from the actual situation. If these deviations are not controlled, as time goes on, the pre-allocation results of job resources may be more and more different from the ideal results. Therefore, in the resource allocation method provided by the embodiment of the present invention, a mutation mechanism (that is, reallocation) may also be introduced. which is:

With reference to any one of the possible implementation manners of the first aspect to the fourth possible implementation manner of the first aspect, in the fifth possible implementation manner of the first aspect, the multiple tasks are allocated according to the first After the configuration is allocated to the operational computing nodes of the plurality of tasks, it also includes: according to the first allocation configuration, determining the first overall allocation objective function value when all tasks in the waiting state are running on the allocated nodes; According to the resource requirements and running time corresponding to all tasks in the waiting state, combined with the preset scheduling policy, determine the second allocation configuration of all tasks in the waiting state, and the second allocation configuration is used to indicate that all tasks in the waiting state The distribution of tasks in the waiting state on the runnable computing nodes of all the tasks in the waiting state; according to the second allocation configuration, determine the second overall allocation target when all the tasks in the waiting state run on the assigned nodes function value; if the second overall allocation objective function value is greater than the first overall allocation objective function value, all the tasks in the waiting state are allocated to the runnable computing nodes of all the tasks in the waiting state according to the second allocation configuration.

Through the above mutation mechanism, the pre-allocation results of job resources can be made to evolve in a better direction.

In a second aspect, a resource manager is provided, and the resource manager includes: a receiving unit, a decomposing unit, an estimating unit, a determining unit, and an allocating unit: the receiving unit is used to receive a job submitted by a client device; the decomposing unit is used to The job is decomposed into multiple tasks, wherein each task in the multiple tasks is configured with a corresponding resource requirement; an estimation unit is used to estimate the running time of each task; a determination unit is used to The resource demand and running time corresponding to each task, combined with the preset scheduling strategy, determine the first allocation configuration of the multiple tasks, and the first allocation configuration is used to indicate the allocation of the multiple tasks in the multiple computing nodes. The distribution situation on the operable computing node, the scheduling strategy includes at least one of the resource utilization priority strategy and the efficiency priority strategy; an allocation unit is used to allocate the multiple tasks to the multiple tasks according to the first allocation configuration on a runnable compute node.

Based on the resource manager provided by the embodiment of the present invention, after the resource manager receives the job submitted by the client device and decomposes the job into a plurality of tasks with corresponding resource requirement configurations, it also estimates the operation of each task. Time, and according to the resource demand and running time of each task, combined with the preset scheduling strategy, determine the first allocation configuration of the multiple tasks, and then allocate the multiple tasks to the Compute nodes that can run multiple tasks. Wherein, the first allocation configuration is used to indicate the distribution of the multiple tasks on the computing nodes that can run the multiple tasks, and the scheduling strategy includes at least one of resource utilization priority strategy and efficiency priority strategy. That is to say, the resource manager considers the running time factor of each task when performing resource allocation, and schedules Tasks with fixed space requirements (that is, the resource requirements of tasks) and time requirements (that is, the time of tasks) to When on the corresponding node, the resource utilization priority strategy and the efficiency priority strategy can be flexibly selected according to the corresponding scheduling strategy for resource allocation, so that the allocation configuration with higher resource utilization rate and/or higher efficiency is finally adopted. On the one hand, since the allocation configuration with high resource utilization can be adopted, that is, the task combination with high resource utilization of those nodes can be scheduled to the nodes through the scheduling strategy, so the resource manager can effectively alleviate the problems in the prior art. The problem of resource fragmentation, thereby improving the resource utilization of the cluster. On the other hand, since a more efficient allocation configuration can be adopted, that is, the task combination with the shortest execution time of the job can be scheduled to the node through the scheduling strategy, so compared with the existing technology, the resource manager can significantly shorten Job execution time, improving job execution efficiency. To sum up, the resource manager provided by the embodiment of the present invention can flexibly select the resource utilization rate priority policy and the efficiency priority policy to allocate resources according to the corresponding scheduling policy, so as to improve the resource utilization rate, and/or improve the efficiency of user jobs. effectiveness.

With reference to the second aspect, in the first possible implementation of the second aspect, if the scheduling strategy is a resource utilization priority strategy, then the first allocation configuration is specifically such that each of the multiple task-running computing nodes The allocation configuration that maximizes the single-node resource utilization of computing nodes.

With reference to the second aspect, in a second possible implementation manner of the second aspect, if the scheduling strategy is an efficiency-first strategy, the first allocation configuration is specifically the allocation configuration that enables the fastest overall job execution speed.

In combination with the second aspect or the first possible implementation of the second aspect or the second possible implementation of the second aspect, in the third possible implementation of the second aspect, the estimation unit is specifically configured to: for each task , are processed according to the following operations for the first task: match the hard information of the first task with the hard information of the historical tasks in the sample library; if the matching is successful, according to the The historical running time estimates the running time of the first task.

Specifically, the hard information in this embodiment of the present invention may specifically include job type, execution user and other information.

It should be noted that this embodiment of the present invention is only an example of a specific implementation of estimating the running time of a task by the estimating unit. Of course, the estimating unit can also estimate the running time of the task in other ways, for example, by pre-running the task Way. That is, by pre-running a small number of job instances to obtain an accurate estimate of the full runtime. Also, the run times of subsequent tasks of the same job will be more accurately estimated by reference to the run times of tasks that have already run. The embodiment of the present invention does not limit the specific implementation manner of estimating the running time of the task by the estimating unit.

With reference to any one of the possible implementation manners from the second aspect to the third possible implementation manner of the second aspect, in the fourth possible implementation manner of the second aspect, the resource manager further includes a classification unit; The resource demand and running time corresponding to each task, combined with the preset scheduling strategy, before determining the first allocation configuration of the multiple tasks, the classification unit is used to classify the multiple tasks according to the type of resources to obtain at least one type of task;

The determination unit is specifically configured to: for each type of task in the at least one type of task, perform processing according to the following operation for the first type of task: according to the resource demand and running time corresponding to each task in the first type of task, combined with The preset scheduling policy determines the sub-allocation configuration of the first type of task, and the sub-allocation configuration is used to indicate the distribution of the first type of task on the runnable computing nodes among the plurality of computing nodes; the at least one A combination of sub-allocation configurations of each type of task in the type of tasks is determined as a first allocation configuration of the plurality of tasks.

Since the resource manager provided by the embodiment of the present invention can first classify multiple tasks according to the types of resources, and then perform resource allocation for each type of task separately, that is to say, it can simultaneously consider the requirements for heterogeneous clusters and jobs with special resource requirements. Therefore, it has wider applicability and better comprehensive performance.

Optionally, take into account that the running time estimates will usually have some deviation from the actual situation. If these deviations are not controlled, as time goes on, the pre-allocation results of job resources may be more and more different from the ideal results. Therefore, when the resource manager provided by the embodiment of the present invention allocates resources, it can also introduce a mutation mechanism (that is, reallocate). which is:

With reference to any one of the possible implementation manners of the second aspect to the fourth possible implementation manner of the second aspect, in the fifth possible implementation manner of the second aspect, the allocation unit assigns the plurality of tasks according to the first allocation After the configuration is allocated to the operational computing nodes of the plurality of tasks, the determining unit is also used to: determine the first overall allocation when all tasks in the waiting state run on the allocated nodes according to the first allocation configuration Objective function value; according to the resource requirements and running time corresponding to all tasks in the waiting state, combined with the preset scheduling strategy, determine the second allocation configuration of all tasks in the waiting state, and the second allocation configuration is used to indicate The distribution of all the tasks in the waiting state on the runnable computing nodes of the tasks in the waiting state; according to the second allocation configuration, determine the first time when all the tasks in the waiting state are running on the allocated nodes Two overall allocation objective function values; the allocation unit is also used to assign all tasks in the waiting state to all waiting states according to the second allocation configuration if the second overall allocation objective function value is greater than the first overall allocation objective function value On the runnable compute node of the task.

Through the above mutation mechanism, the pre-allocation results of job resources can be made to evolve in a better direction.

In combination with the fifth possible implementation of the first aspect, in the sixth possible implementation of the first aspect; or, in combination with the fifth possible implementation of the second aspect, in the sixth possible implementation of the second aspect , when the first overall allocation objective function value is equal to the first allocation configuration, the sum of the single-node allocation objective function values of each node when all tasks in the waiting state run on the allocated nodes;

The second overall allocation objective function value is equal to the sum of the single-node allocation objective function values of each node when all tasks in the waiting state run on the allocated nodes when the second allocation configuration is configured.

Optionally, in a possible implementation manner, the above-mentioned single-node allocation objective function specifically includes: Among them, S _n represents the distribution objective function value of single node n; p represents the job time priority factor, p>0; f represents the job fairness factor, f>0;p+f≤1; m represents the number of jobs; S _{e, n} represents the resource utilization score on node n, r _n represents the resources of node n, r _t represents the resource demand of task t; S _p,j represents the execution progress of job j, T _j indicates how much time is required for job j to be executed, and this value can be obtained from historical statistical data. T ₀ indicates the overall running time of job j; S _{f, j} indicates the fairness score of job j, r _j represents the resource requirement of job j, and r _f represents the due resource of job i under the condition of complete fairness.

Wherein, in the above-mentioned single-node allocation objective function, the resource utilization rate of the node, the fairness of the job, and the execution progress of the job are considered. Wherein, when f=0, p=0, resource utilization is fully considered, that is, the resource manager 110 allocates resources according to the principle of the highest overall resource utilization; when f=1, p=0, fairness is fully considered, that is, resource The manager 110 will allocate resources among different jobs fairly; when f=1 and p=1, time priority is fully considered, that is, the resource manager 110 gives priority to assigning resources to jobs that finish faster. Of course, f and p can also be other values, and the user can set them according to the operation requirements of the job, so that the allocation can be balanced between the optimal resource utilization rate and the optimal job execution time, which is not specifically limited in the embodiment of the present invention.

In a third aspect, a resource manager is provided, and the resource manager includes: a processor, a memory, a bus, and a communication interface; the memory is used to store computer-executed instructions, the processor and the memory are connected through the bus, and when the resource manager is running, The processor executes the computer-executable instructions stored in the memory, so that the resource manager executes the above resource allocation method as shown in the first aspect or any possible implementation manner of the first aspect.

Since the resource manager provided by the embodiment of the present invention can be used to execute the above-mentioned resource allocation method as shown in the first aspect or any possible implementation of the first aspect, the technical effect it can obtain can refer to the above-mentioned The technical effect of the resource allocation method shown in the first aspect or any possible implementation manner of the first aspect will not be repeated here.

In a fourth aspect, a distributed computer system is provided, and the distributed computer system includes a plurality of computing nodes and the first aspect or the resource manager described in any possible implementation of the first aspect; or, the distributed The computer system includes multiple computing nodes and the resource manager described in the third aspect.

Since the distributed computer system provided by the embodiment of the present invention includes the resource manager described in the first aspect or any possible implementation of the first aspect; or includes the resource manager described in the third aspect, therefore, it For the technical effects that can be obtained, reference can be made to the technical effects of the above-mentioned resource manager, and the embodiments of the present invention will not be repeated here.

In a fifth aspect, there is provided a readable medium, including computer-executable instructions. When a processor of the resource manager executes the computer-executable instructions, the resource manager executes any optional The resource allocation method described in Methods.

Wherein, these or other aspects of the present invention will be more concise and understandable in the description of the following embodiments.

Description of drawings

Fig. 1 is the scheduling strategy schematic diagram of existing Hadoop system;

FIG. 2 is a logical architecture diagram of a distributed computing system provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of a physical architecture of a distributed computing system provided by an embodiment of the present invention;

FIG. 4 is a schematic diagram of the principle of a resource allocation method provided by an embodiment of the present invention;

FIG. 5 is a first schematic flow diagram of a resource allocation method provided by an embodiment of the present invention;

FIG. 6 is a second schematic flow diagram of a resource allocation method provided by an embodiment of the present invention;

FIG. 7 is a third schematic flow diagram of a resource allocation method provided by an embodiment of the present invention;

FIG. 8 is a schematic diagram of a variation mechanism of a resource allocation result provided by an embodiment of the present invention;

FIG. 9 is a schematic diagram of resource allocation results provided by an embodiment of the present invention using the principle of resource utilization priority;

FIG. 10 is a schematic diagram of the result of resource allocation using the principle of fairness and priority provided by the embodiment of the present invention;

FIG. 11 is a first structural diagram of a resource manager provided by an embodiment of the present invention;

FIG. 12 is a second structural diagram of a resource manager provided by an embodiment of the present invention;

FIG. 13 is a third structural schematic diagram of a resource manager provided by an embodiment of the present invention.

detailed description

It should be noted that, in order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used for the same or similar items with basically the same function and effect To make a distinction, those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order.

It should be noted that "/" in this article means or, for example, A/B can mean A or B; "and/or" in this article is just an association relationship describing associated objects, indicating that there can be three A relationship, for example, A and/or B, can mean: A exists alone, A and B exist simultaneously, and B exists alone. "A plurality" means two or more than two.

As used in this application, the terms "component," "module," "system" and the like are intended to refer to a computer-related entity, which may be hardware, firmware, a combination of hardware and software, software, or an operating system. software. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. As an example, both an application running on a computing device and the computing device can be components. One or more components can reside within a process and/or thread of execution and a component can be localized on one computer and/or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures thereon. These components can be communicated through, for example, according to having one or more packets of data (e.g., data from a component that interacts with another component in a local system, a distributed system, and/or in the form of network to interact with other systems) to communicate with local and/or remote processes.

The present application presents various aspects, embodiments or features in terms of a system that can include a number of devices, components, modules and the like. It is to be understood and appreciated that the various systems may include additional devices, components, modules, etc. and/or may not include all of the devices, components, modules etc. discussed in connection with the figures. In addition, combinations of these schemes can also be used.

In addition, in the embodiments of the present invention, the word "exemplary" is used as an example, illustration or illustration. Any embodiment or design described herein as "example" is not to be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the word example is intended to present concepts in a concrete manner.

The scenarios described in the embodiments of the present invention are to illustrate the technical solutions of the embodiments of the present invention more clearly, and do not constitute limitations on the technical solutions provided by the embodiments of the present invention. Those of ordinary skill in the art know that with the emergence of new scenarios, The technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

In order to make the description of the following embodiments clear and concise, a brief introduction of related concepts is given first:

Cluster: Cluster refers to the combination of multiple homogeneous or heterogeneous computer nodes through the network, and cooperates with a certain cluster management system to form a facility that can provide unified computing or storage services to the outside world.

Resource: resources refer to the available memory, central processing unit (full English name: central processing unit, English abbreviation: CPU), network, disk and other hardware necessary for running jobs on the distributed cluster.

Job: A job refers to a complete task that can be run submitted by the user to the cluster through the client device.

Task: Task. When a job is submitted to the cluster for execution, it is usually decomposed into many tasks. Each task runs on a specific cluster node and occupies a certain amount of resources.

Scheduler: The scheduler is an engine module used to allocate resources available to tasks to run, and is also the most important part of the cluster management system.

The solution of the embodiment of the present invention can be typically applied to a distributed computing system to realize task scheduling and efficient allocation of resources. Fig. 2 shows a logical architecture diagram of a distributed computing system. According to Fig. 2, the distributed computing system includes a resource pool composed of cluster resources, a resource manager and a computing framework. Computing, storage and other hardware resources, the resource manager is deployed on one or more computing nodes in the cluster, or it can also be used as an independent physical device to manage cluster resources in a unified manner and provide resource scheduling capabilities for the upper computing framework . A distributed computing system can support a variety of different computing frameworks at the same time. The system shown in Figure 2 can support MR (full name in English: map reduce), Storm, S4 (full name in English: simple scalable streaming system) and MPI ( English full name: one or more of computing frameworks such as message passing interface). The resource manager performs unified scheduling on the application programs of different computing framework types sent by the client device, so as to improve resource utilization. Fig. 3 further shows a schematic diagram of the physical architecture of a distributed computing system, including a cluster, a resource manager and a client device, wherein the cluster includes multiple nodes (only three nodes are shown in Fig. 3), and the resource manager Deployed on a node in the cluster, each node can communicate with the resource manager, the client device submits the resource request of the application to the resource manager, and the resource manager allocates the resources of the node to the resource manager according to a specific resource scheduling strategy application, so that the application runs on the node according to the allocated node resources.

The embodiment of the present invention mainly optimizes the resource manager in the distributed computing system, so that it can allocate resources to tasks more reasonably, so as to improve resource utilization. Wherein, FIG. 4 is a schematic diagram of a principle of a method for allocating resources provided by an embodiment of the present invention.

As shown in FIG. 4, in the embodiment of the present invention, after the client device submits the job, the job is decomposed into a series of tasks (Tasks) that can be run in a distributed manner, and each Task is configured with a corresponding resource requirement (Fig. In 2, the horizontal width is used to represent the resource demand). When the Task passes through the experience module (English full name: ExperiencedExpert, English abbreviation: E-Expert) in the resource manager, E-Expert estimates the running time of each Task through the job history execution status in the sample library in the resource manager , and then you can get a Task with fixed space requirements (ie, the resource requirements of the Task) and time requirements (ie, the running time of the Task, which is represented by the vertical length in Figure 2). The packing module (English: Packer) considers a certain scheduling strategy, and schedules tasks with fixed space requirements (that is, the resource requirements of the Task) and time requirements (that is, the running time of the Task) to corresponding nodes. Wherein, the scheduling policy includes a resource utilization priority policy, or an efficiency priority policy. That is to say, the packaging method can flexibly select a resource utilization priority strategy and an efficiency priority strategy according to a corresponding scheduling strategy for resource allocation, so that an allocation configuration with higher resource utilization and/or higher efficiency is finally adopted.

Among them, Task forms a waiting queue at each node, and the length of each waiting queue is approximately equal. During the queuing process of Tasks, there may be mutations that lead to a new round of reallocation of job resources. Update to a new allocation configuration if the overall resource utilization or efficiency of the new allocation configuration is higher.

In addition, each node runs a node tracker (English: node tracker) instance, which is responsible for periodically reporting the running status of the Task to E-Expert, and updating the statistical information in the experience module sample library.

It should be noted that in FIG. 2 , the same padding is used to represent the same resource type, such as CPU or memory, and the embodiment of the present invention does not specifically limit the resource type represented by each padding in FIG. 2 .

Since this scheme takes into account the running time factor of each task, and when the packaging module schedules tasks with fixed space requirements (that is, the resource requirements of the Task) and time requirements (that is, the running time of the Task) to the corresponding nodes, it can According to the corresponding scheduling strategy, the resource utilization priority strategy and the efficiency priority strategy are flexibly selected for resource allocation, so that the allocation configuration with higher resource utilization rate and/or higher efficiency is finally adopted, so the problem of resource fragmentation in the prior art can be alleviated. problems, significantly improve the utilization of cluster resources, and/or, shorten job execution time, and improve job execution efficiency.

The following will clearly and completely describe the technical solution in the embodiment of the present invention based on the principle schematic diagram of the resource allocation method shown in FIG. 4 .

As shown in Figure 5, an embodiment of the present invention provides a resource allocation method, including steps S501-S504:

S501. The resource manager receives a job submitted by a client device, and decomposes the job into multiple tasks, where each task in the multiple tasks is configured with a corresponding resource requirement.

S502. The resource manager estimates the running time of each task.

S503. The resource manager determines the first allocation configuration of the multiple tasks according to the resource demand and running time corresponding to each task, in combination with the preset scheduling policy, and the first allocation configuration is used to indicate the multiple tasks. For the distribution of the executable computing nodes among the plurality of computing nodes, the scheduling strategy includes at least one of resource utilization priority strategy and efficiency priority strategy.

S504. The resource manager allocates the multiple tasks to computing nodes that can run the multiple tasks according to the first allocation configuration.

Specifically, in step S501 of the embodiment of the present invention:

The resource demand may specifically be the demand of CPU resources, and/or the demand of memory resources, and/or the demand of network bandwidth, etc., which are not specifically limited in this embodiment of the present invention.

Specifically, in step S502 of the embodiment of the present invention:

Generally speaking, the specific execution time of a task cannot be known to the outside world. However, in the vast majority of cases, one type of task is performed repeatedly. For example, in an enterprise customer scenario, it may be necessary to perform repetitive data statistics work every day. Therefore, statistics based on historical information can often give an estimate of the running time of a class of tasks. For this reason, a module in the resource manager may be required to maintain a statistical information library for recording historical job information of the cluster, such as the E-Expert module in Figure 4. When a new task arrives, the task is matched to a certain category according to the historical statistical information, and then the running time of the task is estimated according to the running history of this type of task.

The E-Expert module usually divides information into two categories: hard information and soft information.

Hard information includes job type, executing user, etc. Tasks of different job types obviously do not belong to the same class. However, the jobs run by the same user are likely to be of the same type, or even repeated jobs. Hard information is maintained by the sample repository described below.

Soft information includes the size of the amount of data processed by the task, the size of input data, the size of output data, and the like. This kind of information is often not fixed, but there is a close correlation between running time and this kind of information. Soft information requires additional statistics, and the statistical information is maintained by the statistical library described below.

Further, optionally, the resource manager may estimate the running time of each task in the following ways, specifically including:

For each task, proceed as follows for the first task:

Match the hard information of the first task with the hard information of the historical tasks in the sample library.

If the matching is successful, the running time of the first task is estimated according to the historical running time of the historical task matching the hard information of the first task.

It should be noted that when a new task arrives, if the task cannot be matched to a certain category according to the historical statistical information, the running time of the task can be estimated by assigning a global tie value to the task. The value may be the average running time of all historical tasks, which is not specifically limited in this embodiment of the present invention.

It should be noted that the embodiment of the present invention is only an example of a specific implementation of estimating the running time of a task. Of course, the running time of a task may also be estimated in other ways, for example, by pre-running the task. That is, by pre-running a small number of job instances to obtain an accurate estimate of the full runtime. Also, the run times of subsequent tasks of the same job will be more accurately estimated by reference to the run times of tasks that have already run. The embodiment of the present invention does not limit the specific implementation manner of estimating the running time of the task.

Specifically, in step S503 of the embodiment of the present invention:

The scheduling strategy may specifically include at least one of a resource utilization priority strategy and an efficiency priority strategy. in,

If the scheduling strategy is a resource utilization priority strategy, the first allocation configuration may specifically be an allocation configuration that maximizes the single-node resource utilization of each computing node among the executable computing nodes of multiple tasks.

Alternatively, if the scheduling strategy is an efficiency-first strategy, the first allocation configuration may specifically be an allocation configuration that enables the overall execution speed of the job to be the fastest.

The embodiment of the present invention does not specifically limit the specific form of the first allocation configuration.

Based on the resource allocation method provided by the embodiment of the present invention, in the resource allocation method, after receiving the job submitted by the client device and decomposing the job into a plurality of tasks with corresponding resource requirement configurations, each task is further estimated The running time of each task, and according to the resource requirements and running time of each task, combined with the preset scheduling strategy, determine the first allocation configuration of the multiple tasks, and then allocate the multiple tasks according to the first allocation configuration to the runnable compute nodes for the multiple tasks. Wherein, the first allocation configuration is used to indicate the distribution of the multiple tasks on the operational computing nodes of the multiple tasks, and the scheduling strategy includes at least one of resource utilization priority strategy and efficiency priority strategy. That is to say, the scheme takes into account the running time of each task, and when scheduling tasks with fixed space requirements (that is, the resource requirements of the task) and time requirements (that is, the time of the task) to the corresponding nodes, it can According to the corresponding scheduling policy, the resource utilization priority strategy and the efficiency priority strategy are flexibly selected for resource allocation, so that an allocation configuration with higher resource utilization rate and/or higher efficiency is finally adopted. On the one hand, since the allocation configuration with high resource utilization can be adopted, that is, the task combination with high resource utilization of those nodes can be scheduled to the nodes through the scheduling strategy, so this allocation scheme can effectively reduce the resource utilization in the prior art. The problem of fragmentation, thereby improving the resource utilization of the cluster. On the other hand, since a more efficient allocation configuration can be adopted, that is, the task combination with the shortest execution time of the job can be scheduled to the node through the scheduling strategy, so compared with the existing technology, this allocation scheme can significantly shorten the job Execution time, improve job execution efficiency. To sum up, the resource allocation method provided by the embodiment of the present invention can flexibly select the resource utilization rate priority policy and the efficiency priority policy for resource allocation according to the corresponding scheduling policy, thereby improving the resource utilization rate, and/or, improving the efficiency of user jobs. effectiveness.

Optionally, in the resource allocation method provided by the embodiment of the present invention, resource allocation for heterogeneous clusters and jobs with special resource requirements can also be considered at the same time.

That is, as shown in FIG. 6, before the resource manager determines the first allocation configuration of the plurality of tasks according to the resource demand and running time corresponding to each task in combination with the preset scheduling strategy (step S503), May include step S505:

S505. The resource manager classifies the multiple tasks according to resource types to obtain at least one type of task.

The resource types may specifically include heterogeneous resource types and non-heterogeneous resource types, and the heterogeneous resource types may also be subdivided according to the type of heterogeneous resources, which is not specifically limited in this embodiment of the present invention.

Furthermore, the resource manager determines the first allocation configuration of the multiple tasks according to the resource demand and running time corresponding to each task, in combination with the preset scheduling strategy (step S503), which may specifically include steps S503a and S503b:

S503a. For each type of task in the at least one type of task, the resource manager performs the following operations for the first type of task:

According to the resource demand and running time corresponding to each task in the first type of task, combined with the preset scheduling strategy, determine the sub-allocation configuration of the first type of task, and the sub-allocation configuration is used to indicate the first The distribution of class tasks on runnable compute nodes among multiple compute nodes.

S503b. The resource manager determines a combination of sub-allocation configurations of each type of task in the at least one type of task as a first allocation configuration of the plurality of tasks.

Since the resource allocation method provided by the embodiment of the present invention first classifies multiple tasks according to the types of resources, and then allocates resources for each type of task separately, that is to say, resources for heterogeneous clusters and special resource requirements can be considered at the same time. Therefore, it has wider applicability and better comprehensive performance.

Optionally, take into account that the running time estimates will usually have some deviation from the actual situation. If these deviations are not controlled, as time goes on, the pre-allocation results of job resources may be more and more different from the ideal results. Therefore, in the resource allocation method provided by the embodiment of the present invention, a mutation mechanism (that is, reallocation) may also be introduced.

That is, as shown in FIG. 7, after the resource manager allocates the multiple tasks to the runnable computing nodes of the multiple tasks according to the first allocation configuration (step S504), steps S506-S509 may also be included:

S506. According to the first allocation configuration, the resource manager determines a first overall allocation objective function value when all tasks in the waiting state run on the allocated nodes.

S507. The resource manager determines the second allocation configuration of all tasks in the waiting state according to the resource requirements and running time corresponding to the tasks in the waiting state, and in combination with the preset scheduling policy. The second allocation configuration uses Indicates the distribution of all tasks in the waiting state on all computing nodes that are able to run the tasks in the waiting state.

S508. According to the second allocation configuration, the resource manager determines a second overall allocation objective function value when all the tasks in the waiting state run on the allocated nodes.

S509. If the value of the second overall allocation objective function is greater than the value of the first overall allocation objective function, the resource manager allocates all the tasks in the waiting state to the runnable computing nodes of all the tasks in the waiting state according to the second allocation configuration superior.

Optionally, in the embodiment of the present invention, the overall allocation objective function value can be obtained by the following formula (1):

S = ∑ _n S _n formula (1)

Among them, S _n represents the distribution objective function value of single node n; S represents the overall distribution objective function value.

That is, when the first overall allocation objective function value in step S506 is equal to the first allocation configuration, the sum of the single-node allocation objective function values of each node when all tasks in the waiting state run on the allocated nodes.

In step S508, when the second overall allocation objective function value is equal to the second allocation configuration, the sum of the single-node allocation objective function values of each node when all tasks in the waiting state run on the allocated nodes.

Optionally, in this embodiment of the present invention, the single-node allocation objective function may be specifically shown in formula (2):

Formula (2)

Among them, S _n represents the distribution objective function value of single node n; p represents the job time priority factor, p>0; f represents the job fairness factor, f>0;p+f≤1; m represents the number of jobs.

Se _,n represents the resource utilization score on node n, r _n represents the resource of node n, r _t represents the resource requirement of task t.

S _{p, j} represents the execution progress of job j, T _j indicates how much time the job j needs to complete, and this value can be obtained from historical statistical data. T ₀ indicates the overall running time of job j. It can be seen that the value range of S _{p, j} is [1/e, 1], 1/e means that the job has just started running, and 1 means that the job has been completed.

S _f,j represents the fairness score of job j, r _j represents the resource requirement of job j, and r _f represents the due resource of job j under the condition of complete fairness.

It should be noted that in the case of multi-dimensional resources, parameters such as r _n , r _t , and r _f in the above formula are all vectors.

In the above formula (2), resource utilization of nodes, job fairness and job execution progress are taken into consideration. Among them, when f=0, p=0, resource utilization is fully considered, that is, the resource manager allocates resources according to the principle of the highest overall resource utilization; when f=1, p=0, fairness is fully considered, that is, resource management The controller will allocate resources among different jobs fairly; when f=1, p=1, time priority is fully considered, that is, the resource manager assigns resources to jobs that finish faster. Of course, f and p can also be other values, and the user can set them according to the operation requirements of the job, so that the allocation can be balanced between the optimal resource utilization rate and the optimal job execution time, which is not specifically limited in the embodiment of the present invention.

This embodiment of the present invention does not specifically limit it.

It should be noted that the formula (2) is only an example of a specific implementation of a single-node allocation objective function, of course, according to different allocation considerations, the single-node allocation objective function can also be other, the implementation of the present invention The example does not specifically limit this.

Through the above mutation mechanism, the pre-allocation results of job resources can be made to evolve in a better direction. Among them, Fig. 8 schematically shows a schematic diagram of a variation mechanism of job resource allocation results. It can be seen that as time goes on, the difference between the pre-allocation results of job resources and the ideal results may become larger and larger. Through the above mutation mechanism, task 1 on node 3 can be adjusted to node 1, task 2 on node 2 can be adjusted to node 3, and task 3 on node 1 can be adjusted to node 2, so that the job resource The pre-allocation results are evolving in a better direction.

It should be noted that in FIG. 8 , the same padding is used to represent the same resource type, such as CPU or memory, and the embodiment of the present invention does not specifically limit the resource type represented by each padding in FIG. 8 .

The resource allocation methods in the foregoing embodiments will be described below with reference to a specific example.

For example, suppose there are four nodes node1, node2, node3 and node4, among which node1, node2 and node3 are homogeneous nodes, each with 6 cores, 12G memory, and 2Gbps network bandwidth; node4 is a heterogeneous graph computing The node has 2 cores, 2G memory and a 128-core graphics processor (English full name: graphics processing unit, English abbreviation: GPU) graphics card.

In addition, assuming that there are four jobs, the resource requirements of the tasks submitted by each job are as follows. Among them, the three-dimensional numbers in parentheses represent the number of cores required by the Task, the size of the memory, and the size of the network bandwidth:

JobA: 18 Map Tasks (1, 2, 0), 3 Reduce Tasks (0, 0, 2);

JobB: 6 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobC: 6 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobD: 2 Map Tasks (1, 1, 0), requiring GPU;

At the same time, it is assumed that the running time of all JobA, JobB and JobC's Tasks is estimated to be t, while the running time of JobD's Task is estimated to be 4t.

Case 1:

At this time, if the principle of resource utilization priority is adopted, for example, the overall resource utilization rate is the highest, that is, f=0 and p=0 in the above-mentioned single-node allocation objective function (formula (2)), then JobA, JobB, JobC and JobD's Task will be scheduled according to the following process:

Step 1. Classify all tasks according to whether they have heterogeneous resource requirements.

At this point, all tasks of JobA, JobB, JobC, and JobD can be divided into two categories: those that require GPU and those that do not.

Step 2. For the two Map Tasks that require GPU, schedule them to node node4.

After this round of allocation, the status of the Task in the waiting state is:

JobA: 18 Map Tasks (1, 2, 0), 3 Reduce Tasks (0, 0, 2);

JobB: 6 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobC: 6 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobD: 0 Map Task (1, 1, 0), requires GPU.

Step 3. For the remaining Tasks, the current cluster still has three nodes, node1, node2 and node3, with a total of (18, 36, 6) resources.

Step 4. For the node1 node, after calculation, the Tasks package (a total of (6, 12, 0) resources) formed by the 6 Map Tasks in JobA makes the single-node allocation objective function value S _n of the node1 node the largest. Therefore, the The Tasks package is placed on node1.

For node2 and node3 nodes, after calculation, the Tasks package formed by the 6 Map Tasks in JobA (a total of (6, 12, 0) resources) makes the single-node allocation objective function value S _n of the node2 node the largest, and the 6 Map Tasks in JobA The Tasks package (a total of (6, 12, 0) resources) formed by four Map Tasks makes the single-node allocation objective function value S _n of node3 the largest. Therefore, the remaining 12 Map Tasks in JobA are evenly distributed to node2 and node3 nodes.

It should be noted that since this example assumes that the running time estimates of all JobA, JobB, and JobC tasks are t, that is, the running time is equal, and f=0, p=0, the single-node allocation objective function degenerate into Then traverse various combinations, and finally determine that the Tasks package formed by the 6 Map Tasks in JobA (a total of (6, 12, 0) resources) makes the single-node allocation objective function value S _n of the node1 node the largest. At this time,

In contrast, if the two Map Tasks of JobB are taken to form a Tasks package, then the value of the single-node assignment objective function S _n is less than the above value, so we will not verify them one by one here.

Similarly, according to the above degradation formula, it can be determined that the Tasks package formed by the 6 Map Tasks in JobA (a total of (6, 12, 0) resources) makes the single-node allocation objective function value S _n of node2 the largest, and the 6 Map Tasks in JobA The Tasks package (a total of (6, 12, 0) resources) formed by four Map Tasks makes the single-node allocation objective function value S _n of the node3 node the largest, and the embodiment of the present invention will not verify them one by one here.

After this round of allocation, the status of the Task in the waiting state is:

JobA: 0 Map Tasks (1, 2, 0), 3 Reduce Tasks (0, 0, 2);

JobB: 6 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobC: 6 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobD: 0 Map Task (1, 1, 0), requires GPU.

Step 5. Continue to distribute:

After calculation, 1 Reduce Task of JobA and 2 Map Tasks of JobB form a Tasks package (a total of (6, 2, 2) resources) so that the single-node allocation objective function value S _n of the node1 node is the largest. Therefore, the Tasks package placed on node1.

After calculation, 1 Reduce Task of JobA and 2 Map Tasks of JobB form a Tasks package (a total of (6, 2, 2) resources) so that the single-node allocation objective function value S _n of the node2 node is the largest. Therefore, the Tasks package placed on node2.

After calculation, 1 Reduce Task of JobA and 2 Map Tasks of JobB form a Tasks package (a total of (6, 2, 2) resources) so that the single-node allocation objective function value S _n of the node3 node is the largest. Therefore, the Tasks package placed on node3.

In this way, after this round of allocation, the status of the Task in the waiting state is:

JobA: 0 Map Task (1, 2, 0), 0 Reduce Task (0, 0, 2);

JobB: 0 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobC: 6 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobD: 0 Map Task (1, 1, 0) requires GPU.

Step 6. Continue to distribute:

After calculation, 1 Reduce Task of JobB and 2 Map Tasks of JobC form a Tasks package (a total of (6, 2, 2) resources) so that the single-node allocation objective function value S _n of the node1 node is the largest. Therefore, the Tasks package placed on node1.

After calculation, 1 Reduce Task of JobB and 2 Map Tasks of JobC form a Tasks package (a total of (6, 2, 2) resources) so that the single-node allocation objective function value S _n of the node2 node is the largest. Therefore, the Tasks package placed on node2.

After calculation, 1 Reduce Task of JobB and 2 Map Tasks of JobC form a Tasks package (a total of (6, 2, 2) resources) so that the single-node allocation objective function value S _n of the node3 node is the largest. Therefore, the Tasks package placed on node3.

In this way, after this round of allocation, the status of the Task in the waiting state is:

JobA: 0 Map Task (1, 2, 0), 0 Reduce Task (0, 0, 2);

JobB: 0 Map Task (3, 1, 0), 0 Reduce Task (0, 0, 2);

JobC: 0 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobD: 0 Map Task (1, 1, 0), requires GPU.

Step 7. Continue to distribute:

Assign the remaining 3 Tasks of JobC to the three nodes. In this way, all assignments are completed, and the final target assignment configuration is shown in Figure 9.

Case two:

At this time, if the principle of fairness and priority is adopted, for example, f=1 and p=0 in the above-mentioned single-node allocation objective function (formula (2)), the tasks of JobA, JobB, JobC and JobD will be scheduled according to the following process:

Step 1. Classify all tasks according to whether they have heterogeneous resource requirements.

At this point, all tasks of JobA, JobB, JobC, and JobD can be divided into two categories: those that require GPU and those that do not.

Step 2. For the two Map Tasks that require GPU, schedule them to node node4.

After this round of allocation, the status of the Task in the waiting state is:

JobA: 18 Map Tasks (1, 2, 0), 3 Reduce Tasks (0, 0, 2);

JobB: 6 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobC: 6 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobD: 0 Map Task (1, 1, 0), requires GPU.

Step 3. For the remaining Tasks, the current cluster still has three nodes, node1, node2 and node3, with a total of (18, 36, 6) resources.

Step 4. For the node1 node, after calculation, the Tasks package (a total of (6, 12, 0) resources) formed by the 6 Map Tasks in JobA makes the single-node allocation objective function value S _n of the node1 node the largest. Therefore, the The Tasks package is placed on node1.

For the node2 node, after calculation, the Tasks package (a total of (6, 2, 0) resources) formed by the two Map Tasks in JobB makes the single-node allocation objective function value S _n of the node2 node the largest, so the Tasks package is placed in on node2.

For the node3 node, after calculation, the Tasks package formed by the two Map Tasks in JobC (a total of (6, 2, 0) resources) makes the single-node allocation objective function value S _n of the node3 node the largest, so the Tasks package is placed in on node3.

It should be noted that since this example assumes that the running time of all JobA, JobB, and JobC tasks is estimated to be t, that is, the running time is equal, and f=1, p=0, the single-node allocation objective function degenerate into After traversing various combinations, it can finally be determined that the Tasks package (a total of (6, 12, 0) resources) formed by the 6 Map Tasks in JobA makes the single-node allocation objective function value S _n of node1 the largest.

Similarly, according to the above degradation formula, it can be determined that the Tasks package formed by the two Map Tasks in JobB (a total of (6, 2, 0) resources) makes the single-node allocation objective function value S _n of node2 the largest, and the 2 in JobC The Tasks package (a total of (6, 2, 0) resources) formed by four Map Tasks makes the single-node allocation objective function value S _n of the node3 node the largest, and the embodiment of the present invention will not verify one by one here.

After this round of allocation, the status of the Task in the waiting state is:

JobA: 12 Map Tasks (1, 2, 0), 3 Reduce Tasks (0, 0, 2);

JobB: 4 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobC: 4 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobD: 0 Map Task (1, 1, 0), requires GPU.

Step 5. Continue to distribute:

After calculation, the Tasks package formed by the 6 Map Tasks in JobA (a total of (6, 12, 0) resources) makes the single-node allocation objective function value S _n of node1 the largest, so the Tasks package is placed on node1.

After calculation, the Tasks package (a total of (6, 2, 0) resources) formed by the two Map Tasks in JobB makes the single-node allocation objective function value S _n of node2 the largest, so the Tasks package is placed on node2.

After calculation, the Tasks package formed by the two Map Tasks in JobC (a total of (6, 2, 0) resources) maximizes the single-node allocation objective function value S _n of node3, so the Tasks package is placed on node3.

In this way, after this round of allocation, the status of the Task in the waiting state is:

JobA: 6 Map Tasks (1, 2, 0), 3 Reduce Tasks (0, 0, 2);

JobB: 2 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobC: 2 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobD: 0 Map Task (1, 1, 0), requires GPU.

Step 6. Continue to distribute:

After calculation, the Tasks package formed by the 6 Map Tasks in JobA (a total of (6, 12, 0) resources) makes the single-node allocation objective function value S _n of node1 the largest, so the Tasks package is placed on node1.

After calculation, the Tasks package (a total of (6, 2, 0) resources) formed by the two Map Tasks in JobB makes the single-node allocation objective function value S _n of node2 the largest, so the Tasks package is placed on node2.

After calculation, the Tasks package formed by the two Map Tasks in JobC (a total of (6, 2, 0) resources) maximizes the single-node allocation objective function value S _n of node3, so the Tasks package is placed on node3.

In this way, after this round of allocation, the status of the Task in the waiting state is:

JobA: 0 Map Tasks (1, 2, 0), 3 Reduce Tasks (0, 0, 2);

JobB: 0 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobC: 0 Map Tasks (3, 1, 0), 3 Reduce Tasks (0, 0, 2);

JobD: 0 Map Task (1, 1, 0), requires GPU.

Step 7. Continue to distribute:

After calculation, the Tasks package formed by one Reduce Task in JobA (a total of (0, 0, 2) resources) makes the single-node allocation objective function value S _n of node1 the largest, so the Tasks package is placed on node1.

After calculation, the Tasks package formed by one Reduce Task in JobB (a total of (0, 0, 2) resources) maximizes the single-node allocation objective function value S _n of node2, so the Tasks package is placed on node2.

After calculation, the Tasks package formed by one Reduce Task in JobC (a total of (0, 0, 2) resources) makes the single-node allocation objective function value S _n of node3 the largest, so the Tasks package is placed on node3.

In this way, after this round of allocation, the status of the Task in the waiting state is:

JobA: 0 Map Tasks (1, 2, 0), 2 Reduce Tasks (0, 0, 2);

JobB: 0 Map Tasks (3, 1, 0), 2 Reduce Tasks (0, 0, 2);

JobC: 0 Map Tasks (3, 1, 0), 2 Reduce Tasks (0, 0, 2);

JobD: 0 Map Task (1, 1, 0), requires GPU.

Step 8. Continue to assign:

After calculation, the Tasks package formed by one Reduce Task in JobA (a total of (0, 0, 2) resources) makes the single-node allocation objective function value S _n of node1 the largest, so the Tasks package is placed on node1.

After calculation, the Tasks package formed by one Reduce Task in JobB (a total of (0, 0, 2) resources) maximizes the single-node allocation objective function value S _n of node2, so the Tasks package is placed on node2.

After calculation, the Tasks package formed by one Reduce Task in JobC (a total of (0, 0, 2) resources) makes the single-node allocation objective function value S _n of node3 the largest, so the Tasks package is placed on node3.

In this way, after this round of allocation, the status of the Task in the waiting state is:

JobA: 0 Map Task (1, 2, 0), 1 Reduce Task (0, 0, 2);

JobB: 0 Map Task (3, 1, 0), 1 Reduce Task (0, 0, 2);

JobC: 0 Map Task (3, 1, 0), 1 Reduce Task (0, 0, 2);

JobD: 0 Map Task (1, 1, 0), requires GPU;

Step 9. Continue to assign:

After calculation, the Tasks package formed by one Reduce Task in JobA (a total of (0, 0, 2) resources) makes the single-node allocation objective function value S _n of node1 the largest, so the Tasks package is placed on node1.

After calculation, the Tasks package formed by one Reduce Task in JobB (a total of (0, 0, 2) resources) maximizes the single-node allocation objective function value S _n of node2, so the Tasks package is placed on node2.

After calculation, the Tasks package formed by one Reduce Task in JobC (a total of (0, 0, 2) resources) makes the single-node allocation objective function value S _n of node3 the largest, so the Tasks package is placed on node3.

In this way, after this round of allocation, all allocations are completed, and the final target allocation configuration is shown in FIG. 10 .

Case three:

At this time, if the principle of time priority is adopted, for example, the overall execution efficiency is the highest, that is, f=0 and p=1 in the above-mentioned single node allocation objective function (formula (2)), then the Tasks of JobA, JobB, JobC and JobD Time priority will be fully considered when making allocations, that is, the resource manager will give priority to assigning resources to jobs that complete faster. At this point, the single node assigns the objective function degenerate into Furthermore, it is enough to allocate according to the principle that the value of the single-node objective function of each node is the largest, and the embodiment of the present invention will not illustrate this situation in detail.

From the corresponding examples in Figure 9 and Figure 10, it can be seen that if fairness is ignored, the resource utilization priority allocation method can effectively shorten the overall execution time of the job to 4t, and if fairness is considered, the overall allocation method with fairness priority The job execution time is 6t.

As shown in FIG. 11 , an embodiment of the present invention provides a resource manager 110 for executing the resource allocation methods shown in FIGS. 5-7 above. The resource manager 110 may include units corresponding to corresponding steps, for example, may include: a receiving unit 1101 , a decomposing unit 1106 , an estimating unit 1102 , a determining unit 1103 and an allocating unit 1104 .

The receiving unit 1101 is configured to receive a job submitted by a client device.

The decomposing unit 1106 is configured to decompose the job into a plurality of tasks, wherein each task in the plurality of tasks is configured with a corresponding resource requirement.

An estimating unit 1102, configured to estimate the running time of each task.

The determination unit 1103 is configured to determine a first allocation configuration of the multiple tasks according to the resource requirement and running time corresponding to each task, in combination with a preset scheduling policy, and the first allocation configuration is used to indicate the multiple tasks The distribution of tasks on the runnable computing nodes among the plurality of computing nodes, the scheduling strategy includes at least one of resource utilization priority strategy and efficiency priority strategy.

An allocating unit 1104, configured to allocate the multiple tasks to computing nodes that can run the multiple tasks according to a first allocation configuration.

Optionally, if the scheduling strategy is a resource utilization priority strategy, the first allocation configuration may specifically be an allocation configuration that maximizes the single-node resource utilization of each computing node among the executable computing nodes of the multiple tasks .

Optionally, if the scheduling policy is an efficiency-first policy, the first allocation configuration may be the allocation configuration that makes the overall job execution speed the fastest.

The embodiment of the present invention does not specifically limit the specific form of the first allocation configuration.

Optionally, the estimating unit 1102 may specifically be used for:

For each task, proceed as follows for the first task:

Matching the hard information of the first task with the hard information of the historical tasks in the sample library;

If the matching is successful, the running time of the first task is estimated according to the historical running time of the historical task matching the hard information of the first task.

Specifically, the hard information in this embodiment of the present invention includes job type, execution user and other information.

It should be noted that this embodiment of the present invention is only an example of a specific implementation of estimating the running time of a task by the estimating unit 1102. Of course, the estimating unit 1102 can also estimate the running time of a task in other ways, for example, by pre-running the task Way. That is, by pre-running a small number of job instances to obtain an accurate estimate of the full runtime. Also, the run times of subsequent tasks of the same job will be more accurately estimated by reference to the run times of tasks that have already run. The embodiment of the present invention does not limit the specific implementation manner of estimating the running time of the task by the estimating unit 1102 .

Based on the resource manager 110 provided by the embodiment of the present invention, after the resource manager 110 receives the job submitted by the client device and decomposes the job into multiple tasks with corresponding resource requirement configurations, it also estimates the The running time of each task, and according to the resource requirements and running time of each task, combined with the preset scheduling strategy, determine the first allocation configuration of the multiple tasks, and then allocate the multiple tasks according to the first allocation configuration to the runnable compute nodes for the multiple tasks. Wherein, the first allocation configuration is used to indicate the distribution of the multiple tasks on the computing nodes that can run the multiple tasks, and the scheduling strategy includes at least one of resource utilization priority strategy and efficiency priority strategy. That is to say, the resource manager 110 considers the running time factor of each task when performing resource allocation, and schedules tasks with fixed space requirements (i.e. task resource requirements) and time requirements (i.e. task time) When it is on the corresponding node, it can flexibly select the resource utilization rate priority strategy and the efficiency priority strategy according to the corresponding scheduling strategy for resource allocation, so that the allocation configuration with higher resource utilization rate and/or higher efficiency is finally adopted. On the one hand, since the allocation configuration with high resource utilization can be adopted, that is, the task combination with high resource utilization of those nodes can be scheduled to the nodes through the scheduling policy, so the resource manager 110 can effectively alleviate the existing technology. The problem of resource fragmentation in the middle, thereby improving the resource utilization of the cluster. On the other hand, since a more efficient allocation configuration can be adopted, that is, the task combination with the shortest job execution time can be scheduled to the node through the scheduling policy, so compared with the prior art, the resource manager 110 can significantly Shorten job execution time and improve job execution efficiency. To sum up, the resource manager 110 provided by the embodiment of the present invention can flexibly select a resource utilization priority strategy and an efficiency priority strategy for resource allocation according to the corresponding scheduling strategy, thereby improving resource utilization and/or improving user job performance. execution efficiency.

Optionally, when the resource manager 110 provided in the embodiment of the present invention allocates resources, it may also consider resource allocation for heterogeneous clusters and jobs with special resource requirements.

Specifically, as shown in FIG. 12 , the resource manager 110 may further include a classification unit 1105 .

Before the determination unit 1103 determines the first allocation configuration of the multiple tasks according to the resource demand and running time corresponding to each task, combined with the preset scheduling policy, the classification unit 1105 is used to classify the multiple tasks according to the resource Classify the types of tasks to obtain at least one type of task.

The determining unit 1103 is specifically used for:

For each type of task in the at least one type of task, perform the following operations for the first type of task:

According to the resource demand and running time corresponding to each task in the first type of task, combined with the scheduling strategy, determine the sub-allocation configuration of the first type of task, and the sub-allocation configuration is used to indicate that the first type of task is in multiple computing The distribution of runnable compute nodes in a node.

A combination of sub-allocation configurations of each type of task in the at least one type of task is determined as a first allocation configuration of the plurality of tasks.

Since the resource manager 110 provided by the embodiment of the present invention can first classify multiple tasks according to the types of resources, and then perform resource allocation for each type of task separately, that is to say, it can simultaneously consider the tasks for heterogeneous clusters and special resource requirements. Therefore, it has wider applicability and better comprehensive performance.

Optionally, take into account that the running time estimates will usually have some deviation from the actual situation. If these deviations are not controlled, as time goes on, the pre-allocation results of job resources may be more and more different from the ideal results. Therefore, in the resource manager 110 provided by the embodiment of the present invention, after the allocation unit 1104 allocates multiple tasks to the runnable computing nodes of multiple tasks according to the first allocation configuration, the determination unit 1103 is further configured to:

According to the first allocation configuration, a first overall allocation objective function value when all tasks in the waiting state run on the allocated nodes is determined.

According to the resource requirements and running time corresponding to all tasks in the waiting state, combined with the scheduling strategy, determine the second allocation configuration of all tasks in the waiting state, and the second allocation configuration is used to indicate that all tasks in the waiting state are in The distribution of all waiting tasks across the runnable compute nodes.

According to the second allocation configuration, determine a second overall allocation objective function value when all tasks in the waiting state run on the allocated nodes;

The allocation unit 1104 is further configured to allocate all the tasks in the waiting state to the runnable computing nodes of all the tasks in the waiting state according to the second allocation configuration if the value of the second overall allocation objective function is greater than the value of the first overall allocation objective function superior.

Through the above mutation mechanism, the pre-allocation results of job resources can be made to evolve in a better direction. Among them, Fig. 8 schematically shows a schematic diagram of a variation mechanism of job resource allocation results. It can be seen that as time goes on, the difference between the pre-allocation results of job resources and the ideal results may become larger and larger. Through the above mutation mechanism, task 1 on node 3 can be adjusted to node 1, task 2 on node 2 can be adjusted to node 3, and task 3 on node 1 can be adjusted to node 2, so that the job resource The pre-allocation results are evolving in a better direction.

Optionally, in the embodiment of the present invention, the value of the overall allocation objective function can be obtained by the above formula (1), and the embodiment of the present invention will not repeat it here.

That is, when the above-mentioned first overall allocation objective function value is equal to the sum of the single-node allocation objective function values of each node when all tasks in the waiting state run on the allocated nodes in the first allocation configuration.

The value of the above-mentioned second overall allocation objective function is equal to the sum of the single-node allocation objective function values of each node when all tasks in the waiting state run on the allocated nodes in the second allocation configuration.

Optionally, in this embodiment of the present invention, the single-node allocation objective function may specifically include:

Among them, S _n represents the distribution objective function value of single node n; p represents the job time priority factor, p>0; f represents the job fairness factor, f>0;p+f≤1; m represents the number of jobs; S _{e, n} represents the resource utilization score on node n, r _n represents the resources of node n, r _t represents the resource demand of task t; S _p,j represents the execution progress of job j, T _j indicates how much time is required for job j to be executed, and this value can be obtained from historical statistical data. T ₀ indicates the overall running time of job j; S _{f, j} indicates the fairness score of job j, r _j represents the resource requirement of job i, and r _f represents the due resource of job i in the case of complete fairness.

Wherein, in the above-mentioned single-node allocation objective function, the resource utilization rate of the node, the fairness of the job, and the execution progress of the job are considered. Wherein, when f=0, p=0, resource utilization is fully considered, that is, the resource manager 110 allocates resources according to the principle of the highest overall resource utilization; when f=1, p=0, fairness is fully considered, that is, resource The manager 110 will allocate resources among different jobs fairly; when f=1 and p=1, time priority is fully considered, that is, the resource manager 110 gives priority to assigning resources to jobs that finish faster. Of course, f and p can also be other values, and the user can set them according to the operation requirements of the job, so that the allocation can be balanced between the optimal resource utilization rate and the optimal job execution time, which is not specifically limited in the embodiment of the present invention.

It should be noted that the receiving unit 1101 in the embodiment of the present invention can be an interface circuit with a receiving function on the resource manager 110, such as a receiver or a receiver; it can also be a network card or an input circuit with a receiving function on the resource manager 110. /output (English full name: input/output, English abbreviation: I/O) interface, which is not specifically limited in this embodiment of the present invention.

Estimating unit 1102, determining unit 1103, allocating unit 1104 and classifying unit 1105 may be independently established processors, or may be integrated in a certain processor of resource manager 110, and may also be stored in the form of program codes in In the memory of the resource manager 110, a certain processor of the resource manager 110 invokes and executes the functions of the estimation unit 1102, the determination unit 1103, the allocation unit 1104 and the classification unit 1105 above. The processor described here can be a central processing unit (English full name: central processing unit, English abbreviation: CPU), and can also be other general-purpose processors, digital signal processors (English full name: digital signal processing, English abbreviation: DSP) ), application specific integrated circuit (English full name: application specific integrated circuit, English abbreviation: ASIC), field programmable gate array (English full name: field-programmable gate array, English abbreviation: FPGA) or other programmable logic devices, discrete gates or Transistor logic devices, discrete hardware components, and more. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. In addition, the processor may also be a dedicated processor, and the dedicated processor may include at least one of a baseband processing chip, a radio frequency processing chip, and the like. Further, the dedicated processor may also include a chip with other dedicated processing functions of the resource manager 110 .

It can be understood that the resource manager 110 in the embodiment of the present invention may correspond to the resource manager in the resource allocation method shown in FIGS. 5-7 above, and each unit in the resource manager 110 in the embodiment of the present invention The division and/or functions are all for realizing the flow of the resource allocation method shown in FIGS. 5-7 above, and for the sake of brevity, details are not repeated here.

As shown in FIG. 13 , an embodiment of the present invention provides a resource manager 130 , including: a processor 1301 , a memory 1302 , a bus 1303 and a communication interface 1304 .

The memory 1302 is used to store computer-executable instructions, and the processor 1301 is connected to the memory 1302 through a bus. When the resource manager 130 is running, the processor 1301 executes the computer-executable instructions stored in the memory 1303, so that the resource manager 130 executes as shown in Figure 5- The resource allocation method shown in FIG. 7 . For a specific address allocation method, reference may be made to relevant descriptions in the above-mentioned embodiments shown in FIGS. 5-7 , and details are not repeated here.

Wherein, the processor 1301 in the embodiment of the present invention may be a central processing unit (English full name: central processing unit, English abbreviation: CPU), and may also be other general-purpose processors, digital signal processors (English full name: digital signal processing, English abbreviation: DSP), application specific integrated circuit (full English name: applicationspecific integrated circuit, English abbreviation: ASIC), field programmable gate array (English full name: field-programmable gate array, English abbreviation: FPGA) or other programmable logic devices, Discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

In addition, the processor 1301 may also be a dedicated processor, and the dedicated processor may include at least one of a baseband processing chip, a radio frequency processing chip, and the like. Further, the dedicated processor may also include a chip with other dedicated processing functions of the resource manager 130 .

The memory 1302 may include a volatile memory (English: volatile memory), such as a random access memory (English full name: random-access memory, English abbreviation: RAM); the memory 1302 may also include a non-volatile memory (English: non- volatile memory), such as read-only memory (English full name: read-only memory, English abbreviation: ROM), flash memory (English: flash memory), hard disk (English full name: hard disk drive, English abbreviation: HDD) or solid state drive (English full name: solid-state drive, English abbreviation: SSD); in addition, the storage 1302 may also include a combination of the above-mentioned types of storage.

The bus 1303 may include a data bus, a power bus, a control bus, a signal status bus, and the like. In this embodiment, for clarity, various buses are shown as bus 1303 in FIG. 13 .

In a specific implementation process, each step in the flow of the above-mentioned resource allocation method as shown in FIGS. To avoid repetition, details are not repeated here.

Based on the resource manager provided by the embodiment of the present invention, after the resource manager receives the job submitted by the client device and decomposes the job into a plurality of tasks with corresponding resource requirement configurations, it also estimates the operation of each task. Time, and according to the resource demand and running time of each task, combined with the preset scheduling strategy, determine the first allocation configuration of the multiple tasks, and then allocate the multiple tasks to the Compute nodes that can run multiple tasks. Wherein, the first allocation configuration is used to indicate the distribution of the multiple tasks on the computing nodes that can run the multiple tasks, and the scheduling strategy includes at least one of resource utilization priority strategy and efficiency priority strategy. That is to say, the resource manager considers the running time factor of each task when performing resource allocation, and schedules Tasks with fixed space requirements (that is, the resource requirements of tasks) and time requirements (that is, the time of tasks) to When on the corresponding node, the resource utilization priority strategy and the efficiency priority strategy can be flexibly selected according to the corresponding scheduling strategy for resource allocation, so that the allocation configuration with higher resource utilization rate and/or higher efficiency is finally adopted. On the one hand, since the allocation configuration with high resource utilization can be adopted, that is, the task combination with high resource utilization of those nodes can be scheduled to the nodes through the scheduling strategy, so the resource manager can effectively alleviate the problems in the prior art. The problem of resource fragmentation, thereby improving the resource utilization of the cluster. On the other hand, since a more efficient allocation configuration can be adopted, that is, the task combination with the shortest execution time of the job can be scheduled to the node through the scheduling strategy, so compared with the existing technology, the resource manager can significantly shorten Job execution time, improving job execution efficiency. To sum up, the resource manager provided by the embodiment of the present invention can flexibly select the resource utilization rate priority policy and the efficiency priority policy to allocate resources according to the corresponding scheduling policy, so as to improve the resource utilization rate, and/or improve the efficiency of user jobs. effectiveness.

Optionally, an embodiment of the present invention also provides a readable medium, which is used to store computer-executable instructions. When the processor of the resource manager executes the computer-executable instructions, the resource manager executes the computer-executable instructions shown in Figure 5- The resource allocation method shown in FIG. 7 . For a specific resource allocation method, reference may be made to relevant descriptions in the embodiments shown in FIGS. 5-7 above, and details are not repeated here.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the above-described device is only illustrated by dividing the above-mentioned functional modules. In practical applications, the above-mentioned functions can be allocated by different Completion of functional modules means that the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working processes of the above-described systems, devices, and units, reference may be made to the corresponding processes in the foregoing method embodiments, and details are not repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage media include: U disk, mobile hard disk, ROM, RAM), magnetic disk or optical disk and other media that can store program codes.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. Should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (14)

1. A resource allocation method in a distributed computing system, said distributed computing system comprising a plurality of computing nodes, characterized in that said method comprises: receiving a job submitted by a client device, and decomposing the job into a plurality of tasks, wherein each task in the plurality of tasks is configured with a corresponding resource requirement; Estimate the runtime of each of the tasks; According to the resource demand and running time corresponding to each task, combined with a preset scheduling policy, determine a first allocation configuration of the multiple tasks, and the first allocation configuration is used to indicate the multiple tasks Distribution situation on the executable computing nodes among the plurality of computing nodes, the scheduling strategy includes at least one of resource utilization priority strategy and efficiency priority strategy; Allocating the multiple tasks to computing nodes that can run the multiple tasks according to the first allocation configuration. 2. The method according to claim 1, wherein if the scheduling policy is a resource utilization priority policy, the first allocation configuration is such that each of the executable computing nodes of the multiple tasks The allocation configuration with the maximum single-node resource utilization of computing nodes. 3 . The method according to claim 1 , wherein if the scheduling strategy is an efficiency-first strategy, the first allocation configuration is an allocation configuration that enables the overall execution speed of the job to be the fastest. 4 . 4. The method according to any one of claims 1-3, wherein the estimating the running time of each task comprises: For each task described above, it is processed according to the following operations for the first task: Matching the hard information of the first task with the hard information of the historical tasks in the sample library; If the matching is successful, the running time of the first task is estimated according to the historical running time of the historical task matched with the hard information of the first task. 5. The method according to any one of claims 1-4, wherein, when the multiple tasks are allocated to the runable computing nodes of the multiple tasks according to the first allocation configuration After that, also include: According to the first allocation configuration, determine the first overall allocation objective function value when all tasks in the waiting state are running on the allocated nodes; According to the resource requirements and running time corresponding to all the tasks in the waiting state, combined with the scheduling policy, determine the second allocation configuration of all the tasks in the waiting state, and the second allocation configuration is used to indicate The distribution of all tasks in the waiting state on the runnable computing nodes of the tasks in the waiting state; According to the second allocation configuration, determine a second overall allocation objective function value when all the tasks in the waiting state are running on the allocated nodes; If the value of the second overall allocation objective function is greater than the value of the first overall allocation objective function, allocating the tasks in the waiting state to all available tasks in the waiting state according to the second allocation configuration run on a compute node. 6. The method according to claim 5, wherein when the first overall allocation objective function value is equal to the first allocation configuration, when all tasks in the waiting state run on the allocated nodes, each node The sum of the single-node assignment objective function values of ; The second overall allocation objective function value is equal to the sum of the single-node allocation objective function values of each node when all tasks in the waiting state run on the allocated nodes when the second allocation configuration is configured. 7. A resource manager, characterized in that the resource manager comprises: a receiving unit, a decomposing unit, an estimating unit, a determining unit and an allocating unit; The receiving unit is configured to receive a job submitted by a client device; The decomposing unit is configured to decompose the job into a plurality of tasks, wherein each task in the plurality of tasks is configured with a corresponding resource requirement; The estimating unit is configured to estimate the running time of each task; The determining unit is configured to determine a first allocation configuration of the plurality of tasks according to the resource demand and running time corresponding to each task, in combination with a preset scheduling strategy, and the first allocation configuration is used for In order to indicate the distribution of the multiple tasks on the runnable computing nodes among the multiple computing nodes, the scheduling policy includes at least one of a resource utilization priority policy and an efficiency priority policy; The allocating unit is configured to allocate the multiple tasks to computing nodes that can run the multiple tasks according to the first allocation configuration. 8. The resource manager according to claim 7, wherein if the scheduling strategy is a resource utilization priority strategy, the first allocation configuration is such that among the executable computing nodes of the plurality of tasks The allocation configuration that maximizes the single-node resource utilization of each computing node. 9. The resource manager according to claim 8, wherein if the scheduling policy is an efficiency priority policy, the first allocation configuration is the allocation configuration that makes the overall execution speed of the job the fastest . 10. The resource manager according to any one of claims 7-9, wherein the estimation unit is specifically used for: For each task described above, it is processed according to the following operations for the first task: Matching the hard information of the first task with the hard information of the historical tasks in the sample library; If the matching is successful, the running time of the first task is estimated according to the historical running time of the historical task matched with the hard information of the first task. 11. The resource manager according to any one of claims 7-10, wherein the allocation unit allocates the multiple tasks to the available tasks of the multiple tasks according to the first allocation configuration After running on the compute node, the determination unit is also used to: According to the first allocation configuration, determine the first overall allocation objective function value when all tasks in the waiting state are running on the allocated nodes; According to the resource requirements and running time corresponding to all the tasks in the waiting state, combined with the scheduling policy, determine the second allocation configuration of all the tasks in the waiting state, and the second allocation configuration is used to indicate The distribution of all tasks in the waiting state on the runnable computing nodes of the tasks in the waiting state; According to the second allocation configuration, determine a second overall allocation objective function value when all the tasks in the waiting state are running on the allocated nodes; The allocating unit is further configured to allocate all the tasks in the waiting state to the All pending tasks on runnable compute nodes. 12. The resource manager according to claim 11, wherein when the first overall allocation objective function value is equal to the first allocation configuration, all tasks in the waiting state are running on the allocated nodes The sum of the single node distribution objective function values of each node; The second overall allocation objective function value is equal to the sum of the single-node allocation objective function values of each node when all tasks in the waiting state run on the allocated nodes when the second allocation configuration is configured. 13. A resource manager, characterized in that the resource manager comprises: a processor, a memory, a bus, and a communication interface; The memory is used to store computer-executable instructions, the processor is connected to the memory through the bus, and when the resource manager is running, the processor executes the computer-executable instructions stored in the memory to The resource manager is made to execute the resource allocation method in the distributed computing system according to any one of claims 1-6. 14. A distributed computer system, characterized in that the distributed computer system comprises a plurality of computing nodes and the resource manager according to any one of claims 7-12; Alternatively, the distributed computer system includes multiple computing nodes and the resource manager according to claim 13 .