CN106406990A - Batch job-resource matching method and system with security constraint - Google Patents
Batch job-resource matching method and system with security constraint Download PDFInfo
- Publication number
- CN106406990A CN106406990A CN201610749981.9A CN201610749981A CN106406990A CN 106406990 A CN106406990 A CN 106406990A CN 201610749981 A CN201610749981 A CN 201610749981A CN 106406990 A CN106406990 A CN 106406990A
- Authority
- CN
- China
- Prior art keywords
- back end
- data source
- node
- security
- security constraint
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000013507 mapping Methods 0.000 claims abstract description 24
- 238000012544 monitoring process Methods 0.000 claims abstract description 7
- 238000012360 testing method Methods 0.000 claims description 10
- 230000000694 effects Effects 0.000 claims description 6
- 230000000737 periodic effect Effects 0.000 claims description 3
- 238000012913 prioritisation Methods 0.000 claims description 3
- 230000003362 replicative effect Effects 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 abstract 1
- 238000012545 processing Methods 0.000 description 4
- 230000009466 transformation Effects 0.000 description 2
- 239000008358 core component Substances 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/48—Indexing scheme relating to G06F9/48
- G06F2209/484—Precedence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/502—Proximity
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a batch job-resource matching method and system with a security constraint. A security constraint and load balance-based batch job and resource matching method is adopted; the batch job-resource matching method comprises the specific steps of (A) configuring the security at a preprocessing stage; (B) mapping a batch job set to an appropriate resource of meeting the security constraint and a load balance security constraint, sorting jobs on various nodes and sequentially executing the jobs; and (C) dynamically and repeatedly detecting a downtime fault and redistributing the jobs on a fault machine according to the security constraint and the load balance security constraint. The method disclosed by the invention is especially suitable for batch job scheduling applications, such as MapReduce-based police data job scheduling, traffic monitoring job scheduling and the like, with relatively high data security. Secure access of data can be ensured on the basis of obtaining relatively short batch job completion time.
Description
Technical field
The present invention relates to a kind of job stacking-reso urce matching method with security constraint and system, it is particularly suited for data
The higher job stacking scheduling application of security, the such as police service data operation based on MapReduce is dispatched, traffic monitoring operation
Scheduling etc., belongs to Distributed Parallel Computing field.
Background technology
The MapReduce framework of one of the core component as Hadoop is widely used.MapReduce is in data processing
When, big task is divided into calculating operation little one by one, such operation be typical job stacking, include Map operation with
Reduce operation.When a collection of task reaches, MapReduce is divided into very many Map and Reduce operations, now different
The effect that is slightly different of job scheduling plan, the execution efficiency of the resource utilization to cluster for the job stacking scheduling strategy and system
Have a significant impact.
Existing Hadoop scheduler mainly has：(1) prerequisite variable scheduler：The acquiescence scheduler of Hadoop, according to
The order of arrival of operation arranges execution order；(2) Fair Scheduler：The resource occupation amount of configuration operation is so that each task
Resource occupation amount keeps balance；(3) delay dispatching：When data needed for current work is not had on the machine of current idle, this work
Industry postpones a period of time, executes this delay operation when there being the machine idle of data.
Researchers propose some methods with regard to job stacking scheduling, such as take and prefetch strategy to shorten data standard
Standby time [University of Electronic Science and Technology. a kind of data prefetching method based on MapReduce：China, 201510299063.6 [P]
.2015-09-23], different methods is applied to different job scheduling applications, and such as above-mentioned strategy is paying attention to the operation off period
There is preferable effect of optimization in the job scheduling application of constraint.
In practical application, many job scheduling applications have security constraint, such as police service data operation scheduling, and traffic is supervised
Control job scheduling etc., the security requirement of data is higher, and the transfer of data and duplication are somewhat limited, generally require through adding
Close can be transferred or replicate.For this kind of application, existing job scheduling strategy is without reference to the method for security constraint.
In sum, existing job scheduling method can not well adapt to have in practical application security constraint batch at
Reason job stacking scheduling problem.
Content of the invention
Goal of the invention：For problems of the prior art it is contemplated that the specific safety that data has constrains, this
Invention provides a kind of job stacking-reso urce matching method with security constraint and system, meets the demand of practical application, is obtaining
It is ensured that the secure access of data on the basis of shorter batch processing job completion date.
Technical scheme：A kind of job stacking-reso urce matching method with security constraint, comprises the following steps：
Step A, server end control node configure to security；
The current safety facilities of back end, are back end setting safe class；
According to the requirement to security for the data source, it is data source setting safety requirements grade；
According to job requester authority, operation Permission Levels are set；
Step B, server end control node are using the operation based on security constraint and load balance and back end mapping
Method, Map operation is mapped to and meets in safety and the back end of load balance constraint, the operation on each node is pressed preferential
Level is ranked up and executes successively；
Shown security constraint comprises at 2 points：(1) operation can only have permission execution on the back end of execution at it；(2) safety
The data source having high demands must be stored on the low back end of safe class with encrypted state.
Shown load balance refers to control node according to back end computing capability, arranges permissible on each back end
The operation transformation of mapping.
Step C, server end machine monitoring node of delaying is periodically detected back end current state, in real time by fault data section
Point ID feeds back to server end control node；
Step D, server end control node receive fault data node ID, to be currently mapped on this node do not complete
Operation, according to operation and the back end mapping method of security constraint and load balance, operation is remapped to machine data of not delaying
Node.
Using the operation based on security constraint and load balance and back end mapping method, calculate some mapping sides
Case, selects the minimum scheme of cost to be executed；Each data source of scheme cost=setting replicates cost * data source number of copy times
Each encryption expenses * encryption number of times of+setting.It is likely that not having on this back end when Map operation is mapped on back end
Activity Calculation desired data source, needs first to replicate by data source.The process replicating needs to meet security constraint, and that is, safety will
The data source asking high must be stored on the low back end of safe class with encrypted state.Operation is being mapped to by control node
When on back end, it should generation as few as possible replicates and encryption in addition to meeting security constraint and load balance.
A kind of job stacking with security constraint-resource matched system includes control node, back end and machine testing of delaying
Node.
Control node：Receive the operation from user and receive the security constraint from machine testing node of delaying and machine letter of delaying
Breath；After control node receives machine information of delaying, according to fault data node ID, to the operation being currently mapped on this node according to peace
The operation of staff cultivation and load balance and back end mapping method, operation is remapped to machine back end of not delaying；Control
After node receives security constraint information, mate the child node that can receive an assignment.
Control node configures to security：The current safety facilities of back end, are back end setting peace
Congruent level；According to the requirement to security for the data source, it is data source setting safety requirements grade；According to job requester authority,
Setting operation Permission Levels；Using the operation based on security constraint and load balance and back end mapping method, operation is reflected
Be mapped to and meet in safety and the back end of load balance constraint, carry out job scheduling based on security constraint and load balance/
Weight traffic control.
Back end：There is provided the safety level information of each node to control node, after receiving the operation of control node distribution
It is responsible for carrying out prioritization to the operation in respective waiting list and executing, necessary data is carried out according to operation distribution condition
Operation and cryptographic operation are replicated in source, when that is, operation is mapped on back end, if not having Activity Calculation institute on this back end
Need data source, need first to replicate by data source；The high data source of safety requirements is stored in the low back end of safe class
On must encrypt；Gather respective node load information and submit to control node during Job execution；
Delay machine testing node：Carry out periodic back end to delay machine fault detect, the machine back end ID that will delay feeds back to
Control node.
The present invention adopts technique scheme, has the advantages that：
Job stacking-the reso urce matching method with security constraint and system that the present invention provides, dispatched in job stacking
Consider the actual conditions of security constraint in journey, strategy is simply direct, the job scheduling being related to security is applied and has very
Good practicality.
Brief description
Fig. 1 is the configuration diagram of present system preferred embodiment；
Fig. 2 is flow chart of steps in control node for the inventive method preferred embodiment；
Fig. 3 Fig. 2 is flow chart of steps on back end for the inventive method preferred embodiment.
Specific embodiment
With reference to specific embodiment, it is further elucidated with the present invention it should be understood that these embodiments are merely to illustrate the present invention
Rather than restriction the scope of the present invention, after having read the present invention, the various equivalences to the present invention for the those skilled in the art
The modification of form all falls within the application claims limited range.
Traditional Map Reduce system includes control node data node, as shown in figure 1, the band peace that the present invention provides
Job stacking-the reso urce matching method of staff cultivation is realized system and is additionally arranged machine testing node of delaying.Job stacking with security constraint-
Resource matched system includes control node, back end and machine testing node of delaying.
Control node：
Control node：Receive the operation from user and receive the security constraint from machine testing node of delaying and machine letter of delaying
Breath；After control node receives machine information of delaying, according to fault data node ID, to the operation being currently mapped on this node according to peace
The operation of staff cultivation and load balance and back end mapping method, operation is remapped to machine back end of not delaying；Control
After node receives security constraint information, mate the child node that can receive an assignment；
Control node configures to security：The current safety facilities of back end, are back end setting peace
Congruent level；According to the requirement to security for the data source, it is data source setting safety requirements grade；According to job requester authority,
Setting operation Permission Levels；Using the operation based on security constraint and load balance and back end mapping method, operation is reflected
Be mapped to and meet in safety and the back end of load balance constraint, carry out job scheduling based on security constraint and load balance/
Weight traffic control.
Back end：There is provided the safety level information of each node to control node, after receiving the operation of control node distribution
It is responsible for carrying out prioritization to the operation in respective waiting list and executing, necessary data is carried out according to operation distribution condition
Operation and cryptographic operation are replicated in source, when that is, operation is mapped on back end, if not having Activity Calculation institute on this back end
Need data source, need first to replicate by data source；The high data source of safety requirements is stored in the low back end of safe class
On must encrypt；Gather respective node load information and submit to control node during Job execution；
Delay machine testing node：Carry out periodic back end to delay machine fault detect, the machine back end ID that will delay feeds back to
Control node.
Control node, using the operation based on security constraint and load balance and back end mapping method, calculates some
Mapping scheme, selects the minimum scheme of cost to be executed；It is multiple that each data source of scheme cost=setting replicates cost * data source
Each encryption expenses * encryption number of times of number of times+setting processed.
With the job stacking-reso urce matching method of security constraint, comprise the following steps：
Step A, server end control node configure to security；
The current safety facilities of back end, are back end setting safe class；
According to the requirement to security for the data source, it is data source setting safety requirements grade；
According to job requester authority, operation Permission Levels are set；
In step A, safety status classification works based on the security configuration on back end geographic location, back end,
Including fire wall setting, port setting, fail-safe software installation situation, the suggestion to safety status classification for the back end owner；
Step B, server end control node are using the operation based on security constraint and load balance and back end mapping
Method, Map operation is mapped to and meets in safety and the back end of load balance constraint, the operation on each node is pressed preferential
Level is ranked up and executes successively；
Shown security constraint comprises at 2 points：(1) operation can only have permission execution on the back end of execution at it；(2) safety
The data source having high demands must be stored on the low back end of safe class with encrypted state.
Shown load balance refers to control node according to back end computing capability, arranges permissible on each back end
The operation transformation of mapping, on each back end, the operation quantity of mapping not can exceed that specified threshold.
Step C, server end machine monitoring node of delaying is periodically detected back end current state, in real time by fault data section
Point ID feeds back to server end control node；
Step D, server end control node receive fault data node ID, to be currently mapped on this node do not complete
Operation, according to operation and the back end mapping method of security constraint and load balance, operation is remapped to machine data of not delaying
Node.
Using the operation based on security constraint and load balance and back end mapping method, calculate some mapping sides
Case, selects the minimum scheme of cost to be executed；Each data source of scheme cost=setting replicates cost * data source number of copy times
Each encryption expenses * encryption number of times of+setting.It is likely that not having on this back end when Map operation is mapped on back end
Activity Calculation desired data source, needs first to replicate by data source.The process replicating needs to meet security constraint, and that is, safety will
The data source asking high must be stored on the low back end of safe class with encrypted state.Operation is being mapped to by control node
When on back end, it should generation as few as possible replicates and encryption in addition to meeting security constraint and load balance.
As shown in Fig. 2 main step to be processed includes in control node：Setting back end safe class/data source peace
Full requirement etc./operation authority；Operation distribution/reassignment based on security constraint and load balance；
Executed with back end mapping method and minimum scheme according to the operation based on security constraint and load balance
Principle, operation distributes/reassignment scheme is required to meet following condition：
(1) operation can only be assigned to it and have permission on the back end of execution；Job stacking will not be mapped by control node
Do not have on the back end of authority to it；
(2) operation is assigned on the data source place back end that it is calculated as far as possible, reduces data source number of copy times；
If its desired data node is not had on the back end that operation is distributed, data source will be occurred to replicate；In reproduction process, peace
The data source entirely having high demands copies on the low back end of safe class from the high back end of safe class, safe class
Data source first must be encrypted operation by high back end, then the data source after encryption is copied to the low number of safe class
According on node；
(3) if operation cannot be assigned on its data source place back end, avoid safety requirements is high as far as possible
Data source copy on the low back end of safe class, with reduce data source encryption number of times；The data high to safety requirements
Source can be stored on the high back end of safe class with clear-text way；The data source low to safety requirements can be with plaintext side
Formula is stored on the back end that safe class is low or safe class is high；
(4) operation distribution should ensure that the operation quantity on each back end is less than specified threshold, equal with proof load
Weighing apparatus；
As shown in figure 3, back end mainly executes flow process being：(1) receive job request and execute operation：Judge whether to connect
Receive job request.If received, determine whether desired data source whether on this back end, if do not existed, to
Data source place node sends data source duplicate requests and receiving data source copy executes operation and the information that will complete returns to control
Node processed；Without receiving reception job request, then judge (2) whether receiving data source duplicate requests.If received,
Judge whether that data source safety requirements is high and back end safe class of request data source is low, if it is need data source is entered
Row encryption；Data source is sent to request data node.Without receiving or data source is replicated, then judge whether
Delay machine, without generation, then (3) periodically send heartbeat to machine testing node of delaying；Otherwise terminate.
Table 1 is when occurring data source to replicate, and data source cryptographic operation executes table.Back end can be pacified according to back end
Congruent level data source safety requirements grade, decides whether data source cryptographic operation with reference to this table.
When occurring data source to replicate, data source cryptographic operation executes table to table 1
The Map operation authority list of table 2 pretreatment stage
Service end control node configures to security, and the current safety facilities of back end, are back end
Setting safe class；
Have 3 back end in current system, server controls node to the safe class set by these back end,
As shown in table 3.In the present embodiment, safe class only has high and low two kinds of grades.
Table 3
| Back end ID | Safe class | Current state |
| Back end 1 | High | Normally |
| Back end 2 | Low | Normally |
| Back end 3 | High | Normally |
Control node requirement to security according to data source, is data source setting safety requirements grade；
Table 4 shows the safety requirements grade of data source on each back end current.If current data source is all not yet encrypted.
Table 4
| Data source ID | Safety requirements grade | Whether encrypt | Back end ID |
| Data source a | Low | No | Back end 1 |
| Data source b | High | No | Back end 1 |
| Data source c | Low | No | Back end 2 |
| Data source d | Low | No | Back end 2 |
| Data source e | High | No | Back end 3 |
| Data source f | Low | No | Back end 3 |
| Data source g | High | No | Back end 3 |
Control node, according to job requester authority, arranges operation Permission Levels；
Control node receives 10 job request within this cycle.Control node views each operation from job request
Desired data source and the information of requestor, arrange the authority of each job request.The operation that table 5 display control node is currently received
Authority on each back end.0 expression lack of competence, 1 expression has permission；Table 6 shows the data source that each operation is asked
And job state.
Table 5
Table 6
| Operation ID | Data source ID | Job state |
| Operation 1 | Data source c | Do not complete |
| Operation 2 | Data source g | Do not complete |
| Operation 3 | Data source b | Do not complete |
| Operation 4 | Data source d | Do not complete |
| Operation 5 | Data source e | Do not complete |
| Operation 6 | Data source f | Do not complete |
| Operation 7 | Data source g | Do not complete |
| Operation 8 | Data source a | Do not complete |
| Operation 9 | Data source b | Do not complete |
| Operation 10 | Data source c | Do not complete |
Control node, according to back end computing capability, is arranged in the operation quantity that can map on each back end
Limit, as shown in table 7.
Table 7
| Back end ID | Load threshold |
| Back end 1 | 3 |
| Back end 2 | 5 |
| Back end 3 | 3 |
Server end control node, will using the operation based on security constraint and load balance and back end mapping method
Map operation be mapped to meet safety and load balance constraint back end on, the operation on each node is ranked up and according to
Secondary execution.Control node, according to table 3- table 7 information, based on safety and load balancing, calculates some mapping schemes.Every
In individual scheme, by integer representation, integer value is lower for job priority, represents that priority is higher；In numerical procedure cost, setting
Replicate cost for 10/ every time, encryption expenses are each for 15/.
Scheme one, as shown in table 8.Scheme one needs to carry out 6 times replicating and 3 encryptions altogether, and total cost is 105.
Table 8
Scheme two, as shown in table 9.Scheme two has carried out 5 times replicating and 1 encryption altogether, and total cost is 65.
Table 9
| Operation ID | Back end ID | Whether replicate | Whether encrypt | Priority |
| Operation 1 | Back end 2 | It is | No | 0 |
| Operation 2 | Back end 1 | It is | No | 0 |
| Operation 3 | Back end 3 | It is | No | 0 |
| Operation 4 | Back end 2 | No | No | 1 |
| Operation 5 | Back end 3 | No | No | 1 |
| Operation 6 | Back end 2 | It is | No | 2 |
| Operation 7 | Back end 2 | It is | It is | 3 |
| Operation 8 | Back end 1 | No | No | 1 |
| Operation 9 | Back end 1 | No | No | 2 |
| Operation 10 | Back end 2 | No | No | 4 |
Compared according to cost, scheme two cost is relatively low.Control node determines to carry into execution a plan two.
Carry into execution a plan after two, table 4 changes, as shown in table 10 because data source replicates.
Table 10
| Data source ID | Safety requirements grade | Whether encrypt | Back end ID |
| Data source a | Low | No | Back end 1 |
| Data source b | High | No | Back end 1 |
| Data source b | High | No | Back end 3 |
| Data source c | Low | No | Back end 2 |
| Data source d | Low | No | Back end 2 |
| Data source e | High | No | Back end 3 |
| Data source f | Low | It is | Back end 2 |
| Data source g | High | No | Back end 1 |
| Data source g | High | It is | Back end 2 |
Server end machine monitoring node of delaying is periodically detected back end current state, in real time will be anti-for fault data node ID
Feed server end control node；
During above-mentioned 10 Job executions sometime, server end machine monitoring node of delaying detects back end
1 delays machine.Now table 1 information changes, as shown in table 11
Table 11
| Back end ID | Safe class | Current state |
| Back end 1 | High | Normally |
| Back end 2 | Low | Normally |
| Back end 3 | High | Delay machine |
Current work performance is as shown in table 12.Find that back end 3 has operation 5 not complete above, need again to reflect
It is mapped to the back end of normal operation.
Table 12
| Operation ID | Job state | Back end ID |
| Operation 1 | Complete | Back end 2 |
| Operation 2 | Complete | Back end 1 |
| Operation 3 | Complete | Back end 3 |
| Operation 4 | Do not complete | Back end 2 |
| Operation 5 | Do not complete | Back end 3 |
| Operation 6 | Do not complete | Back end 2 |
| Operation 7 | Do not complete | Back end 2 |
| Operation 8 | Do not complete | Back end 1 |
| Operation 9 | Do not complete | Back end 1 |
| Operation 10 | Do not complete | Back end 2 |
Control node finds according to table 5, and operation 5 only has permission on back end 1 data node 3, current only number
According to node 1 normal operation, so operation 5 desired data source e is copied to node 1 at backup by control node, need not encrypt.And
Operation 5 is remapped to back end 1.
By said process, the present invention realizes the batch processing job stacking scheduling based on security constraint, shorter obtaining
It is ensured that the secure access of data on the basis of batch processing job completion date.
Claims (10)
1. a kind of job stacking-reso urce matching method with security constraint is it is characterised in that comprise the following steps：
Step A, server end control node configure to security；
Step B, server end control node adopt the operation and back end mapping method based on security constraint and load balance,
Map operation is mapped to and meets in safety and the back end of load balance constraint, the operation on each node is according to priority entered
Row sequence simultaneously executes successively；
Step C, server end machine monitoring node of delaying is periodically detected back end current state, in real time by fault data node ID
Feed back to server end control node；
Step D, server end control node receive fault data node ID, to be currently mapped on this node do not complete operation
According to operation and the back end mapping method of security constraint and load balance, operation is remapped to machine data section of not delaying
Point.
2. as claimed in claim 1 job stacking-the reso urce matching method with security constraint it is characterised in that server end
Control node carries out configuration and includes to security：
The current safety facilities of back end, are back end setting safe class；
According to the requirement to security for the data source, it is data source setting safety requirements grade；
According to job requester authority, operation Permission Levels are set.
3. as claimed in claim 1 job stacking-the reso urce matching method with security constraint it is characterised in that shown safety
Constraint comprises at 2 points：（1）Operation can only have permission execution on the back end of execution at it；（2）The high data source of safety requirements must
Must be stored on the low back end of safe class with encrypted state.
4. as claimed in claim 1 job stacking-the reso urce matching method with security constraint it is characterised in that shown load
Balance refers to control node according to back end computing capability, arranges in the operation quantity that can map on each back end
Limit, on each back end, the operation quantity of mapping not can exceed that specified threshold.
5. as claimed in claim 1 job stacking-the reso urce matching method with security constraint it is characterised in that using being based on
The operation of security constraint and load balance and back end mapping method, calculate some mapping schemes, select the minimum side of cost
Case is executed.
6. as claimed in claim 1 job stacking-the reso urce matching method with security constraint it is characterised in that Map operation is reflected
When being mapped on back end, if Activity Calculation desired data source is not had on this back end, need first that data source is copied
Come；The process replicating needs to meet security constraint.
7. as claimed in claim 1 job stacking-the reso urce matching method with security constraint it is characterised in that to safety will
The data source asking high can be stored on the high back end of safe class with clear-text way；The data source low to safety requirements can
To be stored on the back end that safe class is low or safe class is high with clear-text way.
8. as claimed in claim 5 job stacking-the reso urce matching method with security constraint it is characterised in that scheme cost=
Each data source of setting replicates each encryption expenses * encryption number of times of cost * data source number of copy times+setting.
9. a kind of job stacking with security constraint-resource matched system is it is characterised in that include control node, back end
With machine testing node of delaying：
Control node：Receive the operation from user and receive the security constraint from machine testing node of delaying and machine information of delaying；
After control node receives machine information of delaying, according to fault data node ID, to the operation being currently mapped on this node according to safety
The operation of constraint and load balance and back end mapping method, operation is remapped to machine back end of not delaying；
Control node configures to security：The current safety facilities of back end, are back end setting safety etc.
Level；According to the requirement to security for the data source, it is data source setting safety requirements grade；According to job requester authority, arrange
Operation Permission Levels；Using the operation based on security constraint and load balance and back end mapping method, operation is mapped to
Meet in safety and the back end of load balance constraint, carry out the job scheduling/readjustment based on security constraint and load balance
Degree work；
Back end：The safety level information of each node is provided to control node, is responsible for after receiving the operation of control node distribution
Prioritization is carried out to the operation in respective waiting list and executes, necessary data source is carried out according to operation distribution condition multiple
System operation and cryptographic operation, when that is, operation is mapped on back end, if do not have Activity Calculation requisite number on this back end
According to source, need first to replicate by data source；The high data source of safety requirements is stored in must on the low back end of safe class
Must encryption；Gather respective node load information and submit to control node during Job execution；
Delay machine testing node：Carry out periodic back end to delay machine fault detect, the machine back end ID that will delay feeds back to control
Node.
10. as claimed in claim 9 the job stacking with security constraint-resource matched system it is characterised in that described control
Node, using the operation based on security constraint and load balance and back end mapping method, calculates some mapping schemes, choosing
Select the minimum scheme of cost to be executed；Each data source of scheme cost=setting replicates cost * data source number of copy times+setting
Each encryption expenses * encryption number of times.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610749981.9A CN106406990B (en) | 2016-08-26 | 2016-08-26 | A kind of job stacking-reso urce matching method and system with security constraint |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610749981.9A CN106406990B (en) | 2016-08-26 | 2016-08-26 | A kind of job stacking-reso urce matching method and system with security constraint |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106406990A true CN106406990A (en) | 2017-02-15 |
| CN106406990B CN106406990B (en) | 2018-11-09 |
Family
ID=58002466
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610749981.9A Expired - Fee Related CN106406990B (en) | 2016-08-26 | 2016-08-26 | A kind of job stacking-reso urce matching method and system with security constraint |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106406990B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107122849A (en) * | 2017-04-19 | 2017-09-01 | 西安电子科技大学 | Product checking total complete time minimization method based on SparkR |
| CN109634727A (en) * | 2018-11-14 | 2019-04-16 | 中车青岛四方机车车辆股份有限公司 | Data processing method and device |
| CN110125939A (en) * | 2019-06-03 | 2019-08-16 | 湖南工学院 | A kind of method of Robot Virtual visualization control |
| CN114265675A (en) * | 2021-12-08 | 2022-04-01 | 阿里巴巴（中国）有限公司 | Task scheduling method, system, equipment and storage medium for asynchronous training |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102769615A (en) * | 2012-07-02 | 2012-11-07 | 北京大学 | Task scheduling method and system based on MapReduce mechanism |
| CN103294558A (en) * | 2013-05-29 | 2013-09-11 | 北京大学 | MapReduce scheduling method supporting dynamic trust evaluation |
| CN103414761A (en) * | 2013-07-23 | 2013-11-27 | 北京工业大学 | Mobile terminal cloud resource scheduling method based on Hadoop framework |
| CN105022666A (en) * | 2014-04-24 | 2015-11-04 | 中国电信股份有限公司 | Method, device and system for controlling MapReduce task scheduling |
-
2016
- 2016-08-26 CN CN201610749981.9A patent/CN106406990B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102769615A (en) * | 2012-07-02 | 2012-11-07 | 北京大学 | Task scheduling method and system based on MapReduce mechanism |
| CN103294558A (en) * | 2013-05-29 | 2013-09-11 | 北京大学 | MapReduce scheduling method supporting dynamic trust evaluation |
| CN103414761A (en) * | 2013-07-23 | 2013-11-27 | 北京工业大学 | Mobile terminal cloud resource scheduling method based on Hadoop framework |
| CN105022666A (en) * | 2014-04-24 | 2015-11-04 | 中国电信股份有限公司 | Method, device and system for controlling MapReduce task scheduling |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107122849A (en) * | 2017-04-19 | 2017-09-01 | 西安电子科技大学 | Product checking total complete time minimization method based on SparkR |
| CN107122849B (en) * | 2017-04-19 | 2021-01-05 | 西安电子科技大学 | Spark R-based product detection total completion time minimization method |
| CN109634727A (en) * | 2018-11-14 | 2019-04-16 | 中车青岛四方机车车辆股份有限公司 | Data processing method and device |
| CN110125939A (en) * | 2019-06-03 | 2019-08-16 | 湖南工学院 | A kind of method of Robot Virtual visualization control |
| CN110125939B (en) * | 2019-06-03 | 2020-10-20 | 湖南工学院 | Virtual visual control method for robot |
| CN114265675A (en) * | 2021-12-08 | 2022-04-01 | 阿里巴巴（中国）有限公司 | Task scheduling method, system, equipment and storage medium for asynchronous training |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106406990B (en) | 2018-11-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102063336B (en) | Distributed computing multiple application function asynchronous concurrent scheduling method | |
| CN106406990B (en) | A kind of job stacking-reso urce matching method and system with security constraint | |
| CN102137133B (en) | Method and system for distributing contents and scheduling server | |
| CN104601664B (en) | A kind of control system of cloud computing platform resource management and scheduling virtual machine | |
| CN103713956B (en) | Method for intelligent weighing load balance in cloud computing virtualized management environment | |
| CN103927231B (en) | The energy optimization data set distribution method that a kind of data-oriented processes | |
| CN102508714A (en) | Green-computer-based virtual machine scheduling method for cloud computing | |
| CN104239123A (en) | Campus-cloud-platform-oriented virtual machine management dispatching method and system | |
| CN103763365A (en) | Method and system for load balancing of metadata service under cloud storage | |
| CN104023042B (en) | Cloud platform resource scheduling method | |
| CN104536804A (en) | Virtual resource dispatching system for related task requests and dispatching and distributing method for related task requests | |
| CN103957280A (en) | Connection allocation and scheduling method of sensor network in Internet of things | |
| CN105786909A (en) | Message queue backlog load self-adaptive application triggering method and system | |
| CN106681656A (en) | Dynamically-adjustable cloud storage system fault tolerance method | |
| Ahmed et al. | A hybrid and optimized resource scheduling technique using map reduce for larger instruction sets | |
| CN104156505A (en) | Hadoop cluster job scheduling method and device on basis of user behavior analysis | |
| CN102098223B (en) | Method, device and system for scheduling node devices | |
| Biswas et al. | A novel resource aware scheduling with multi-criteria for heterogeneous computing systems | |
| JP5327314B2 (en) | Event processing system, event processing method, local system, dispatcher, and program storage medium | |
| CN118193188A (en) | Dynamic load balancing system and method based on virtualization platform | |
| CN108469990A (en) | A kind of parallel calculating method and system | |
| KR101451781B1 (en) | System and method for device duplication of virtualization system in virtual machine environment | |
| Barazandeh et al. | Two hierarchical dynamic load balancing algorithms in distributed systems | |
| Mishra et al. | A memory-aware dynamic job scheduling model in Grid computing | |
| CN106899392B (en) | Method for carrying out fault tolerance on instantaneous fault in EtherCAT message transmission process |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20181018 Address after: 411105 Yuhu District, Xiangtan, Hunan Applicant after: Xiangtan University Address before: 210096 Shuangjing lane, Xuanwu District, Nanjing, Jiangsu 7 Applicant before: Li Yibo |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20181109 Termination date: 20190826 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |