JP4286481B2 - Clustering system - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a clustering system that achieves high-speed processing by combining relatively inexpensive computers, and more particularly to a clustering system that improves usability for both users and administrators.
[0002]
[Prior art]
There is known a clustering system that uses a plurality of computers, in particular, a relatively inexpensive computer such as a personal computer, and realizes a processing speed comparable to that of a very expensive supercomputer as a whole. The conventional clustering system has the following features.
-The administrator etc. registers all the computers that make up the cluster in advance.
-An administrator or the like installs an execution program in advance on a registered computer.
-Start and execute the execution program on each computer.
-A clustering system provides a library for inter-program communication. The user himself / herself performs programming for performing communication using this library, and each execution program distributes input data required and collects data output by each program.
[0003]
[Problems to be solved by the invention]
However, the conventional clustering system has the following problems. That is, as described above, it is necessary for an administrator or the like to register all the computers constituting the clustering system in a setting file or the like in advance, and maintenance and management of the setting file is troublesome. In addition, the administrator or the like has to install the execution program in advance on all the computers registered in the setting file, which is troublesome and easily causes work errors. Furthermore, since it is necessary for the user himself to perform programming for performing communication between programs, the cost of creating the program is increased.
[0004]
SUMMARY OF THE INVENTION An object of the present invention is to provide a clustering system that reduces the user's programming effort and each computer constituting the clustering system.
[0005]
Another object of the present invention is to provide a clustering system that reduces the labor involved in management and each computer that constitutes the clustering system.
[0006]
Another object of the present invention is to efficiently distribute processing in order to improve overall processing performance.
[0007]
[Means for Solving the Problems]
A clustering system according to one aspect of the present invention is a clustering system having one master machine and a plurality of slave machines connected to the master machine via a network.
The master machine is
A slave management means for storing a communication address of each slave machine and a status indicating a job execution state;
Program storage means for storing one or more job programs for executing a predetermined job;
Program transmission means for transmitting the job program to any of the slave machines;
Among the slave machines managed by the slave management means, the job waiting for execution is in a standby state in which the registered status can be assigned and executed , and the job is required to be executed. Job allocation processing means that assigns priority from the slave machine with high processing capacity, using the average value of the required time as an index ,
Data storage means for storing a file for storing data accessed by the job ;
When an open notification including a job ID and a file name for a file stored in the data storage means is received from the slave machine, the file corresponding to the file name specified in the open notification is opened, and the slave is opened for the opened file. I / O processing means for performing read or write requested from the machine, receiving a close notification including a job ID and a file name, closing the file corresponding to the file name specified in the close notification, and returning the result With
Each slave machine is
Means for receiving the job program;
An input / output processing means for processing a data input / output request of a job executed by the received job program ,
When the input / output processing unit of each slave machine detects a data input / output request of the job being executed, the input / output processing unit notifies the master machine of a file open including a job ID and a file name, and issues a read or write request from the job. When received, the master machine is notified of a read or write request, and when a file close request is received from a job, the master machine is notified of a file close including a job ID and a file name.
[0008]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0009]
FIG. 1 shows an overall configuration of a clustering system 1 that is an embodiment to which the present invention is applied. The system 1 includes a master machine 10 and a plurality of slave machines 20 that are connected to each other via a network 9. Each slave machine 20 executes assigned processing. The master machine 10 manages tasks executed by the slave machine 20 and the present system. The clustering system 1 according to the present embodiment includes only one master machine 10, but may include two or more master machines 10.
[0010]
Each of the master machine 10 and the slave machine 20 is configured by, for example, a general-purpose computer, and individual components or functions in each machine described below execute, for example, a computer program on the computer hardware. Is realized.
[0011]
The master machine 10 includes a communication processing unit 11, an input / output processing unit 12, a slave management unit 13, a job registration processing unit 14, a job assignment processing unit 15, and a storage unit 17. The storage unit 17 includes a program area 17a for storing a job execution program, which will be described later, and a data area 17b for storing data used in the job. Further, the master machine 10 is connected to a display device 7 such as a CRT or a liquid crystal display and an input device 8 such as a keyboard or a pointing device.
[0012]
The communication processing unit 11 automatically detects the slave machine 20 connected to the network 9 and registers the detected communication address of the slave machine 20 in the slave management unit 13. For example, the communication processing unit 11 broadcasts an inquiry for confirming the slave machine 20 existing on the network and its status, and receives a reply from the slave machine 20 in response thereto. Then, the communication address of the slave machine 20 and the status of the slave machine 20 included in this reply are registered in the slave management unit 13. The communication processing unit 11 performs data transmission / reception and the like with the slave machine 20 registered in the slave management unit 13.
[0013]
Further, when the communication processing unit 11 receives a voluntary notification from the slave machine 20 regarding the communication address and status, the communication processing unit 11 updates information regarding the slave machine 20 in the slave management unit 13.
[0014]
The slave management unit 13 manages the slave machine 20 connected to the network. Specifically, the slave management unit 13 has a slave management table 40 as shown in FIG. The slave management table 40 associates the communication address (here, IP address) 41 of the slave machine 20 connected to the network 9 with the status 42 of each slave machine 20. The status 42 is “executing” indicating that the job program is assigned and the processing is being executed, and “waiting” in which the job can be assigned and executed instead of executing the job. "Transition" that indicates that the pre-processing for executing the job program or post-processing after execution is being performed and the state is being changed from being executed to waiting or from waiting to being executed. There is "inside".
[0015]
The input / output processing unit 12 performs processing related to input / output of the storage unit 17. In particular, when an access instruction for a file stored in the data area 17b is received from the slave machine 20, the file is opened / closed, read / written, etc. according to the instruction, and the result is returned. Details of file access will be described later.
[0016]
The task / job registration processing unit 14 includes a job management table 50 (see FIG. 4), accepts task and job registration, and stores them in the job management table 50. Here, the task is composed of one or more jobs. One job is implemented by one job program. The registered execution type job program is stored in the program area 17a. Each job executes a predetermined process, whereby a predetermined function (task) is achieved as a whole. In the present embodiment, one job program is realized by one Java (registered trademark) program. In the system 1, for example, each of a plurality of job programs constituting one task is executed by a separate slave machine 20 to perform parallel distributed processing, or a single job program constituting one task is executed. Multiple slave machines 20 can be executed to process a large amount of data simultaneously.
[0017]
The task registration is performed by causing the display device 7 to display a task / job registration screen 100 shown in FIG. The registration screen 100 includes areas 101 to 107 for receiving input of the following items in order to receive input in units of job programs. That is, the registration screen 100 accepts input of items of a task name 101, a preceding task name 102, a job program execution program name 103, a main class 104, a VM parameter 105, a job program argument 106, and a runtime directory 107. . The user inputs the task name 101 to which the job program belongs. If there is a task to be executed prior to the task to which this job belongs, the user inputs the task name in the preceding task name 102. The user inputs an execution format program name including a path in the execution program name 103. Since the present embodiment shows a case where the implementation is performed using Java, the main class 104 and the VM parameter 105 are designated as necessary.
[0018]
Each item input from the registration screen 100 is registered in the job management table 50 shown in FIG. The job management table 50 includes a job ID 51, a task name 52, an execution program name 53, an execution condition (main class, argument, VM parameter, and runtime directory) 54, a preceding task name 55, a job status 56, As an item. The job ID 51 is identification information for uniquely identifying the execution program, and is automatically generated when it is registered in the job management table 50. The job status 56 includes “waiting” waiting for machine assignment, “executing” processing in any of the slave machines 20, and “finished” indicating that the processing has been completed. Note that “end” includes “end (normal)” that ends normally and “end (abnormal)” that ends abnormally.
[0019]
The job allocation processing unit 15 determines which slave machine 20 is to execute a job waiting for execution. That is, when there is a job waiting for machine assignment, the job assignment processing unit 15 refers to the slave management table 40 and requests the “standby” slave machine 20 to execute the job. A detailed processing procedure at this time will be described later.
[0020]
Each slave machine 20 includes a communication processing unit 21 and an input / output processing unit 22. Each slave machine 20 executes the job program transmitted from the master machine 10.
[0021]
The communication processing unit 21 transmits and receives data and the like with the master machine 10. For example, the communication processing unit 21 broadcasts a notification of its own communication address and status to the network 9 in order to notify the master machine 10 of its existence. This notification can be performed at any time, but is preferably performed when the slave machine 20 is activated.
[0022]
The communication processing unit 21 may receive this address and status notification from another slave machine 20, but does not respond to this.
[0023]
Further, the communication processing unit 21 returns its own communication address and status in response to an inquiry from the master machine 10.
[0024]
The input / output processing unit 22 performs processing for a job being executed on the slave machine 20 to access the data area 17 b of the master machine 10. Hereinafter, the file access process will be described in detail together with the process performed by the input / output processing unit 12 of the master machine 10.
[0025]
When the input / output processing unit 22 of the slave machine 20 detects the data input / output request of the job being executed, the input / output processing unit 22 notifies the master machine 10 of the opening of the file. FIG. 5 shows a format example when notifying this file open (or close). That is, the file open / close notification 60 includes the address 61 of the master machine that is the transmission destination, the address 62 of the slave machine that is the transmission source, the job ID 63, the file name 64 to be accessed, and the open / close type 65. Including. When notifying the file open, the open / close type 65 is “open”, and when notifying the file close, the open / close type 65 is “closed”. The address 61 of the master machine is notified from the master machine 10 when the job program is started (details will be described later).
[0026]
The input / output processing unit 12 of the master machine 10 opens the file having the file name 64 when the open / close type 65 of the open / close notification 60 is “open”.
[0027]
When the input / output processing unit 22 of the slave machine 20 receives a read or write request from a job, it notifies the master machine 10 of the read or write request as it is.
[0028]
The input / output processing unit 12 of the master machine 10 reads or writes the opened file in accordance with the notified read or write request.
[0029]
When the input / output processing unit 22 of the slave machine 20 receives a file close request from the job, the open / close type 65 is set to “closed” and a file open / close notification 60 is transmitted to the master machine 10.
[0030]
The input / output processing unit 12 of the master machine 10 closes the file with the file name 64 when the open / close type 65 of the open / close notification 60 is “closed”.
[0031]
In the present embodiment, the input / output processing unit 12 and the input / output processing unit 22 are provided as, for example, a Java input / output library.
[0032]
As described above, a job executed on the slave machine 20 can access data on the master machine 10 by requesting data access to the input / output processing unit 22. Thereby, even if each job program is programmed without being conscious of distributed processing, it can operate correctly on the clustering system 1 of the present embodiment. That is, the user of the present system 1 does not need to program the communication function between programs by himself / herself, and the cost of creating a program is reduced.
[0033]
Next, an example of the procedure of distributed processing in this embodiment will be described using the flowchart shown in FIG.
[0034]
The job allocation processing unit 15 refers to the job management table 50 of the job registration processing unit 14 and checks whether there is a job waiting to be allocated to a machine whose job status 56 is “waiting” (waiting for machine allocation) (see FIG. S11). If there is a job waiting to be executed (S11: Yes), the job is executed according to the following procedure. If the preceding task 55 is set, it is a condition that the preceding task has been completed.
[0035]
First, the job allocation processing unit 15 refers to the slave management table 40 of the slave management unit 13 and checks whether there is a “standby” slave machine 20 that is not executing a job (S12). When there is a standby slave machine 20 (S12: Yes), the job allocation processing unit 15 instructs the slave management unit 13 to set the status 42 of the slave machine 20 to “in transit”. The job assignment processing unit 15 acquires the communication address 41 of the slave machine 20 and checks whether or not the execution program 53 of the job to be executed has already been distributed to the slave machine 20 (S13). Checking whether the execution program has been distributed is performed as follows, for example. That is, when the execution program is distributed to the slave machine 20, the slave machine 20 stores the execution program in a storage device (not shown). The job allocation processing unit 15 refers to a storage device (not shown) of the slave machine 20 and confirms whether or not they are the same execution program using the file name and time stamp of the execution program.
[0036]
When the execution program is not distributed (S13: No), the job allocation processing unit 15 transmits the execution program to the slave machine 20 (S14). The format at that time is shown in FIG. That is, the transmission data 70 of the execution program includes the address 71 of the slave machine that is the transmission destination, the address 72 of the master machine that is the transmission source, the job ID 73, the execution program 74, the main class 75, and the argument 76. Including. The slave machine 20 that has received the transmission data 70 stores these data in a storage device (not shown).
[0037]
The slave machine 20 receives the notification from the job assignment processing unit 15 and activates the execution program to execute the job (S15). The execution program start notification to the slave machine 20 includes the communication address of the master machine 10. In the slave machine 20, the communication address of the master machine 10 is held in a storage device (not shown) until the execution of the program is completed.
[0038]
The job allocation processing unit 15 notifies the slave management unit 13 that the execution program has been started, together with the communication address of the slave machine 20 that has started the execution program. In response to the notification from the job allocation processing unit 15, the slave management unit 13 changes the status 42 of the corresponding slave machine 20 to “in execution”.
[0039]
When the job ends, the slave machine 20 notifies the master machine 10 of the end of the job. FIG. 7B shows the format of the job end notification. The end notification 80 includes an address 81 of the master machine that is the transmission destination, an address 82 of the slave machine that is the transmission source, a job ID 83, and an end state 84 that indicates whether the end is normal end or abnormal end. After the job completion notification, the communication address of the master machine 10 held in a storage device (not shown) is discarded.
[0040]
The job allocation processing unit 15 notifies the slave management unit 13 and the job registration processing unit 14 of the slave machine address 82, the job ID 83, and the end state 84 included in the end notification. Upon receiving this notification, the slave management unit 13 changes the status 42 of the corresponding slave machine 20 to “waiting”. Upon receiving this notification, the job registration processing unit 14 changes the job status 56 of the corresponding job to the state indicated by the end state 84.
[0041]
As a result, a task composed of a plurality of jobs can be distributed to the slave machines and executed for each job. In particular, in the present embodiment, execution programs are registered in a batch on the master machine, and when executing a task, the execution programs are automatically distributed as necessary, and jobs are distributed and executed. Thereby, in this embodiment, unlike the conventional clustering system, it is not necessary to distribute the execution program to all the slave machines in advance.
[0042]
Further, the job assignment processing unit 15 may assign priorities to each slave machine 20 and assign jobs according to the priorities. The method for determining the priority order is arbitrary, but an example is shown below.
[0043]
First, the first method measures the processing capacity of a slave machine, and as a result, assigns jobs with priority from slave machines with high processing capacity. As an index indicating the processing capability of the slave machine, an average value of the time required for executing the job is used. As the required time, for example, the time from when the job allocation processing unit 15 instructs the slave machine to execute a job until it receives a job end notification is used. The processing time in this case is affected by the communication speed of the network 9, but can be considered as the total processing capacity of the slave machine as seen from the master machine. As the required time, the processing time for each slave machine for a specific job may be used. The specific job is a job that has been repeatedly executed and has a record of execution on a plurality of slave machines.
[0044]
Next, in the second method, the communication speed between the master machine and each slave machine is measured, and as a result, a job is preferentially assigned from a slave machine having a high communication speed. The communication speed is calculated by measuring the amount of data communicated by the job allocation processing unit 15 and the time required for communication.
[0045]
According to the present embodiment, jobs can be efficiently distributed in order to improve the processing performance of the system and process tasks at high speed.
[0046]
The above-described embodiments of the present invention are examples for explaining the present invention, and are not intended to limit the scope of the present invention only to those embodiments. Those skilled in the art can implement the present invention in various other modes without departing from the gist of the present invention.
[0047]
【The invention's effect】
According to the present invention, the user's programming effort is reduced. In addition, management effort is reduced. Furthermore, the processing performance can be improved by efficiently distributing the processing.
[Brief description of the drawings]
FIG. 1 is a diagram showing an overall configuration of a clustering system 1 which is an embodiment to which the present invention is applied.
FIG. 2 is a diagram illustrating an example of a slave management table 40;
FIG. 3 is a diagram showing an example of a task / job registration screen 100;
4 is a diagram illustrating an example of a job management table 50. FIG.
FIG. 5 is a diagram illustrating an example of a format of a file open / close notification 60;
FIG. 6 is a flowchart illustrating a procedure of job distribution processing;
7A is a format example of transmission data 70 when distributing an execution program, and FIG. 7B is a format example of a program end notification 80;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Clustering system, 10 ... Master machine, 11 ... Communication processing part, 12 ... Input / output processing part, 13 ... Slave management part, 14 ... Job registration processing part, 15 ... Job allocation processing part, 17 ... Storage part, 20 ... Slave machine, 21 ... communication processing unit, 22 ... input / output processing unit, 40 ... slave management table, 50 ... job management table, 60 ... file open / close notification, 70 ... send data, 80 ... end notification, 100 ... task Job registration screen.

Claims (3)

In a clustering system having one master machine and a plurality of slave machines connected to the master machine via a network,
The master machine is
A slave management means for storing a communication address of each slave machine and a status indicating a job execution state;
Program storage means for storing one or more job programs for executing a predetermined job;
Program transmission means for transmitting the job program to any of the slave machines;
Among the slave machines managed by the slave management means, the job waiting for execution is in a standby state in which the registered status can be assigned and executed , and the job is required to be executed. Job allocation processing means that assigns priority from the slave machine with high processing capacity, using the average value of the required time as an index ,
Data storage means for storing a file for storing data accessed by the job ;
When an open notification including a job ID and a file name for a file stored in the data storage means is received from the slave machine, the file corresponding to the file name specified in the open notification is opened, and the slave is opened for the opened file. I / O processing means for performing read or write requested from the machine, receiving a close notification including a job ID and a file name, closing the file corresponding to the file name specified in the close notification, and returning the result With
Each slave machine is
Means for receiving the job program;
An input / output processing means for processing a data input / output request of a job executed by the received job program ,
When the input / output processing unit of each slave machine detects a data input / output request of the job being executed, the input / output processing unit notifies the master machine of a file open including a job ID and a file name, and issues a read or write request from the job. When receiving, the master machine is notified of a read or write request, and when a file close request is received from a job, the master machine is notified of a file close including a job ID and a file name. system. The clustering system according to claim 1,
The job allocation processing means checks whether or not the execution program of the job to be executed has already been distributed to the slave machine to which the job is allocated, and transmits the execution program to the slave machine when it has not been distributed. A clustering system characterized by The clustering system according to claim 2,
The job allocation processing means uses the file name and time stamp of the execution program stored in the slave machine to check whether the execution program to be distributed is the same program, so that the execution program has already been distributed. A clustering system characterized by checking whether or not

Images