JP5285044B2 - Cluster system recovery method, server, and program - Google Patents

Description

  The present invention relates to a cluster system recovery method, a server, and a program, and more particularly to a cluster system for detecting and recovering from a failure of a failed server in a system that operates a plurality of server systems in cooperation as a single system. The present invention relates to a recovery method, a server, and a program.

  As the importance of services increases, so does the demand for systems with low downtime. For this reason, a highly available cluster software such as Heartbeat and Pacemaker that enables continuation of services by building a redundant cluster system with multiple servers and automatically switching servers when some failure occurs. It has been developed (see Non-Patent Document 1).

  High-availability cluster software monitors resources, networks, shared disks, etc. on the server. If a failure is detected on a server that is in service, the server is switched to a standby server in advance and the service is continued.

  FIG. 1 shows a schematic diagram of a cluster system using high-availability cluster software. The cluster system includes a plurality of servers (active machine and spare machine) connected to a network and a shared disk that is shared and used by the plurality of servers.

  The active machine and the spare machine each have an operating system (OS), high-availability cluster software, and resources that are components necessary for providing a service. The high-availability cluster software detects the occurrence of a failure in the active machine and automatically switches to a spare machine when a failure occurs. The service operating status, resource operating status, and failure status of the server are stored in the internal disk status management information storage unit, and detailed information such as failure location is stored in the internal disk log storage unit, including failure status Server cluster state management information is stored in a state management information storage unit.

  The active machine and the spare machine are connected to a network called a service LAN, and provide services based on resources to clients. In addition, the current machine and the spare machine are connected to a network called an interconnect LAN, and exchange information such as service operating status, resource operating status, and fault status in the server. Furthermore, the current machine and the spare machine are connected to a network called a management LAN, and can accept commands from the maintenance terminal.

  In addition, the active machine and the spare machine can include a forced power-off function for forcibly turning off the power of other servers when a failure occurs in the high-availability cluster software. The forced power-off function cuts off the power of the other server by sending an instruction to turn off the power to the hardware control board of the other server via the management LAN.

  The shared disk is a storage device that stores data used for service provision in order to maintain service consistency. With the shared disk, the service can be continued using the same data even after switching from the current machine to the spare machine.

  As described above, the failure of the resource is monitored by the high availability cluster software. Therefore, when a resource failure occurs, the service can be continued with the spare machine. After the system is switched to the spare machine, the service is continued on the spare machine (see Patent Document 1).

Japanese Patent No. 4353005, “System Switching Method for Cluster Configuration Computer System”

Mitsunori Mitsuno et al., “Development of OSS Middle Heartbeat to Improve Service Availability”, NTT Technical Journal, March 2009, pages 46-49

  In the above conventional technology, the high availability cluster software can detect the occurrence of a failure in the active machine and can automatically switch to the spare machine when a failure occurs. It is assumed that it has not occurred. For example, as shown in FIG. 2, when at least one of the internal disk failure, the network failure, and the shared disk failure has occurred on the spare device side, but the service is operating normally on the active device side. There is no problem, but if any failure occurs on the active machine side, even if you try to switch the system to the spare machine side, the system switchover cannot be performed, and the active machine is implemented on the active machine side. It will be in the state of "SBY [Transitioning]" waiting for the stop processing to finish normally.

  In addition, as shown in FIG. 3, if there is a failure of the forced power-off function in either the active machine or the standby machine, the forced power-off function of the cluster software cannot be executed normally, but the service of the active machine There is no effect on the operating status, and even if a system switching process occurs, it can be switched to a spare machine. However, if the service stoppage fails during the system switchover, the forced power-off function is not executed, and there is a problem that the active machine enters a state of “SBY [in transit]” as described above.

  The present invention has been made in view of the above points, and a server operating as an active machine of a cluster system is in a state where system switching has occurred due to a failure or the like, but system switching has not been completed due to a resource stop failure or the like. An object of the present invention is to provide a cluster system recovery method, server and program capable of avoiding this.

In order to achieve the above object, a cluster system recovery method of the present invention (claim 1) includes failure monitoring means for monitoring a failure state, and a cluster indicating service operating states of active and standby devices based on the failure state. State management means for managing the state, and the status of the service in operation (ACT), the state that can be changed to ACT, or the state that can be changed to ACT is stored (SBY: online) and the cluster state and failure state information are stored. state management information storage means and the working machine, each containing a spare machine, and, in a cluster system constituted by the shared disk shared by the developing for machine and said spare machine, the developing for machine incorporated in a cluster configuration, the A cluster system recovery method in a state where it is unknown whether the spare machine can transition to ACT (SBY: online)
A fault status acquisition step in which the status check means of the active machine acquires fault status information from the status management information storage means via the status management means of the active machine;
If the failure status information indicates a failure of the forced power-off function, a failure of the forced power-off function on the spare unit side is suspected, so a forced power-off function error output step that outputs an error to the maintenance terminal,
When it is instructed by the maintenance terminal on the spare unit in an unknown state (SBY: online) whether the transition to ACT is possible or not, the continuity confirmation unit of the spare unit will instruct the hardware control unit on the active unit side. If the continuity is confirmed and the continuity is confirmed, a failure of the forced power-off function monitoring resource of the state management information storage means is detected as a failure of the forced power-off function of the spare machine. A failure count clearing step for transitioning to a state (SBY: online) that clears the count and enables system switching from the active machine.

  In the present invention (Claim 2), when the failure state information acquired in the failure state acquisition step indicates a network failure, the continuity from the continuity confirmation means on the spare unit side to the router is confirmed. In the case of failure, a network error output step of outputting an error to the maintenance terminal is further provided.

  Further, the present invention (Claim 3) is a disk error output step for outputting an error to the maintenance terminal when the failure state information acquired in the failure state acquisition step indicates a failure of the shared disk or the built-in disk. It has further.

The present invention (Claim 4) includes a failure monitoring unit for monitoring a failure state, a state management unit for managing a cluster state indicating a service operation state of the active machine and the spare unit based on the failure state, and a state during service operation (ACT), state management information storage means for storing cluster state and failure state information including a state that can be changed to ACT or whether it is possible to change to ACT (SBY: online) and a spare And a shared disk shared by the working machine and the spare machine, the working machine is incorporated into the cluster configuration , and it is unknown whether the spare machine can transition to ACT (SBY: online) If the cluster system recovery system,
The current machine is
Failure state acquisition means for acquiring failure state information from the state management information storage means via the state management means;
If the failure status information indicates a failure of the forced power-off function, a failure of the forced power-off function on the spare unit side is suspected, so a forced power-off function error output means that outputs an error to the maintenance terminal,
Have
The spare machine is
When it is in an unknown state (SBY: online) whether it is possible to transition to ACT, when continuity confirmation is instructed from the maintenance terminal, continuity confirmation means for confirming continuity to the hardware control means on the active machine side,
When continuity is confirmed by the continuity confirmation means, the number of failure of the forced power-off function monitoring resource of the state management information storage means is cleared as a failure of the forced power-off function on the spare machine side, and the system from the active machine Fault number clearing means for transitioning to a state (SBY: online) that enables switching.

In addition, the active machine of the present invention (Claim 5), when the failure state information acquired by the failure state acquisition means indicates a network failure, router conduction means for confirming conduction to the router,
Network error output means for outputting an error to the maintenance terminal when the connection by the router conduction means fails.

In addition, the working machine according to the present invention (Claim 6) has an error to the maintenance terminal when the failure status information acquired by the failure status acquisition means indicates a failure of the shared disk or the built-in disk. Disk error output means for outputting.

The present invention (Claim 7) includes failure monitoring means for monitoring a failure state, state management means for managing a cluster state indicating a service operation state of the active machine and the spare machine based on the failure state, and a state in which the service is in operation (ACT), state management information storage means for storing cluster state and failure state information including a state that can be changed to ACT or whether it is possible to change to ACT (SBY: online) and a spare And a shared disk shared by the working machine and the spare machine, the working machine is incorporated in the cluster configuration, and it is unknown whether the spare machine can transition to ACT (SBY: online) A server that operates as an active machine when
Failure state acquisition means for acquiring failure state information via the state management means;
If the failure status information indicates a failure of the forced power-off function, a failure of the forced power-off function on the spare unit side is suspected, so a forced power-off function error output means that outputs an error to the maintenance terminal,
If the failure state information acquired by the failure state acquisition means indicates a network failure, the router conduction means for confirming continuity to the router;
A network error output means for outputting an error to the maintenance terminal when the connection by the router conduction means fails;
And disk error output means for outputting an error to the maintenance terminal when the failure state information acquired by the failure state acquisition means indicates a failure of the shared disk or the built-in disk.

The present invention (Claim 8) includes a failure monitoring means for monitoring a failure state, a state management means for managing a cluster state indicating a service operation state of an active machine and a spare machine based on the failure state, and a state during service operation (ACT), state management information storage means for storing cluster state and failure state information including a state that can be changed to ACT or whether it is possible to change to ACT (SBY: online) and a spare And a shared disk shared by the working machine and the spare machine, the working machine is incorporated into the cluster configuration, and it is unknown whether the spare machine can transition to ACT (SBY: online) A server that acts as a spare machine,
When continuity confirmation is instructed from the maintenance terminal, continuity confirmation means for confirming continuity with respect to the hardware control means on the active machine side,
When continuity is confirmed by the continuity confirmation means, the number of failure of the forced power-off function monitoring resource of the state management information storage means is cleared as a failure of the forced power-off function on the spare machine side, and the system from the active machine Fault number clearing means for transitioning to a state (SBY: online) that enables switching.

  The present invention (Claim 9) is a program for causing a computer to function as each means constituting the server according to Claim 7 or 8.

  As described above, according to the present invention, in the state in which the active device of the cluster system is incorporated in the cluster configuration, when a failure occurs and the system is switched to the standby device, whether the network failure is detected from the state management information. Determine whether it is a forced power-off function failure or a shared disk / built-in disk failure. If it is a network failure or a shared disk failure, determine that the spare is in a cluster state that cannot be switched. In the case of a failure of the forced power-off function of the standby system, it is possible to determine that the cluster is in a state where system switchover is possible, so that the fault tolerance can be improved by restoring the forced power-off function. Is possible.

1 is a schematic diagram of a cluster system using high availability cluster software. FIG. It is a figure which shows the failure state of a spare machine. It is a figure which shows the failure of the forced power-off function of an active machine or a spare machine. It is a functional block diagram of the cluster system in one embodiment of this invention. It is a state transition diagram of the cluster state managed by the state management unit. It is an example of the information stored in the state management information storage part in one embodiment of this invention. It is a figure which shows the state after completion | finish of a recovery procedure from the normal operation state in one embodiment of this invention. It is a flowchart of the failure detection procedure in one embodiment of the present invention. It is a figure which shows the operation | movement of step 102 of FIG. 8 in one Example of this invention. It is a figure which shows the operation | movement of step 103 of FIG. 8 in one Example of this invention. It is a figure which shows operation | movement of the step 104 of FIG. 8 in one Example of this invention. It is a figure which shows operation | movement of the step 105 of FIG. 8 in one Example of this invention. It is a figure which shows the operation | movement of step 106 of FIG. 8 in one Example of this invention.

  Embodiments of the present invention will be described below with reference to the drawings.

  Before describing the cluster system and method according to the embodiment of the present invention in detail, first, terms used in the embodiment of the present invention will be described.

  Cluster configuration: A technology for connecting a plurality of servers to each other and making a user or other server providing a service behave as if they are one server as a whole. With a cluster configuration, even if one server fails, the service can be continued throughout the system, and failure repair or replacement can be performed while the service is continuing.

  -Active machine: A server that is in service when a service has started and no failure has occurred in a cluster system.

  Spare machine: A server that takes over services when a failure occurs in the active machine in a cluster system. The spare machine may take over the service of one working machine or may take over the services of a plurality of working machines. That is, the relationship between the active machine and the spare machine may be a one-to-one relationship or an N-to-one relationship.

  High availability cluster software: Software for providing a cluster configuration. High-availability cluster software monitors server failures and performs system switching when a failure occurs.

  ・ Resource: Refers to the components necessary to provide a service. A resource in a cluster configuration refers to an application that is subject to control by the highly available cluster software such as starting, stopping, and monitoring. Resources include databases and the like.

  Cluster state: Refers to the service operating state in the server. The cluster state includes “ACT”, “SBY [online]”, “SBY [standby]”, “SBY [in transit]”, “OUS”, and “NONE”.

  Resource status: This refers to the operating status of resources in the server. The resource state includes a state where the resource is operating on another server, a state where the resource is operating on the local server, a state where the resource is stopped, and a state where the resource is not managed.

  ・ "ACT": Indicates that the service is running on the server. In a cluster configuration, it refers to the state of a server on which resources that provide services such as databases are operating.

  “SBY [online]”: A state in which transition to ACT is possible. In a cluster configuration, when system switching occurs due to a failure or the like, the state of a server that can switch resources from ACT is called.

  “SBY [standby]”: A state where transition to “ACT” is suppressed. In a cluster configuration, this refers to the status of servers that are prevented from transitioning to "ACT" even when system switchover occurs due to a failure.

  ・ "SBY [Transitioning]": Says the status during system switching. In a cluster configuration, this refers to the state of a server that has undergone system switchover due to a failure or the like, but has failed to stop resources and has not yet switched over.

  ・ "OUS": Indicates that the resource is faulty on the server. A state in which a resource failure has occurred in a cluster configuration.

  -“NONE”: A state in which the server is not incorporated in the cluster configuration. The state of a server that is not built into the cluster configuration, such as when the highly available cluster software is stopped.

<Configuration of cluster system>
FIG. 4 shows a functional block diagram of the cluster system in one embodiment of the present invention. The cluster system includes a plurality of servers (current machine 10 and spare machine 20) connected to each other, and a shared disk 30 that is shared and used by the plurality of servers. The active machine 10 and the spare machine 20 provide services to clients via the router 40. Note that the performance of the current machine 10 may be superior to the performance of the spare machine 20. Further, the cluster system may be composed of two or more active machines and one spare machine.

  The active machine 10 includes a resource 101, high availability cluster software 110, an execution control unit 120, a state management information storage unit 119, an operating system (OS) 151, a power supply control unit 153, and a power supply 155.

  The high availability cluster software 110 is a state management unit, and includes a failure monitoring unit 111, a resource start / stop unit 113, a state management unit 115, a forced power-off function unit 116, and a forced power-off function monitoring unit 117. The Specifically, in the present invention, high availability cluster software is used.

  The control execution unit 120 includes a continuity confirmation unit 123, a cluster configuration activation unit 125, a system switching instruction unit 127, a state confirmation unit 131, and a command execution unit 133.

  The spare machine 20 includes a resource 201, high availability cluster software 210, an execution control unit 220, a state management information storage unit 219, an operating system (OS) 251, a power supply control unit 253, and a power supply 255.

  The high availability cluster software 210 is a state management unit, and includes a failure monitoring unit 211, a resource start / stop unit 213, a state management unit 215, a forced power-off function unit 216, and a forced power-off function monitoring unit 217. The

  The control execution unit 220 includes a continuity confirmation unit 223, a cluster configuration activation unit 225, a system switching instruction unit 227, a state confirmation unit 231, and a command execution unit 233.

  Resources 101 and 201 are applications that provide services to clients. Resources 101 and 201 are activated on a server whose cluster status is “ACT” during service operation.

  The failure monitoring units 111 and 211 of the high availability cluster software 110 and 210 monitor the failure state of the server. For example, resources, networks, shared disks / built-in disks, etc. are monitored. The resource is monitored only by the “ACT” server during service operation, but the network and the shared disk / built-in disk are monitored by both the current machine 10 and the spare machine 20. When a failure is detected in the active machine 10, the failure state is stored in the state management information storage unit 119 via the state management unit 115. For example, an error status indicating the number of failures and failure occurrence timing (resource start failure, failure during resource monitoring, resource stop failure) is stored in the state management information storage unit 119 as a failure state. As will be described below, the failure state of the active machine 10 is also stored in the state management information storage unit 219 from the state management unit 115 via the state management unit 215 of the spare machine. Similarly, when a failure is detected in the spare machine 20, the failure state of the spare machine is stored in the state management information storage unit 219 via the state manager 215, and further via the state manager 115 of the active machine 10. And stored in the state management information storage unit 119.

  The resource start / stop units 113 and 213 start and stop resources based on the cluster state and the failure state. When the resources of other servers are stopped when the cluster state of the server is in the state “SBY [online]” where the cluster state can be changed to ACT, the resource start / stop units 113 and 213 start the resources. If a failure occurs when the cluster state of the server is “ACT” during service operation, the resource start / stop units 113 and 213 stop the resource.

  The state management units 115 and 215 manage the cluster state based on the failure state. The status management unit 115 of the active machine 10 and the status management unit 215 of the standby machine 20 exchange information such as the failure status (number of failures, error status) and cluster status stored in the status management information storage unit, and each server Is stored in the state management information storage units 119 and 219.

  The forced power-off function units 116 and 216 monitor the hardware control board of the opposite device (the active device if it is the active device, the active device if it is the standby device) via the maintenance management LAN, and the monitoring is timed out In the state management information storage unit, the number of failures of the opposite device is “1”, and the error status is “2”.

  The forced power-off function monitoring units 117 and 217 monitor the forced power-off function units 116 and 216 in the self-device, and in the case of a process failure of the forced power-off function unit, indicate the number of failures in the state management information storage unit. 1 ”and error status“ 2 ”.

  FIG. 5 shows a state transition diagram of the cluster state managed by the state management units 115 and 215. The cluster state includes “ACT”, “SBY [online]”, “SBY [standby]”, “SBY [in transit]”, “OUS”, and “NONE”. When a resource failure occurs in the “ACT” server, the cluster status changes from “ACT” to “OUS” (T1). When a failure other than resources (failure of network, shared disk, etc.) occurs in the “ACT” server, the cluster status changes from “ACT” to “SBY [Transitioning]” (T2). When the failure state of the “OUS” server is cleared, the cluster state is changed from “OUS” to “SBY [standby]” (T3). When a system switchover occurs due to a failure, etc., and the "SBY [online]" server takes over the service, the cluster status changes from "SBY [online]" to "SBY [in transit]" (T4). ACT "(T5). When the service operation is inhibited on the “ACT” server in order to take over the service from the “ACT” server to another server, the cluster status changes from “ACT” to “SBY [standby]” (T6). When the transition suppression to “ACT” is canceled in the server “SBY [standby]”, the cluster state is changed from “SBY [standby]” to “SBY [standby]” (T7). When the transition to ACT is inhibited in the server “SBY [standby]”, the cluster state is changed from “SBY [standby]” to “SBY [standby]” (T8). Further, when the high availability cluster software is stopped due to the power supply stop, the operating system stop or the high availability cluster software itself, the cluster state becomes “NONE” (T9 to T13). When the high availability cluster software is activated, the cluster state changes from “NONE” to “SBY [online]” (T14). If the high availability cluster software is activated when the cluster status of both the active machine and the standby machine is NONE, the cluster status changes from NONE to ACT (T15).

  The state management information storage units 119 and 219 store a cluster state and a failure state for each server. Specifically, the state management information storage units 119 and 219 store both the information on the active machine 10 and the information on the spare machine 20, respectively, and exchange the information between the state management unit 115 and the state management unit 215. The information stored in the management information storage unit 119 and the information stored in the state management information storage unit 219 are held the same. Note that when referring to the state management information storage units 119 and 219, an identifier such as a server ID that can uniquely identify the server is used.

  FIG. 6 shows an example of information stored in the state management information storage units 119 and 219. The state management information storage units 119 and 219 store a cluster state, the number of failures, an error status, a resource state, and an interface attribute value for each server. The state management information storage units 119 and 219 have, as cluster states, “ACT”, “SBY [online]”, “SBY [standby]”, “SBY [in transit]”, “OUS”, “ Memorize either "NONE". As the number of failures, the number of failures (values 0 to N) is stored. As an error status indicating the failure occurrence timing, any one of a no error state, a state in which resource start has failed, a state in which a failure has been detected during resource monitoring, and a state in which resource stop has failed is stored. As the resource status, any of the status where the resource is operating on another server, the status where the resource is operating on the local server, the status where the resource is stopped, and the status where the resource is not managed is stored. . As an interface attribute value, there is no error “0”, there is a PING error, “Link is failure” is displayed “1”, and there is a disk error “Disk is failure” is displayed “2”. Remember.

  The continuity confirmation unit 123 confirms continuity to the router 40 when the state confirmation unit 131 estimates that the failure location is a network failure. If the continuity check is successful, it is considered a temporary failure due to an instantaneous network interruption. Further, when it is estimated by the state confirmation unit 131 that the failure location is the power supply control unit (hardware control board) 253, the power control unit (hardware control board) 253 of the other server (spare unit 20) Check continuity. PING may be used for these conduction confirmations.

  When the state confirmation unit 231 estimates that the failure location is a failure of the forced power-off function, the conduction confirmation unit 223 confirms conduction to the power control unit 153 of the active machine 10. PING may be used for continuity confirmation. If the continuity check is successful, it is considered a temporary failure due to an instantaneous network interruption.

  The cluster configuration activation unit 125 incorporates the active device 10 into the cluster configuration, and changes the cluster state of the active device stored in the state management information storage unit 119 to a state where the service device can be shifted to service operation. Specifically, the cluster configuration activation unit 125 activates the high availability cluster software 110 of the active machine 10. For example, the cluster configuration activation unit 125 may activate the state management unit 115, and the state management unit 115 may activate the failure management unit 111 and the resource activation / deactivation unit 113. By this activation, the cluster state of the active device 10 stored in the state management information storage unit 219 is also transitioned via the state management unit 215, and the cluster state of the active device 10 becomes “SBY [online]”. The same applies to the cluster configuration activation unit 225 of the spare machine 20.

  The system switching instruction unit 127 instructs system switching from the standby machine 20 to the active machine 10 via the state management unit 115. Specifically, the cluster state of the spare machine 20 stored in the state management information storage unit 119 is shifted to a state where the transition to service operation is suppressed. Due to this transition inhibition, the cluster state of the active machine 10 transitions to service operation. As a result, the cluster status of the standby machine 20 and the active machine 10 stored in the status management information storage unit 219 is also changed via the status management unit 215, and the cluster status of the standby machine 20 becomes “SBY [standby]”. The cluster state of the active machine 10 becomes ACT. Then, the resource 201 of the standby machine 20 is stopped, and the resource 101 of the active machine 10 is started. The system switching instruction unit 227 of the spare machine 20 is the same.

  The state confirmation unit 131 confirms the information stored in the state management information storage unit 119 via the state management unit 115. For example, the cluster status, the number of failures, the error status, and the resource status of both the active machine 10 and the spare machine 20 are confirmed. The same applies to the state confirmation unit 231 of the spare machine 20.

  The operating systems 151 and 251 are basic software for operating the high availability cluster software 110 and 210 and applications on the server.

  In the present embodiment, the power control unit 153 receives an instruction to forcibly turn off the power from the forced power-off function unit 216 of another server (spare unit 20), and turns on the power source 155 that supplies power to the server. Turn off. The same applies to the power control unit 253 of the spare machine.

  First, the failure state of the active machine 10 and the spare machine 20 will be described.

  FIG. 7 shows a state where the restoration procedure is completed from the normal operation state according to the embodiment of the present invention.

  (1) As shown in FIG. 5A, the active machine 10 is in service because the cluster status is “ACT”, and the spare machine 20 is in the “SBY [online]” status. It is in a state where transition to ACT "is possible.

(2) From the above state, as shown in FIG. 2, it is unknown whether or not the spare unit 20 can transition to “ACT” due to a failure of the forced power-off function unit 216 or an interface failure. (FIG. 7B). At this time, since the resource 101 is operating on the active machine 10, no action has occurred. That is, the state of the active machine 10 and the spare machine 20 at this time is
(Current machine)
Cluster status: "ACT"
Number of failures: 0
Error status: 0
Resource status: 1 (Started)
I / F attribute value: 0
(Spare machine)
Cluster status: "SBY [online]"
Number of failures: 0
Error status: 0
Resource status: 0
I / F attribute value: 1 or 2
It is. In the spare machine 20, since the interface (I / F) attribute value of the state management information storage unit 219 is “1” or “2”, it can be estimated that either a network failure or a disk failure occurs. In this case, when a system switching process occurs from the active machine 10 to the standby machine 20, the I / F attribute value is a value other than “0”, so the system switching cannot be executed and the active machine 10 fails. The system switchover occurs due to, for example, the cluster status of "SBY [Transition]", which is the state of the server where the system switchover has not been completed because the resource stop failed.

  Moreover, the case as shown in FIG. 3 can be considered as the state of FIG.

  The example of FIG. 3 is a case where a failure has occurred in the forced power-off function unit of either the active machine 10 or the standby machine 20.

At this time, the status of the active machine 10 and the spare machine 20 is as follows:
(Current machine)
Cluster status: "ACT"
Number of failures: 0
Error status: 0
Resource status: 1 (Started)
I / F attribute value: 0
(Spare machine)
Cluster status: "SBY [online]"
Number of failures: 1
Error status: 2
Resource status: 0
I / F attribute value: 0
It is. In the above, the error status of the spare machine 20 is “2” (error is detected during resource monitoring). In this case, the forced power-off function unit 216 cannot be executed normally, but the service operation status is not affected, and the system can be switched to the spare machine 20 even when a system switching process from the active machine 10 to the spare machine 20 occurs. Is possible. However, if the service stoppage fails during system switching, the processing of the forced power-off function unit 216 is not executed, and the active machine 10 enters a cluster state of “SBY [in transit]” as in FIG.

  In the state as described above (FIG. 7B), the present invention performs failure recovery processing for only the spare machine 20, so that when system switching occurs from the active machine 10, as shown in FIG. 7C. The spare machine 20 is changed to the state “SBY [online]” that can be changed to “ACT”.

  FIG. 8 is a flowchart of the failure detection procedure in one embodiment of the present invention.

  Step 101) Log in to the active machine 10 and the spare machine 20 from the maintenance terminal. At this time, if the login to the spare machine 20 is successful, the process proceeds to step 102, and if it is unsuccessful, the process proceeds to step 103. The states of the current machine 10 and the spare machine 20 at this time are as follows.

(Current machine)
Cluster status: "ACT"
Number of failures: 0
Error status: 0
Resource status: 1 (Started)
I / F attribute value: 0
(Spare machine)
Cluster status: "SBY [online]"
Number of failures: 0 or 1
Error status: 0 or 2 (no error or error detected during resource monitoring)
Resource status: 0 (resource is running on another server)
I / F attribute value: 1 or 2 (no error or link error)
Step 102) The state confirmation unit 131 of the active device 10 acquires the state management information in the state management information storage unit 119 by causing the high availability cluster software 110 to execute a state confirmation command. When the acquired state management information is the failure count “1” and the error status “2”, the monitoring time-out of the forced power-off function unit 216 of the standby machine 20 or the process of the forced power-off function unit 216 of the standby machine 20 Since it is estimated that a failure has occurred, an error of the forced power-off function is output to the maintenance terminal, and the process proceeds to step 105. In other cases (failure count “0”, error status “0”), the cause is another cause, and the process proceeds to step 103.

  Step 103) The state confirmation unit 131 of the active device 10 acquires the state management information in the state management information storage unit 119 by causing the high availability cluster software 110 to execute a state confirmation command. The failure location is estimated from the I / F attribute value of the acquired state management information. If the I / F attribute value is “1” (Link is failure), it is estimated that the communication between the spare unit 20 and the router 40 (network) is interrupted, and the process proceeds to step 104. If the I / F attribute value is “2” (Disk is failure), a disk failure is assumed, and the process proceeds to step 107.

  Step 104) In the continuity confirmation unit 223 of the spare machine 20, in order to improve the fault estimation accuracy, the continuity confirmation by the PING command is performed on the router 40. If the continuity is impossible, the process proceeds to Step 107. If it can be conducted, the process proceeds to step 106.

  Step 105) If it is determined in Step 102 that the forced power-off function unit 216 of the spare machine 20 is out of order, the maintenance terminal displays an error from the active machine 10, so the maintenance person logs in the spare machine 20 Then, the spare machine 20 is instructed to confirm the continuity with the power control unit 153 of the working machine 10. Thereby, the continuity confirmation unit 223 of the standby machine 20 performs a continuity confirmation by executing a PING command to the power supply control unit 153 of the active machine 10 in order to improve failure estimation accuracy. If it can be conducted, it is presumed not a failure of the hardware control board 160 but a temporary failure of the process of the forced power-off function unit 216 on the spare unit side. If conduction is not possible, the process proceeds to step 107.

  Step 106) If the connection to the router 40 is successful in Step 104 or the connection to the power control unit 153 of the active machine 10 is successful in Step 105, the state management unit 115 and the spare machine 20 of the active machine 10 are used. The state management unit 215 clears the number of failures of the forced power-off function monitoring resource in the state management information storage unit 119 to 0, and proceeds to step 108. This process makes it possible to use the forced power-off function even if a resource stop failure occurs during system switching due to a failure or the like.

  Step 107) Output the error to the maintenance terminal.

  Step 108) Log out.

  By performing the processing as described above, as shown in FIG. 7C, the state management unit 210 of the spare unit 20 makes the state of the spare unit 20 state management information storage unit 219 transitionable to “ACT”. As a result, the spare machine 20 can be prepared for a system switching process by the system switching instruction unit 127 of the active machine 10.

  In the following, according to the flowchart of FIG. 8, the configuration including the failure monitoring unit 111, the resource start / stop unit 113, and the state management unit 115 of the active machine 10 shown in FIG. The failure monitoring unit 211, the resource start / stop unit 213, and the state management unit 215 will be described as high availability cluster software 210.

  Step 101) Log in to the active machine 10 and the spare machine 20 from the maintenance terminal 50. If login to both is successful, the process proceeds to step 102, and if login to the spare machine 20 fails, the process proceeds to step 103.

  Step 102) As shown in FIG. 9, the state confirmation unit 131 of the control execution unit 120 causes the high availability cluster software 110 to execute a state confirmation command, and requests the number of failures and the error status of the spare unit 20.

Thereby, the high availability cluster software 110 refers to the state management information storage unit 119, and the spare machine 20
・ Number of failures "1";
・ Error status “2”;
In this case, since it is considered that the forced power-off function has failed, the routine proceeds to step 105. On the other hand, the spare machine 20 is
・ Frequency count “0”;
・ Error status “0”;
In the case of, since the cause of failure is another cause, the process proceeds to step 103.

  Step 103) As shown in FIG. 10, the state confirmation unit 131 of the control execution unit 120 causes the high availability cluster software 110 to execute a state confirmation command, and causes the state management information storage unit 119 to execute the I / F attribute value of the spare unit 20. Request. If the I / F attribute value of the spare machine 20 is “1” (link error), it can be estimated that the communication error has occurred, and the process proceeds to step 104. On the other hand, when the I / F attribute value of the spare machine 20 is “2” (disk error), it can be estimated that the disk is faulty, so an error is output to the maintenance terminal 50 and the user is logged out.

  Step 104) As shown in FIG. 11, the continuity confirmation unit 123 of the control execution unit 120 executes PING in the OS function and confirms continuity to the router 40. If the continuity is not possible, it is considered that a conductor or router has failed, so an error is output to the maintenance terminal 50 and the user is logged out.

  Step 105) In the case where it is estimated in Step 102 that the forced power-off function has failed, the maintenance terminal 50 logs into the spare machine 20 and confirms the continuity of the control execution unit 220 of the spare machine 20 as shown in FIG. From the unit 223, PING which is an OS function is executed, and conduction to the power supply control unit 153 of the active machine 10 is confirmed. If the ping from the standby machine 20 is successful, the power control unit 153 of the active machine 10 is normal, and a failure of the forced power-off function unit 216 in the high availability cluster software 210 of the standby machine 20 is predicted. .

  Step 106) The command execution unit 233 of the control execution unit 220 of the standby machine 20 causes the high availability cluster software 210 to execute a failure count clear command as shown in FIG. By executing the clear command, the failure count “1” and error status “2” of the status management information storage units 119 and 219 managed by the high availability cluster software 110 and 210 are changed to “0” and error status ”. Update to 0 ".

  As described above, if both the failure count and error status values are "0", high availability cluster software is forced even if a resource stop failure occurs during system switching from the active machine 10 to the standby machine 20. Since the power-off function can be used, in the active machine 10, "SBY [Transitioning]" (system switching has occurred due to a failure or the like, but resource switching has failed and system switching has not ended) It becomes possible to avoid becoming.

  For convenience of explanation, the system according to the embodiment of the present invention is described using a functional block diagram. However, the system of the present invention may be realized by hardware, software, or a combination thereof. For example, each functional unit of the server (active machine and spare machine) may be realized by software and installed on the operation system. In addition, the functional units may be used in combination as necessary.

  Although the embodiments and examples of the present invention have been described above, the present invention is not limited to the above-described embodiments and examples, and various modifications and applications are possible within the scope of the claims. is there.

10 servers (current machine)
20 servers (spare machine)
30 shared disk 40 router 50 maintenance terminal 101, 201 resource 110, 210 high availability cluster software 111, 211 failure monitoring unit 113, 213 resource / start / stop unit 115, 215 state management unit 116, 216 forced power off function unit 117, 217 Forced power-off monitoring unit 119, 219 State management information storage unit 120, 220 Control execution unit 123, 223 Continuity confirmation unit 125, 225 Cluster configuration activation unit 127, 227 System switching instruction unit 131, 231 State confirmation unit 133, 233 Command execution 151,251 OS (Operating System)
153, 253 Power control unit 155, 255 Power supply

Claims (9)

Fault monitoring means for monitoring the fault status, status management means for managing the cluster status indicating the service operating status of the active machine and the spare machine based on the fault status, status in which the service is operating (ACT), and status that can be changed to ACT Or status management information storage means for storing cluster status and failure status information including an unknown status (SBY: online) whether or not it is possible to transit to ACT, respectively, and a current machine and a spare machine, and the current machine and the This is a cluster system recovery method in a cluster system composed of shared disks shared by a spare machine, in which the current machine is incorporated into the cluster configuration and it is unknown whether the spare machine can transition to ACT (SBY: online) . And
A failure status acquisition step in which the status confirmation means of the active machine acquires failure status information from the status management information storage means via the status management means of the active machine;
If the failure status information indicates a failure of the forced power-off function, a failure of the forced power-off function on the spare unit side is suspected, so a forced power-off function error output step for outputting an error to the maintenance terminal When,
When continuity confirmation is instructed from the maintenance terminal on the spare machine side in an unknown state (SBY: online) whether it is possible to transition to ACT, the continuity confirmation means of the spare machine controls the hardware control on the active machine side If the continuity is confirmed with respect to the means, and if the continuity is confirmed, the state management means of the spare machine is forced to turn off the state management information storage means as a failure of the forced power-off function on the spare machine side. Clearing the number of failures of the function monitor resource, and clearing the number of failures to transition to a state (SBY: online) that enables system switching from the active machine,
A cluster system recovery method comprising: If the failure state information acquired in the failure state acquisition step indicates a network failure, check the continuity from the continuity confirmation means on the spare unit side to the router, and if continuity fails, The cluster system recovery method according to claim 1, further comprising a network error output step of outputting an error to the maintenance terminal. The failure status information acquired in the failure status acquisition step further includes a disk error output step of outputting an error to the maintenance terminal when the failure of the shared disk or the built-in disk is indicated. The cluster system recovery method according to claim 1. Fault monitoring means for monitoring the fault status, status management means for managing the cluster status indicating the service operating status of the active machine and the spare machine based on the fault status, status in which the service is operating (ACT), and status that can be changed to ACT Or status management information storage means for storing cluster status and failure status information including an unknown status (SBY: online) whether or not it is possible to transit to ACT, respectively, and a current machine and a spare machine, and the current machine and the A cluster system recovery system configured with a shared disk shared by a spare machine, the current machine is incorporated in a cluster configuration , and whether the spare machine can be changed to ACT or is in an unknown state (SBY: online) ,
The working machine is
Failure state acquisition means for acquiring failure state information from the state management information storage means via the state management means;
If the failure state information indicates a failure of the forced power-off function, a failure of the forced power-off function on the spare unit side is suspected, so a forced power-off function error output means for outputting an error to the maintenance terminal When,
Have
The spare machine is
When it is in an unknown state (SBY: online) whether or not it is possible to transition to ACT, when continuity confirmation is instructed from the maintenance terminal, continuity confirmation means for confirming continuity to the hardware control means on the working machine side When,
When continuity is confirmed by the continuity confirmation means, the number of failures of the forced power-off function monitoring resource of the state management information storage means is cleared as a failure of the forced power-off function on the spare machine side, and the working machine A failure frequency clearing means for transitioning to a state (SBY: online) that enables system switching from
A cluster system recovery system comprising: The working machine is
If the failure state information acquired by the failure state acquisition means indicates a network failure, router conduction means for confirming conduction to the router;
A network error output means for outputting an error to the maintenance terminal when the connection by the router conduction means fails;
The cluster system recovery system according to claim 4, further comprising: The working machine is
A disk error output means for outputting an error to the maintenance terminal when the failure status information acquired by the failure status acquisition means indicates a failure of a shared disk or a built-in disk;
The cluster system recovery system according to claim 4, further comprising: Fault monitoring means for monitoring the fault status, status management means for managing the cluster status indicating the service operating status of the active machine and the spare machine based on the fault status, status in which the service is operating (ACT), and status that can be changed to ACT Or status management information storage means for storing cluster status and failure status information including an unknown status (SBY: online) whether or not it is possible to transit to ACT, respectively, and a current machine and a spare machine, and the current machine and the It is configured with a shared disk shared by a spare machine , and the active machine is incorporated into a cluster configuration, and operates as the active machine when it is unknown whether the spare machine can transition to ACT (SBY: online) A server,
Failure state acquisition means for acquiring failure state information via the state management means;
If the failure state information indicates a failure of the forced power-off function, a failure of the forced power-off function on the spare unit side is suspected, so a forced power-off function error output means for outputting an error to the maintenance terminal When,
If the failure state information acquired by the failure state acquisition means indicates a network failure, router conduction means for confirming conduction to the router;
A network error output means for outputting an error to the maintenance terminal when the connection by the router conduction means fails;
When the failure status information acquired by the failure status acquisition means indicates a failure of the shared disk or internal disk, disk error output means for outputting an error to the maintenance terminal;
The server characterized by having. Fault monitoring means for monitoring the fault status, status management means for managing the cluster status indicating the service operating status of the active machine and the spare machine based on the fault status, status in which the service is operating (ACT), and status that can be changed to ACT Or status management information storage means for storing cluster status and failure status information including an unknown status (SBY: online) whether or not it is possible to transit to ACT, respectively, and a current machine and a spare machine, and the current machine and the A server that is configured with a shared disk shared by a spare machine, and that operates as a spare machine when the active machine is incorporated in a cluster configuration and the spare machine is in an unknown state (SBY: online) whether it can be changed to ACT Because
When continuity confirmation is instructed from the maintenance terminal, continuity confirmation means for confirming continuity with respect to the hardware control means on the active machine side,
When continuity is confirmed by the continuity confirmation means, the number of failures of the forced power-off function monitoring resource of the state management information storage means is cleared as a failure of the forced power-off function on the spare machine side, and the working machine A failure frequency clearing means for transitioning to a state (SBY: online) that enables system switching from
The server characterized by having.   The program for functioning a computer as each means which comprises the server of Claim 7 or 8.

Images