JP6056408B2 - Fault tolerant system - Google Patents

Description

  The present invention relates to a fault tolerant system, a synchronization method, a computer, and a program.

  As a computer system with high availability, an HA cluster (High availability cluster) and an FT server (Fault Tolerant server) having a hot standby configuration are known.

  The HA cluster interconnects a plurality of servers to achieve system redundancy. When a failure occurs in a server that is currently operating (active system), the server that has been prepared as a standby system takes over the processing, so the cluster as a whole seems to continue operating without any problems. The main methods include an active standby method and a replication method.

  In an active standby HA cluster, storage is shared between the active system and the standby system. The active system writes application-dependent information necessary for synchronizing the standby system to the shared storage, and the standby system performs recovery using this information at the time of failover. In other words, in the active standby system, recovery processing for each application is required at the time of failover, and thus availability cannot be obtained transparently from the viewpoint of the application or the OS. Further, there is a problem that it takes time for failover and the service cannot be provided during that time.

  In a replication-type HA cluster, the active system and the standby system have separate storages, and a request that arrives at the active system application is transferred to the standby system, thereby causing the standby system to make the same state transition as the active system. In general, matching the states of a plurality of systems is called synchronization. If a failure occurs in the active system and the service is stopped, the status of the standby system application is synchronized with the active system, so the service can be continued by disconnecting the active system. However, since a replication mechanism has to be added for each application to be clustered, availability cannot be obtained transparently from the viewpoint of the application or OS.

  Another form of replication-type HA cluster has been proposed as a first related technique related to the present invention (see, for example, Patent Document 1). In this first related technology, when a processing program running on the active OS updates the processing data in the memory, the update information of the processing data is transferred from the active system to the standby system, so that the active system and the standby system Synchronize the processing data of the system computers. In the first related technique, the standby system is made redundant in order to prepare for a failure of the standby system. For this purpose, the active system transfers the update information of the processing data to all the standby systems.

  As described above, in the HA cluster, a mechanism for availability must be added to the application and OS. On the other hand, there is no such need in the FT server. That is, the FT server can continue the service transparently without requiring any special processing by the application or the OS. The methods for realizing the FT server are roughly classified into a hardware method and a software method.

  The hardware-type FT server makes main hardware components such as a CPU, a memory, and a storage redundant. A module including a CPU, a memory, and a chip set is referred to as a CPU subsystem, and a module including various IO devices is referred to as an IO subsystem. The duplexing method differs between the CPU subsystem and the IO subsystem. The CPU subsystem is completely synchronized in hardware on a clock basis. This is called a lock step. Since both operate completely the same, when a failure occurs, the target CPU subsystem is logically disconnected, and the operation is continued with a normal CPU subsystem. Therefore, the CPU subsystem has no concept of active system and standby system. On the other hand, the IO subsystem uses one as an active system and the other as a standby system, and performs duplex control by software. When a failure occurs in the active IO subsystem, this is detected by software and immediately switched to the standby side. A hardware-based FT server can realize extremely high availability. However, since it is composed of special hardware, the introduction cost is higher than that of a PC server having similar performance.

  The software-based FT server uses a virtualization technology that enables a plurality of OSes to operate on a physical computer. A computer virtually constructed on a physical computer is called a virtual computer or a virtual machine. In a software-type FT server, physical computers are made redundant, and an active virtual computer and a standby virtual computer are arranged on different physical computers. When a failure such as a hardware failure occurs in the physical computer to which the active virtual computer belongs, the processing performed by the virtual computer is continuously executed by the standby virtual computer on another physical computer. In order to continue the service transparently from the viewpoint of the application and the OS, the software-type FT server performs processing to match the states of the active and standby virtual machines, that is, synchronization.

  There are mainly two methods for synchronizing the active and standby virtual computers: a virtual lockstep method and a checkpoint method. In the virtual lockstep method, an input to an active virtual machine is also given to a standby virtual machine, and the state of the standby virtual machine is changed in the same manner as the active virtual machine. While there is an advantage that the amount of data necessary for synchronization between virtual machines can be reduced, there is a problem that the CPU does not operate if the type of CPU differs between the active system and the standby system.

  On the other hand, the checkpoint method periodically sends an image (CPU, memory, storage, etc.) of the active virtual machine to the standby system, and matches the state of the standby virtual machine with the state of the active virtual machine. . The checkpoint method is easier to implement than the virtual lockstep method, and does not depend on a specific function of the CPU. On the other hand, since the image of the virtual computer has a large amount of data, there is a problem that the overhead required for one synchronization is larger than that of the virtual lockstep method.

  In order to solve this problem, it is proposed as a second related technique related to the present invention to send only the image of the active virtual machine updated since the previous checkpoint to the standby system (for example, non- Patent Document 1). In this second related technique, when a checkpoint arrives, the active virtual machine is temporarily stopped to interrupt the update to the main memory, and the main memory page has been updated since the previous checkpoint. All dirty pages are locally copied to a buffer in the main memory. When the local copy is completed, the active virtual machine that has been stopped is restarted, and in parallel, the copied dirty page is transferred from the buffer to the standby system.

JP 2009-217765 A

Brendan Cully, 5 others, “Remus: High Availability via Asynchronous Virtual Machine Replication”, [online], [searched on September 5, 2012], Internet <URL: http://www.cs.ubc.ca/ ~ andy / papers / remus-nsdi-final.pdf>

  Higher availability can be obtained by making the standby system redundant as described above. However, if the standby system is made redundant, the performance of the active system decreases. The reason for this is that, as described in the first related art related to the present invention, the active system must transmit update data to all the standby systems, and therefore communication with some of the standby systems. This is because the delay causes an immediate decrease in processing performance of the operating system. For this reason, especially when considering the redundancy of the standby system in the FT server of the checkpoint method, the active system transfers the status of the active system to all the standby systems for every checkpoint that arrives in a short cycle of several tens of ms. Since it must be transmitted, the performance of the active system is greatly reduced.

  An object of the present invention is to provide a fault tolerant system that solves the above-described problem, that is, the problem that the performance of an active system decreases when a standby system is made redundant.

The fault tolerant system according to the first aspect of the present invention is:
A first physical computer on which an active virtual computer operates, a second physical computer connected to the first physical computer and on which a first standby virtual computer operates, and the second physical computer A third physical computer connected and operating a second standby virtual computer;
Each time the first physical computer detects a first checkpoint, which is a timing at which the active virtual computer and the first standby virtual computer are synchronized, the active virtual computer And obtain the snapshot difference information and send it to the second physical computer,
Based on the snapshot difference information of the active virtual machine received from the first physical computer, the second physical computer changes the status of the first standby virtual machine to the first checkpoint. Each time a second checkpoint is detected that matches the state of the active virtual machine at the time and further synchronizes the first standby virtual machine with the second standby virtual machine. In addition, the snapshot difference information of the first standby virtual machine is acquired and transmitted to the third physical machine,
The third physical computer changes the state of the second standby virtual computer based on the snapshot difference information of the first standby virtual computer received from the second physical computer. To match the state of the first standby virtual machine at the time of the checkpoint.

The computer according to the second aspect of the present invention is:
Connected to the first and third physical computers,
Having a first standby virtual machine,
Based on the snapshot difference information of the active virtual machine operating on the first physical computer received from the first physical computer for each first checkpoint, the first standby virtual machine To match the state of the active virtual machine at the time of the first checkpoint,
Each time a second checkpoint is detected, which is the timing for synchronizing the first standby virtual machine and the second standby virtual machine running on the third physical computer, the first checkpoint is detected. The snapshot difference information of the standby virtual computer is acquired and transmitted to the third physical computer.

The synchronization method according to the third aspect of the present invention includes
A method of synchronizing computers connected to first and third physical computers and having a first standby virtual computer,
Based on the snapshot difference information of the active virtual machine operating on the first physical computer received from the first physical computer for each first checkpoint, the first standby virtual machine To match the state of the active virtual machine at the time of the first checkpoint,
Each time a second checkpoint is detected, which is the timing for synchronizing the first standby virtual machine and the second standby virtual machine running on the third physical computer, the first checkpoint is detected. The snapshot difference information of the standby virtual computer is acquired and transmitted to the third physical computer.

The program according to the fourth aspect of the present invention is:
A computer connected to the first and third physical computers,
While functioning as a first standby virtual machine,
Based on the snapshot difference information of the active virtual machine operating on the first physical computer received from the first physical computer at each first checkpoint to the computer, the first standby Processing to match the state of the active virtual machine with the state of the active virtual machine at the time of the first checkpoint, and to operate on the first standby virtual machine and the third physical computer Each time the second checkpoint, which is the timing for synchronizing with the second standby virtual machine, is detected, the snapshot difference information of the first standby virtual machine is acquired and the third physical point is acquired. The process to send to the computer is executed.

  Since this invention has the structure mentioned above, it can provide the fault tolerant system which can suppress the fall of the performance of an operation system, having several standby systems.

It is a block diagram of a 1st embodiment of the present invention. It is a sequence chart which shows the flow of a process including acquisition of the snapshot implemented on each physical computer in the 1st Embodiment of this invention, and execution of a synchronization. FIG. 3 is a state transition diagram of a plurality of standby virtual machines according to the first embodiment of this invention. It is a conceptual diagram which shows an example of the failover at the time of an active system failure in the 1st Embodiment of this invention. It is a conceptual diagram which shows the other example of the failover at the time of an active system failure in the 1st Embodiment of this invention. It is a block diagram of the 2nd Embodiment of this invention. FIG. 6 is a block diagram illustrating a configuration example of physical computers 310 and 320 installed in a main site 300 in the second embodiment of the present invention. FIG. 6 is a block diagram illustrating a configuration example of physical computers 410 and 420 arranged in a backup site 400 in the second embodiment of the present invention. It is a sequence chart for demonstrating the operation | movement of the 2nd Embodiment of this invention. It is a flowchart which shows the process example of a snapshot acquisition means, a snapshot transmission means, and a response reception means when the virtual computer corresponding to a self-physical computer is an active system in the 2nd Embodiment of this invention. It is a flowchart which shows the process example of the synchronizer in the 2nd Embodiment of this invention. 10 is a flowchart illustrating a processing example of a snapshot acquisition unit, a snapshot transmission unit, and a response reception unit when a virtual computer corresponding to the own physical computer is a standby system in the second embodiment of the present invention. It is a block diagram of the 2nd Embodiment of this invention.

Next, embodiments of the present invention will be described in detail with reference to the drawings.
[First embodiment]
Referring to FIG. 1, a fault tolerant system 100 according to the first embodiment of the present invention includes three physical computers 110, 120, and 130 as hardware. The physical computer 110 and the physical computer 120 are connected to be able to communicate with each other through a communication path 140, and the physical computer 120 and the physical computer 130 are connected to be able to communicate with each other through a communication path 150.

  The physical computer 110 includes a hypervisor 111 that provides a virtual computer environment, and a virtual computer 112 that operates under the virtual computer environment provided by the hypervisor 111. The physical computer 120 includes a hypervisor 121 that provides a virtual computer environment, and a virtual computer 122 that operates under the virtual computer environment provided by the hypervisor 121. Furthermore, the physical computer 130 includes a hypervisor 131 that provides a virtual computer environment, and a virtual computer 132 that operates under the virtual computer environment provided by the hypervisor 131.

  Here, the virtual machine 112 operates as an active system, and the virtual machines 122 and 132 operate as standby systems. That is, when a failure such as a hardware failure occurs in the physical computer 110 on which the virtual computer 112 operates, the processing performed by the virtual computer 112 is continuously executed by the standby virtual computer 122 on the physical computer 120. Further, when a failure such as a hardware failure occurs in the physical computer 110 on which the virtual computer 112 operates and the physical computer 120 on which the virtual computer 122 operates, the processing performed by the virtual computer 112 is performed on the standby system virtual machine on the physical computer 130. It is continuously executed by the computer 132.

  The hypervisor 111 of the physical computer 110 includes a snapshot management unit 113 and a checkpoint detection unit 114.

  The checkpoint detection unit 114 has a function of detecting a checkpoint that is a timing at which the active virtual machine 112 and the standby virtual machine 122 are synchronized. For example, the checkpoint detection unit 114 detects that a checkpoint has arrived every preset time period T1. Hereinafter, the checkpoint detected by the checkpoint detection unit 114 at time t is referred to as CT1t.

  Further, the snapshot management unit 113 acquires a snapshot of the active virtual machine 112 and transmits it from the physical computer 110 to the physical computer 120 every time a checkpoint is detected by the checkpoint detection unit 114. It has a function. With this snapshot acquisition function, the state of the virtual machine 112 at the checkpoint CT1t is acquired as a snapshot. The state of the virtual machine 112 acquired as a snapshot is the data necessary to restore the state of the virtual machine 112 to the time of the snapshot based on the acquired snapshot, for example, the CPU assigned to the virtual machine 112. This includes status (program counter and register status) and memory status.

  The hypervisor 121 of the physical computer 120 includes a snapshot management unit 123 and a checkpoint detection unit 124.

  The checkpoint detection unit 124 has a function of detecting a checkpoint that is a timing at which the standby virtual machine 122 and the standby virtual machine 132 are synchronized. For example, the checkpoint detection unit 124 detects that a checkpoint has arrived every preset time period T2. Here, the time period T2 is set to be the same as or longer than the time period T1 described above. That is, T2 ≧ T1. Hereinafter, the checkpoint detected by the checkpoint detection unit 124 at time t is referred to as CT2t.

  The snapshot management unit 123 also synchronizes the state of the standby virtual machine 122 with the state of the active virtual machine 112 based on the snapshot of the active virtual machine 112 received from the physical computer 110. And a snapshot acquisition function for acquiring a snapshot of the standby virtual machine 122 and transmitting it from the physical computer 120 to the physical machine 130 each time a checkpoint is detected by the checkpoint detection unit 124. In the synchronization function, based on the snapshot at the time of checkpoint CT1t of the active virtual machine 112 received from the physical computer 110, the state of the standby virtual machine 122 is changed to that of the active virtual machine 112 at the time of checkpoint CT1t. Match the state. The snapshot acquisition function acquires the state of the virtual machine 122 at the checkpoint CT2t as a snapshot.

  The hypervisor 131 of the physical computer 130 has a snapshot management unit 133. The snapshot management unit 133 has a synchronization function for matching the state of the standby virtual computer 132 with the state of the standby virtual computer 122 based on the snapshot of the standby virtual computer 122 received from the physical computer 120. . More specifically, based on the snapshot at the checkpoint CT2t of the standby virtual machine 122 received from the physical computer 120, the status of the standby virtual machine 132 is changed to the standby virtual machine 122 at the checkpoint CT2t. Match the state of.

  Next, operation | movement of the fault tolerant system 100 concerning this embodiment is demonstrated. FIG. 2 is a sequence chart showing the flow of processing including snapshot acquisition and execution of synchronization performed on each physical computer of the present embodiment.

  The checkpoint detector 114 included in the hypervisor 111 of the physical computer 110 detects the checkpoint CP1t at intervals of the time period T1. In FIG. 2, the checkpoint detection unit 114 detects checkpoints CT1t1, CT1t3, and CT1t5 at times t1, t3, and t5, respectively. At each checkpoint, the following processing is performed.

  When the checkpoint detection unit 114 detects the checkpoint CP1t1 at time t1, the checkpoint CP1t1 is notified to the snapshot management unit 113 of the own hypervisor 111 (S11). The snapshot management unit 113 acquires the state at the time t1 of the active virtual machine 112 on the own physical machine 110 as a snapshot (S12), and transmits it to the physical machine 120 through the communication path 140 (S13). When the snapshot management unit 123 included in the hypervisor 121 of the physical computer 120 receives the above-described snapshot transmitted from the physical computer 110, the snapshot management unit 123 of the standby computer on the own physical computer 120 is received based on the received snapshot. The state of 122 is matched with the state of the active virtual machine 112 at time t1 (S14).

  Further, when the checkpoint detection unit 114 detects the checkpoint CP1t3 at time t3, processes S21 to S24 similar to the processes S11 to S14 executed at time t1 are executed, so that the standby system on the physical computer 120 is executed. The state of the virtual machine 122 is made to match the state of the active virtual machine 112 at the time t3.

  Further, when the checkpoint detection unit 114 detects the checkpoint CP1t5 at time t5, the processes S31 to S34 similar to the processes S11 to S14 or S21 to S24 executed at the time t1 or t3 are executed, so that the physical The state of the standby virtual computer 122 on the computer 120 is matched with the state of the active virtual computer 112 at time t5.

  On the other hand, the checkpoint detector 124 included in the hypervisor 121 of the physical computer 120 detects the checkpoint CP2t at intervals of the time period T2. In FIG. 2, the checkpoint detection unit 124 detects checkpoints CT2t2 and CT2t4 at times t2 and t4, respectively. Here, t1 <t2 <t3 <t4 <t5. At each checkpoint, the following processing is performed.

  When the checkpoint detection unit 124 detects the checkpoint CP2t2 at time t2, the checkpoint CP2t2 is notified to the snapshot management unit 123 of the own hypervisor 121 (S15). The snapshot management unit 123 acquires the state at the time t2 of the standby virtual machine 122 on the own physical computer 120 as a snapshot (S16), and transmits it to the physical computer 130 through the communication path 150 (S17). When the snapshot management unit 133 included in the hypervisor 131 of the physical computer 130 receives the snapshot transmitted from the physical computer 120, the snapshot virtual unit 133 on the own physical computer 130 is based on the received snapshot. The state of 132 is matched with the state of the standby virtual machine 122 at time t2 (S18).

  Further, when the checkpoint detection unit 124 detects the checkpoint CP2t4 at time t4, processes S25 to S28 similar to the processes S15 to S18 executed at time t2 are executed, so that the standby system on the physical computer 130 is executed. The state of the virtual computer 132 is matched with the state of the standby virtual computer 122 at time t4.

  FIG. 3 is a state transition diagram of the standby virtual machines 122 and 132. The standby virtual machine 122 transitions in order from the active virtual machine 112 to the state at time t1, the state at time t3, and the state at time t5. On the other hand, the standby virtual machine 132 transitions in turn from the standby virtual machine 122 to the state at time t2 and to the state at time t4. Here, since the state of the standby virtual machine 122 is a state at a certain time of the active virtual machine 112, the state of the standby virtual machine 132 is the state of the standby system active computer 122 at a certain time. At the same time, it can be said that the operating virtual computer 112 is in a state at a certain point in time. Accordingly, when the standby virtual machine 122 at the time t2 and the time t4 is the active virtual machine 112 at the time t1 and the time t3, the standby virtual machine 132 is set to the active virtual machine 112. Transition to the state at time t1 and the state at time t3. As a result, both the virtual machine 122 and the virtual machine 132 become standby systems for the active virtual machine 112.

  Therefore, for example, as shown in FIG. 4, when a failure such as a hardware failure occurs in the physical computer 110, the processing performed by the virtual computer 112 in the active system of the physical computer 110 is processed in the standby system on the physical computer 120. Failover that is continuously executed by the virtual machine 122 is possible.

  For example, as illustrated in FIG. 5, when a failure such as a hardware failure occurs in both the physical computer 110 and the physical computer 120 after time t5, the active virtual computer 112 of the physical computer 110 performs the failure. Failover in which processing is continuously executed by the standby virtual computer 132 on the physical computer 130 is possible.

  As described above, according to the present embodiment, it is possible to provide a fault tolerant system that has a plurality of standby systems and can suppress a decrease in performance of the active system. The reason is that if the physical computer 110 having the active virtual computer 112 acquires a snapshot of the active virtual computer 112 for each predetermined checkpoint and transmits it to the physical computer 120, the physical computer 110 will be on the physical computer 120. This is because the standby virtual computer 122 can be generated and the other standby virtual computer 132 is generated on the other physical computer 130 by the autonomous operation of the physical computer 120.

[Second Embodiment]
Next, a second embodiment of the fault tolerant system according to the present invention will be described in detail. The fault tolerant system according to the present embodiment realizes disaster recovery by a remote backup method.

  Referring to FIG. 6, a fault tolerant system 200 according to the second embodiment of the present invention includes two physical computers 310 and 320 installed at the main site 300 as hardware and a backup geographically separated from the main site 300. And two physical computers 410 and 420 installed at the site 400. Further, the physical computer 310 and the physical computer 320 are connected to be able to communicate with each other through a communication path 330, and the physical computer 320 and the physical computer 410 are connected to be able to communicate with each other through a network 400 such as a WAN (Wide Area Network). The physical computer 410 and the physical computer 420 are connected to each other through a communication path 430 so that they can communicate with each other. Here, the communication paths 330 and 430 are high-speed and broadband communication paths compared to the network 500.

  Referring to FIG. 7, the physical computers 310 and 320 installed in the main site 300 operate under a hypervisor 311 and 321 that provide a virtual computer environment and a virtual computer environment provided by the hypervisor 311 and 321, respectively. Virtual computers 312 and 322. Referring to FIG. 8, the physical computers 410 and 420 installed at the backup site 400 are respectively hypervisors 411 and 421 that provide virtual computer environments, and virtual computer environments provided by the hypervisors 411 and 421. Virtual machines 412 and 422 operating in

  Here, the virtual machine 312 operates as an active system, and the virtual machines 322, 412, and 422 operate as standby systems. That is, when a failure such as a hardware failure occurs in the physical computer 310 on which the virtual computer 312 operates, the processing performed by the virtual computer 312 is continuously executed by the virtual computer 322 on the physical computer 320. Further, when a failure occurs in the physical computer 320, the processing executed by the virtual computer 322 is continuously executed by the virtual computer 412 on the physical computer 410 installed in the backup site 400. Further, when a failure occurs in the physical computer 410, the processing performed by the virtual computer 412 is continuously executed by the virtual computer 422 on the physical computer 420.

  The hypervisor 311 of the physical computer 310 installed in the main site 300 includes a snapshot management unit 313 and a checkpoint detection unit 314.

  The checkpoint detection unit 314 has a function of detecting a checkpoint that is a timing for synchronizing the virtual machine 312 and the virtual machine 322. For example, the checkpoint detection unit 314 detects that a checkpoint has arrived every predetermined time period Tcp1.

  The snapshot management unit 313 includes a snapshot acquisition unit 315, a snapshot transmission unit 316, and a response reception unit 317.

  The snapshot acquisition unit 315 has a function of stopping the operation of the virtual machine 312 each time the checkpoint detection unit 314 detects a checkpoint, a snapshot indicating the state of the virtual machine 312 at the previous checkpoint, and the current check. It has a function of acquiring difference information (snapshot difference information) indicating a difference from a snapshot indicating the state of the virtual computer 312 at the point.

  The snapshot transmission unit 316 has a function of transmitting the snapshot difference information acquired by the snapshot acquisition unit 315 to the physical computer 320 via the communication path 330.

  The response receiving unit 317 has a function of starting the virtual computer 312 when a response signal indicating that the snapshot difference information has been normally received is transmitted from the physical computer 320.

  The snapshot management unit 313 and the checkpoint detection unit 314 can be realized by a computer. In this case, for example, the following is performed. A disk, a semiconductor memory, and other recording media on which a program for causing the computer to function as the snapshot management unit 313 and the checkpoint detection unit 314 are prepared are prepared, and the computer is caused to read the program. The computer controls the operation of itself according to the read program, thereby realizing the snapshot management unit 313 and the checkpoint detection unit 314 on the computer.

  Further, the hypervisor 321 of the physical computer 320 includes a snapshot management unit 323 and a checkpoint detection unit 324.

  The checkpoint detection unit 324 has a function of detecting a checkpoint that is a timing at which the virtual machine 322 and the virtual machine 412 are synchronized. For example, the checkpoint detection unit 324 detects that a checkpoint has arrived every predetermined time period Tcp2. The time period Tcp2 is set as the same period as the above-described time period Tcp1 or a longer period (Tcp2 ≧ Tcp1).

  The snapshot management unit 323 includes a snapshot acquisition unit 325, a snapshot transmission unit 326, a response reception unit 327, and a synchronization unit 328.

  The snapshot acquisition unit 325 performs different processing depending on whether the virtual machine 322 is an active system or a standby system. When the virtual machine 322 is in the active system, the virtual machine 322 is stopped every time a checkpoint is detected by the checkpoint detection unit 324, and then snapshot difference information about the virtual machine 322 is acquired. If the check point is detected by the check point detection unit 324, the snapshot difference information for the virtual computer 322 is acquired.

  The snapshot transmission unit 326 has a function of transmitting the snapshot difference information acquired by the snapshot acquisition unit 325 to the physical computer 410 via the network 500.

  The response receiving unit 327 performs different processing depending on whether the virtual computer 322 is an active system or a standby system. When the virtual computer 322 is in the active system, the virtual computer 322 is activated by receiving a response signal sent from the physical computer 410, and then control is passed to the snapshot acquisition means 325. When the virtual computer 322 is in a standby system, control is transferred to the snapshot acquisition unit 325 by receiving a response signal sent from the physical computer 410.

  The synchronization unit 328 has a function of transmitting a response signal indicating that to the physical computer 310 when the snapshot difference information about the virtual computer 312 sent from the physical computer 310 is normally received, and the received snapshot difference information. Based on the above, the state of the standby virtual machine 322 is matched with the state at the latest checkpoint of the active virtual machine 312.

  Note that the snapshot management unit 323 and the checkpoint detection unit 324 can be realized by a computer in the same manner as described above.

  On the other hand, referring to FIG. 8, the hypervisor 411 of the physical computer 410 installed at the backup site 400 includes a snapshot management unit 413 and a checkpoint detection unit 414.

  The checkpoint detection unit 414 has a function of detecting a checkpoint that is a timing at which the virtual machine 412 and the virtual machine 422 are synchronized. For example, the checkpoint detection unit 414 detects that a checkpoint has arrived every predetermined time period Tcp3. The time period Tcp3 is set as a period (Tcp3 ≧ Tcp2) that is the same as or longer than the above-described time period Tcp2.

  The snapshot management unit 413 includes a snapshot acquisition unit 415, a snapshot transmission unit 416, a response reception unit 417, and a synchronization unit 418.

  The snapshot acquisition unit 415 performs different processing depending on whether the virtual machine 412 is an active system or a standby system. When the virtual machine 412 is in the active system, the virtual machine 412 is stopped every time a checkpoint is detected by the checkpoint detection unit 414, and thereafter, snapshot difference information about the virtual machine 412 is acquired. If the check point is detected by the check point detection unit 414, the snapshot difference information about the virtual machine 412 is acquired.

  The snapshot transmission unit 416 has a function of transmitting the snapshot difference information acquired by the snapshot acquisition unit 415 to the physical computer 420 via the communication path 430.

  The response receiving unit 417 performs different processing depending on whether the virtual computer 412 is in the active system or the standby system. When the virtual machine 412 is in the active system, the virtual machine 412 is activated by receiving a response signal sent from the physical machine 420, and then the control is passed to the snapshot acquisition unit 415. When the virtual machine 412 is in a standby system, control is passed to the snapshot acquisition unit 415 by receiving a response signal sent from the physical machine 420.

  The synchronization unit 418 is configured to return a response signal to the physical computer 320 when the snapshot difference information for the virtual computer 322 from the physical computer 320 is normally received, or based on the received snapshot difference information. A function of matching the state of 412 with the state at the latest checkpoint of the virtual machine 322 is provided.

  The snapshot management unit 413 and the checkpoint detection unit 414 can be realized by a computer in the same manner as described above.

  The snapshot management unit 423 provided in the hypervisor 421 of the physical computer 420 includes a synchronization unit 428.

  The synchronization unit 428 performs virtual response based on the function of returning a response signal to the physical computer 410 when the snapshot difference information about the virtual computer 412 sent from the physical computer 410 is normally received, and the received snapshot difference information. It has a function of matching the state of the computer 422 with the state of the virtual computer 412 at the latest checkpoint.

  The snapshot management unit 423 can be realized by a computer in the same manner as described above.

  Next, the operation of the fault tolerant system 200 according to the present embodiment will be described with reference to FIGS. 9 is a sequence chart for explaining the state transition of the virtual computers 322, 412, and 422 on the physical computers 320, 410, and 420. FIG. 10 is a diagram when the virtual computers 312, 322, and 412 are in the active system. FIG. 11 is a flowchart showing a processing example of the synchronization means 328, 418, 428, and FIG. 12 is a diagram when the virtual machines 322, 412 are in a standby system. 4 is a flowchart showing a processing example of the snapshot management units 323 and 413.

  As shown in FIG. 9, it is assumed that the checkpoint detection unit 314 in the physical computer 310 in which the active virtual computer 312 is operating detects a checkpoint (CP) at time t11.

  When the checkpoint detection unit 314 detects a checkpoint at time t11 (Yes in step S101 in FIG. 10), the snapshot acquisition unit 315 temporarily stops the virtual computer 312 and then the virtual computer 312 at the previous checkpoint. Snapshot difference information indicating the difference between the snapshot indicating the state and the snapshot indicating the state of the virtual machine at the current checkpoint is acquired (steps S102 and S103). Here, as a method of acquiring the snapshot difference information, for example, a method of acquiring the contents of a page (dirty page) updated after the previous checkpoint as the snapshot difference information can be employed.

  Thereafter, the snapshot transmission means 316 transmits the snapshot difference information to the physical computer 320 via the communication path 330 (step S104).

  When receiving the snapshot difference information (Yes in step S111 in FIG. 11), the synchronization unit 328 in the physical computer 320 returns a response signal to the physical computer 310 (step S112). Thereafter, the synchronization unit 328 matches the state of the standby virtual machine 322 with the state at the latest checkpoint of the active virtual machine 312 based on the received snapshot difference information (step S113), and again, The process of step S111 is performed.

  When receiving a response signal (Yes in step S105 in FIG. 10), the response receiving unit 317 in the physical computer 310 determines that the snapshot difference information has been normally received in the physical computer 320 and restarts the operation of the virtual computer 312. (Step S106). Here, the operation of the virtual computer 312 is from the time when the check point detecting unit 314 detects the check point until the response receiving unit 317 receives the response signal (from time 11 to time 12 in FIG. 9). Stop temporarily. However, in this embodiment, the physical computers 310 and 320 are arranged close to each other, and the physical computers 310 and 320 are connected by the broadband communication path 330 as compared to the network 500, so that the communication response delay can be shortened. As a result, the stop time of the virtual machine 312 can be shortened.

  Thereafter, when the checkpoint detection unit 314 detects a checkpoint at time t13, the snapshot acquisition unit 315, the snapshot transmission unit 316, the response reception unit 317, and the synchronization unit 328 perform the same processing as that described above (FIG. 10 and the process shown in the flowchart of FIG. 11), the state of the standby virtual machine 322 changes to the state of the active virtual machine 312 at time t13.

  Thereafter, at time t15, when the checkpoint detection unit 324 in the physical computer 320 detects a checkpoint (Yes in step S121 in FIG. 12), the snapshot acquisition unit 325 acquires the snapshot difference information for the virtual computer 322. Then, the snapshot transmission means 326 transmits the snapshot difference information to the physical computer 410 via the network 500 (steps S122 and S123).

  Here, the time period Tcp2 in which the checkpoint detection unit 324 in the physical computer 320 detects the checkpoint is longer than the time period Tcp1 in which the checkpoint detection unit 314 in the physical computer 310 detects the checkpoint as described above. It is a cycle. Therefore, for example, if the time period Tcp2 is five times the time period Tcp1, the snapshot acquisition unit 325 takes the snapshot difference for the virtual computer 322 every time the snapshot difference information is sent from the physical computer 310 five times. You will get information. Generally, due to the locality of the program, there is a high probability that the contents of the same dirty page are sent from the physical computer 310 between the previous check point and the current check point in the physical computer 320. For this reason, the time cycle Tcp2 of the checkpoint detection unit 324 is made longer than the time cycle Tcp1 of the checkpoint detection unit 314, and each time the snapshot difference information is sent from the physical computer 310 a plurality of times, a snap for the virtual computer 322 is performed. Compared to the case where the snapshot difference information is acquired, the time periods Tcp1 and Tcp2 are set to the same period, and the snapshot difference information is acquired every time the snapshot difference information is sent once from the physical computer 310, compared to the case where the snapshot difference information is acquired. The amount of data transfer can be reduced.

  Upon receiving the snapshot difference information from the physical computer 320 (Yes in step S111 in FIG. 11), the synchronization unit 418 in the physical computer 410 returns a response signal to the physical computer 320, and then, based on the snapshot difference information. The state of the virtual machine 412 is made to coincide with the state at the latest checkpoint of the virtual machine 322 (steps S112 and S113). Here, since the state at the latest checkpoint of the virtual machine 322 matches the state at the time t13 of the virtual machine 312, the state of the virtual machine 412 transitions to the state of the virtual machine 312 at the time t13. Become.

  When receiving the response signal from the physical computer 410 at time t16 (Yes in step S124), the response receiving unit 327 in the physical computer 320 passes control to the snapshot acquisition unit 325 to perform the process of step S121. Note that snapshot difference information and response signals are also exchanged between the physical computer 310 and the physical computer 320 from time t15 to time t16, but are not shown in FIG.

  Here, the physical computer 320 and the physical computer 410 are installed at geographically distant points, and the network 500 has a narrower bandwidth compared to the communication path 330, and therefore the physical computer 310 and 320 communicate with each other. Response delay is large. However, since the communication response delay between the physical computer 310 and the physical computer 320 in the main site 300 is very small, even if the communication response delay between the physical computers 320 and 410 is large, the processing of the active virtual computer 312 is performed. There is almost no decline in ability.

  Thereafter, when the checkpoint detection unit 414 in the physical computer 410 detects a checkpoint at time t17 (Yes in step S121 in FIG. 12), the snapshot acquisition unit 415 acquires snapshot difference information for the virtual computer 412. The snapshot transmission means 416 transmits the snapshot difference information to the physical computer 420 via the communication path 430 (steps S122 and S123).

  Upon receiving the snapshot difference information (Yes in step S111 in FIG. 11), the synchronization unit 428 of the physical computer 420 returns a response signal to the physical computer 410, and then changes the status of the virtual computer 422 to the latest of the virtual computer 412. Match the state at the time of checkpoint (steps S112 and S113). Here, since the state at the latest checkpoint of the virtual machine 412 matches the state of the virtual machine 312 at time t13, the state of the virtual machine 422 transitions to the state of time t13 of the virtual machine 312.

  As described above, the states of the virtual machines 322, 412, and 422 coincide with the state of the active virtual machine 312 at a certain time. Accordingly, the virtual machines 322, 412, and 422 become standby systems for the active virtual machine 312.

  Next, an operation when a failure such as a hardware failure occurs in the physical computer 310 on which the active virtual computer 312 is operating will be described.

  The snapshot management unit 323 on the physical computer 320 detects whether there is a failure in the physical computer 310 in which the active virtual computer 312 is operating. The method for detecting a failure is arbitrary. For example, the snapshot management unit 323 detects a failure by detecting the presence / absence of reception of a specific signal (for example, a heartbeat signal) transmitted from the physical computer 310 to the physical computer 320 via the communication path 330 at a fixed period. The presence or absence of may be detected.

  When detecting the failure of the physical computer 310, the snapshot management unit 323 switches the virtual computer 322 corresponding to the own physical computer 320 from the standby system to the active system. As a result, the virtual machine 322 that has newly become an active system continues to execute the processing performed by the virtual machine 312 (failover). In addition, the snapshot acquisition unit 325, snapshot transmission unit 326, and response reception unit 327 in the snapshot management unit 323 have the virtual computer 322 corresponding to the own physical computer 320 switched to the active system. Instead of the process shown in the flowchart, the process shown in the flowchart of FIG. 10 is executed.

  Next, an operation when a failure occurs in the physical computer 320 in addition to the physical computer 310 will be described.

  When the snapshot management unit 413 on the physical computer 410 detects a failure of the physical computer 320 in which the active virtual computer 322 is operating, the snapshot management unit 413 switches the virtual computer 412 corresponding to the own physical computer 410 from the standby system to the active system. . As a result, the virtual machine 412 that has newly become an active system continues to execute the processing performed by the virtual machine 322. In addition, the snapshot acquisition unit 415, snapshot transmission unit 416, and response reception unit 417 in the snapshot management unit 413 are switched to the standby system because the virtual computer 412 corresponding to the own physical computer 410 is switched to the standby system. Instead of the process shown in the flowchart, the process shown in the flowchart of FIG. 10 is executed.

  Next, an operation when a failure occurs in the physical computer 410 in addition to the physical computers 310 and 320 will be described.

  When the snapshot management unit 423 on the physical computer 420 detects a failure of the physical computer 410 on which the active virtual computer 412 is operating, the snapshot management unit 423 switches the virtual computer 422 corresponding to the own physical computer 420 from the standby system to the active system. . As a result, the virtual machine 422 that has newly become an active system continues to execute the processing performed by the virtual machine 412. The snapshot management unit 423 executes the process shown in the flowchart of FIG. 10 instead of the process shown in the flowchart of FIG. 12 because the virtual machine 422 corresponding to the own physical computer 420 has switched from the standby system to the active system. To do.

  Regarding the checkpoint detection units 324 and 414, when the virtual computers 332 and 412 corresponding to the own physical computers 320 and 410 are switched from the standby system to the active system, the time periods Tcp2 and Tcp3 for detecting the checkpoint are shorter. The time periods Tcp2 ′ and Tcp3 ′ (Tcp1> Tcp1 ′, Tcp2> Tcp2 ′) may be changed.

  According to the present embodiment, since the checkpoint cycle by the checkpoint detection unit 324 is longer than the checkpoint cycle by the checkpoint detection unit 314, the amount of snapshot difference information transferred via the network 500 Can be reduced. Furthermore, in this embodiment, since the physical computers 310 and 320 are installed at the main site 300 and the physical computers 410 and 420 are installed at the backup site 400 that is geographically separated from the main site 400, disaster recovery is possible.

[Third embodiment]
Next, a third embodiment of the fault tolerant system according to the present invention will be described.

  Referring to FIG. 13, the fault tolerant system according to the present embodiment is connected to the first physical computer 600 on which the active virtual computer 610 operates and the first physical computer 600, and the first standby computer It has a second physical computer 700 on which the virtual computer 710 operates and a third physical computer 800 connected to the second physical computer 700 and on which the second standby virtual computer 810 operates.

  The first physical computer 600 performs the above operation every time a first checkpoint is detected, which is a timing at which the active virtual computer 610 and the first standby virtual computer 710 are synchronized. The snapshot difference information of the system virtual machine 610 is acquired and transmitted to the second physical machine 700.

  The second physical computer 700 changes the state of the first standby virtual computer 710 based on the snapshot difference information of the active virtual computer 610 received from the first physical computer 600. Secondly, the second virtual machine 610 is synchronized with the first standby virtual machine 810 and the second standby virtual machine 810 to match the state of the active virtual machine 610 at the time of the first checkpoint. Each time a checkpoint is detected, snapshot difference information of the first standby virtual machine 710 is acquired and transmitted to the third physical computer 800.

  In addition, the third physical computer 800 uses the snapshot difference information of the first standby virtual computer 710 received from the second physical computer 700 to determine the second standby virtual computer 810. The state is matched with the state of the first standby virtual machine 710 at the time of the second checkpoint.

  As described above, according to the present embodiment, it is possible to provide a fault tolerant system that has a plurality of standby systems and can suppress a decrease in performance of the active system. The reason is that the first physical computer 600 having the active virtual computer 600 acquires the snapshot difference information of the active virtual computer 610 for each predetermined checkpoint and transmits it to the physical computer 700. The first standby virtual machine 710 can be generated on the second physical computer 700, and the second standby virtual machine 810 can perform the third physical operation by the autonomous operation of the second physical computer 700. This is because it is generated on the computer 800.

100 Fault tolerant system 110, 120, 130 Physical computer 111, 121, 131 Hypervisor 112, 122, 132 Virtual computer 113, 123, 133 Snapshot management unit 114, 124 Checkpoint detection unit 200 Fault tolerant system 300 Main site 310, 320 Physical computer 311, 321 Hypervisor 312, 322 Virtual computer 313, 323 Snapshot management unit 314, 324 Checkpoint detection unit 315, 325 Snapshot acquisition unit 316, 326 Snapshot transmission unit 317, 327 Response reception unit 328 Synchronization unit 330 Communication path 400 Backup site 410, 420 Physical computer 411, 421 Hypervisor 412, 422 Virtual computer 413, 4 23 Snapshot management unit 414 Check point detection unit 415 Snapshot acquisition unit 416 Snapshot transmission unit 417 Response reception unit 418, 428 Synchronization unit 430 Communication channel 500 Network 600, 700, 800 Physical computer 610, 710, 810 Virtual computer

Claims (21)

A first physical computer on which an active virtual computer operates, a second physical computer connected to the first physical computer and on which a first standby virtual computer operates, and the second physical computer A third physical computer connected and operating a second standby virtual computer;
Each time the first physical computer detects a first checkpoint, which is a timing at which the active virtual computer and the first standby virtual computer are synchronized, the active virtual computer And obtain the snapshot difference information and send it to the second physical computer,
Based on the snapshot difference information of the active virtual machine received from the first physical computer, the second physical computer changes the status of the first standby virtual machine to the first checkpoint. Each time a second checkpoint is detected that matches the state of the active virtual machine at the time and further synchronizes the first standby virtual machine with the second standby virtual machine. In addition, the snapshot difference information of the first standby virtual machine is acquired and transmitted to the third physical machine,
The third physical computer changes the state of the second standby virtual computer based on the snapshot difference information of the first standby virtual computer received from the second physical computer. A fault tolerant system that matches the state of the virtual machine in the first standby system at the checkpoint time. The first physical computer and the second physical computer are connected by a high-speed and wide-band communication path compared to a communication path connecting the second physical computer and the third physical computer. Item 2. The fault tolerant system according to Item 1. 3. The fault tolerant system according to claim 1, wherein a period of the second checkpoint is the same as or longer than a period of the first checkpoint. The first physical computer and the second physical computer are installed at a main site,
The fault tolerant system according to any one of claims 1 to 3, wherein the third physical computer is installed at a backup site geographically separated from the main site. A fourth physical computer connected to the third physical computer and operating a third standby virtual computer;
The third physical computer is
Every time the third checkpoint is a timing synchronized with the virtual machine and the third standby virtual machine of said second standby is detected, the virtual machine of said second standby A function of acquiring snapshot difference information and transmitting the snapshot difference information from the third physical computer to the fourth physical computer;
The fourth physical computer is
Based on the snapshot difference information of the second standby virtual machine received from the third physical computer, the status of the third standby virtual machine is changed to the second time at the third checkpoint time. Ru to match the state of the virtual computer of the standby system
Fault tolerant system according to any one of 請 Motomeko 1 to 4. Connected to the first and third physical computers,
Having a first standby virtual machine,
Based on the snapshot difference information of the active virtual machine operating on the first physical computer received from the first physical computer for each first checkpoint, the first standby virtual machine To match the state of the active virtual machine at the time of the first checkpoint,
Each time a second checkpoint is detected, which is the timing for synchronizing the first standby virtual machine and the second standby virtual machine running on the third physical computer, the first checkpoint is detected. A computer that acquires the snapshot difference information of the standby virtual computer and transmits it to the third physical computer. The computer according to claim 6, wherein the first physical computer is connected by a high-speed and broadband communication path compared to a communication path connected to the third physical computer. The computer according to claim 6 or 7, wherein the cycle of the second checkpoint is the same cycle as the cycle of the first checkpoint or a longer cycle. 9. The system according to claim 6, wherein the third physical computer is installed together with the first physical computer at a main site that is geographically distant from the backup site where the third physical computer is installed. Calculator. A method of synchronizing computers connected to first and third physical computers and having a first standby virtual computer,
Based on the snapshot difference information of the active virtual machine operating on the first physical computer received from the first physical computer for each first checkpoint, the first standby virtual machine To match the state of the active virtual machine at the time of the first checkpoint,
Each time a second checkpoint is detected, which is the timing for synchronizing the first standby virtual machine and the second standby virtual machine running on the third physical computer, the first checkpoint is detected. A synchronization method of acquiring snapshot difference information of the standby virtual computer and transmitting it to the third physical computer. 11. The synchronization method according to claim 10, wherein the first physical computer is connected to the first physical computer via a communication path that is faster and wider than the communication path connected to the third physical computer. 12. The synchronization method according to claim 10, wherein a period of the second checkpoint is the same as or longer than a period of the first checkpoint. 13. The system according to claim 10, wherein the third physical computer is installed together with the first physical computer at a main site that is geographically separated from a backup site where the third physical computer is installed. Synchronization method. A first physical computer on which an active virtual computer operates, a second physical computer on which a first standby virtual computer operates, and a third physical computer on which a second standby virtual computer operates A fault tolerant system synchronization method comprising:
Wherein the first physical computer, the first check point by point a timing synchronized with the operation system of the virtual machine and the first standby-system virtual machine, snapshot the virtual machine before Ki稼働系Get shot difference information and send it from the first physical computer to the second physical computer,
Based on the snapshot difference information of the active virtual computer received from the first physical computer by the second physical computer, the status of the first standby virtual computer is changed to the first checkpoint. Match the state of the active virtual machine at the time,
For each second checkpoint at which the second physical computer synchronizes the first standby virtual machine and the second standby virtual computer, the first standby system Obtain snapshot difference information of the virtual computer and send it from the second physical computer to the third physical computer,
Based on the snapshot difference information of the first standby virtual machine received from the second physical computer, the third physical computer changes the state of the second standby virtual machine to the second A fault tolerant system synchronization method that matches the state of the virtual machine in the first standby system at the checkpoint time. The first physical computer and the second physical computer are connected by a high-speed and wide-band communication path compared to a communication path connecting the second physical computer and the third physical computer. Item 15. The fault tolerant system synchronization method according to Item 14. 16. The fault tolerant system synchronization method according to claim 14, wherein the period of the second checkpoint is the same period as or longer than the period of the first checkpoint. The first physical computer and the second physical computer are installed at a main site,
The fault tolerant system synchronization method according to any one of claims 14 to 16, wherein the third physical computer is installed at a backup site geographically separated from the main site. A computer connected to the first and third physical computers,
While functioning as a first standby virtual machine,
Based on the snapshot difference information of the active virtual machine operating on the first physical computer received from the first physical computer at each first checkpoint to the computer, the first standby Processing to match the state of the active virtual machine with the state of the active virtual machine at the time of the first checkpoint, and to operate on the first standby virtual machine and the third physical computer Each time the second checkpoint, which is the timing for synchronizing with the second standby virtual machine, is detected, the snapshot difference information of the first standby virtual machine is acquired and the third physical point is acquired. A program for executing processing to be sent to a computer. 19. The program according to claim 18, wherein the first physical computer and the computer are connected by a high-speed and wide-band communication path compared to a communication path connecting the third physical computer and the computer. The program according to claim 18 or 19, wherein the cycle of the second checkpoint is the same cycle as the cycle of the first checkpoint or a longer cycle. The first physical computer and the computer are installed at a main site,
The program according to any one of claims 18 to 20, wherein the third physical computer is installed at a backup site geographically separated from the main site.

Images