JP2014085883A - Device, method, and program for data matching - Google Patents

Abstract

PROBLEM TO BE SOLVED: To match redundant data sets without deteriorating performance of a host application.SOLUTION: A data matching device of the present invention comprises: a comparison unit; and a unification unit. The comparison unit compares data length of redundant data sets, which are copies of a same data set and are stored in a respective plurality of storage units, when a write process by a host application terminates abnormally. When the comparison unit determines that the redundant data sets do not have an identical data length, the unification unit adjusts the data length of all the redundant data sets to a data length of the shortest redundant data set of all the redundant data sets stored in the plurality of storage units.

Description

  The present invention relates to a technique for maintaining consistency between redundant data.

  Conventionally, when a device that stores data fails, data stored in the device is prevented from being damaged or lost, and redundancy for storing the same data in a plurality of devices has been performed. For example, a distributed file system that manages replicas of the same data distributed to a plurality of servers has been realized.

  For example, Non-Patent Document 1 discloses an example of a distributed file system. In this system, each cluster is composed of one master and a plurality of servers. Multiple clients access the cluster. When storing a file in a cluster, the file is divided into a plurality of fixed-length blocks, and a copy of each block is stored in the server. Thus, by duplicating files to make them redundant and holding them distributed on a plurality of servers, fault tolerance can be improved, parallel processing on a plurality of servers can be performed, and processing efficiency can be improved.

Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, “The Google File System”, 19th ACM Symposium on Operating Systems Principles, October 2003, p. 20-43 Vijayan Prabhakaran, Lakshmi N. Bairavasundaram, Nitin Agrawal, Haryadi S. Gunawi, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau, "IRON File Systems", SOSP '05 Proceedings of the twentieth ACM symposium on Operating systems principles, October 2005, p. 206-220

  However, in the above technique, when maintaining consistency between a plurality of redundant data (hereinafter also referred to as “redundant data”. The redundant data includes data at the copy source and data at the copy destination). In addition, there is a problem that the processing performance deteriorates.

  For example, when the writing process to one server of the file system ends abnormally in the middle, the data stored in the server does not match the data stored in the other server. To deal with this problem, transaction processing and checksums are used to detect inconsistencies and automatically match redundant data with each other. Redundant data is re-executed from a higher-level application to exchange redundant data. There is a method to match.

  However, with the mismatch detection method using transaction processing, the identity of redundant data is maintained, but the processing procedure for writing data to a file on the file system increases, and the writing performance of the higher-level application decreases. To do.

  On the other hand, for example, in the technique disclosed in Non-Patent Document 2 for detecting a mismatch state using a checksum, first, a checksum of redundant data is acquired. When the checksums are compared between the plurality of redundant data, for example, periodically or when accessing the redundant data or when the writing process fails, the redundant data becomes the same. Redundant data is rewritten as described above.

  However, in the mismatch detection method using the checksum, the process of accessing the file on the file system from the upper application and the process of comparing the checksum in the file system conflict. For this reason, the performance of the write processing and read processing of the file system also deteriorates.

  In the method of re-execution of the writing process from the upper application, when the writing fails, the upper application writes the target data of the failed writing again. According to this method, the load applied to maintaining the consistency of redundant data can be controlled by the upper application, and the deterioration of the writing performance of the upper application is suppressed compared to the method of automatically maintaining the consistency of redundant data. be able to.

  However, in the method of re-execution of the writing process from the upper application, if the upper application that is performing the writing process terminates abnormally, the upper application itself has to make a permanent file such as a local file in order to perform the writing process again. The processing result needs to be written in the processing apparatus. For this reason, the write processing time of the upper application becomes longer, and the write performance of the upper application is degraded.

  The present invention has been made in view of the above, and an object of the present invention is to provide a data matching apparatus, a data matching method, and a data matching program capable of matching redundant data without degrading the performance of a higher-level application. And

  In order to solve the above-described problems and achieve the object, the present invention provides a data length of redundant data, which is a duplicate of the same data stored in each of a plurality of storage units, by a write process from a higher-level application. When it is determined that the data length of the redundant data is not the same as in the case of abnormal termination, other redundant data is added to the data length of the shortest redundant data among the redundant data stored in the plurality of storage units. It is characterized by the same data length.

  The data matching apparatus, the data matching method, and the data matching program according to the present invention have an effect that redundant data can be matched without lowering the performance of the host application.

FIG. 1 is a block diagram showing an outline of a data matching apparatus according to the first embodiment of the present invention. FIG. 2 is a flowchart showing an example of the flow of data matching processing of the data matching apparatus according to the first embodiment. FIG. 3 is a block diagram showing an outline of a data matching system according to the second embodiment of the present invention. FIG. 4A is a diagram for explaining an example of information stored in the redundant data position table according to the second embodiment. FIG. 4B is a diagram for explaining the correspondence between the information stored in the redundant data position table according to the second embodiment and the disk storing the data. FIG. 5 is a diagram illustrating an example of data stored in the processing target list according to the second embodiment. FIG. 6 is a flowchart illustrating an example of the flow of the data arrangement control process during the writing process by the data arrangement control unit according to the second embodiment. FIG. 7 is a flowchart illustrating an example of a flow of write control processing by the data arrangement control unit according to the second embodiment. FIG. 8 is a flowchart illustrating an example of a flow of a writing process by the writing unit according to the second embodiment. FIG. 9 is a flowchart illustrating an example of a flow of pre-write execution processing in the data access unit according to the second embodiment. FIG. 10 is a flowchart illustrating an example of the flow of the write execution process in the data access unit according to the second embodiment. FIG. 11 is a diagram schematically showing an example of a normal processing flow in the data matching system according to the second embodiment. FIG. 12 is a flowchart illustrating an example of the flow of the data arrangement control process during the reading process by the data arrangement control unit according to the second embodiment. FIG. 13 is a flowchart illustrating an example of a flow of reading processing by the reading unit according to the second embodiment. FIG. 14 is a flowchart illustrating an example of the flow of read execution processing in the data access unit according to the second embodiment. FIG. 15 is a diagram for explaining the state of redundant data when the data access unit according to the second embodiment abnormally ends during the writing process. FIG. 16 is a flowchart illustrating an example of the flow of data matching processing in the data matching apparatus according to the second embodiment. FIG. 17 is a flowchart illustrating an example of the flow of the identification process in the data matching apparatus according to the second embodiment. FIG. 18 is a diagram showing that the information processing by the data matching program, which is a program for executing a series of processes by the data matching system, is specifically realized using a computer.

  Hereinafter, embodiments of a data matching apparatus, a data matching method, and a data matching program according to the present invention will be described in detail with reference to the drawings. In addition, this invention is not limited by this embodiment.

[First Embodiment]
[Example of configuration of data matching device]
FIG. 1 is a block diagram showing an overview of a data matching apparatus 10 according to the first embodiment of the present invention. A data matching apparatus 10 according to the first embodiment will be described with reference to FIG. The data matching apparatus 10 according to the first embodiment detects an inconsistency of redundant data that occurs when a plurality of redundant data created by duplicating the same data is stored in different storage devices. Match the data length of the.

  As shown in FIG. 1, the data matching apparatus 10 includes a comparison unit 11 and an identification unit 12. The comparison unit 11 performs a comparison process for comparing the data lengths of the redundant data with respect to a plurality of storage units (not shown) that respectively store redundant data created by duplicating data. If the comparison unit 11 determines that the data length of the redundant data is not the same as a result of the comparison process, the identification unit 12 adds the data of the other redundant data to the redundant data having the shortest data length among the redundant data. Align the length.

  The data matching device 10 may be, for example, a master device of a distributed file system. Further, for example, the function of the data matching apparatus 10 may be configured to be realized by a client library. However, the present invention is not limited to a distributed file system, and can be applied to a system that has a plurality of independent storage units and stores the same replication data in each storage unit to ensure system redundancy. For example, the present invention can be applied to a system in which parallel processing for each storage unit is executed by external access, and a plurality of external instructions are adjusted and processed.

  Although not shown in FIG. 1, the data matching device 10 is connected to a plurality of storage units for storing redundant data, an information processing device for transmitting data read requests and write requests to the storage units, and the like via a network. Connected. The plurality of storage units may be, for example, a physical server and a disk device connected to the server, a virtual server and a virtual storage that are virtually constructed in the data center, and the like. Further, the information processing apparatus may be a client apparatus that uses a physical server or a data center.

  Further, any network such as a wide area network (WAN) or a local area network (LAN) can be used as the network.

[Example of data consistency processing flow]
FIG. 2 is a flowchart illustrating an example of the flow of data matching processing of the data matching apparatus 10 according to the first embodiment. With reference to FIG. 2, an example of the flow of data matching processing by the data matching apparatus 10 according to the first embodiment will be described.

  As shown in FIG. 2, the data matching apparatus 10 first receives a matching request (step S21). For example, the matching request is issued from a client connected to the data matching apparatus 10 and using data stored in the storage unit. For example, the client issues a matching request to the data matching apparatus 10 when, for example, the process of writing data in the storage unit ends abnormally. However, the matching request may be automatically issued to the data matching apparatus 10 when it is detected that the writing process to the storage unit has ended abnormally in the client.

  The matching request is passed to the comparison unit 11. The comparison unit 11 accesses a plurality of storage units that store redundant data specified by the matching request, and extracts the data length of the stored redundant data (step S22). Then, the comparison unit 11 compares the extracted data lengths with each other (step S23). When the comparison unit 11 determines that the data lengths are the same (Yes at Step S24), the data matching process ends. On the other hand, when the comparison unit 11 determines that the data lengths are not the same (No at Step S24), the comparison unit 11 passes the processing to the identification unit 12. The identification unit 12 aligns the data lengths of the redundant data compared by the comparison unit 11 (step S25). That is, the identifying unit 12 aligns the data length of other redundant data with the redundant data having the shortest data length among the plurality of redundant data. This completes the data matching process.

[Effect of the first embodiment]
As described above, the data matching apparatus 10 according to the first embodiment includes the comparison unit 11 that compares the data length of redundant data that is a duplicate of the same data stored in each of the plurality of storage units, and the comparison unit 11. When the data lengths of the redundant data are not the same, the identification unit 12 aligns the data length of the other redundant data to the data length of the shortest redundant data among the redundant data stored in the plurality of storage units. . For this reason, the data matching apparatus 10 according to the first embodiment can ensure consistency among redundant data stored in a plurality of storage units.

  Further, the data matching apparatus 10 may be configured to perform data matching processing when receiving a matching request, and not to deal with inconsistencies between redundant data unless particularly necessary. With this configuration, it is possible to execute redundancy data consistency control in accordance with the needs of applications used on the client side.

  For example, if an application used on the client side can operate without any problem even if there is some inconsistency in redundant data, the client continues processing without issuing an alignment request. Then, when the processing load on the client or the data matching apparatus 10 is low, a matching request can be issued to execute the data matching process. In addition, if the application used on the client side requires high consistency of redundant data, the client immediately issues a consistency request when the write operation is terminated unexpectedly. Consistency can be ensured.

  Further, the data matching apparatus 10 may be set to always execute the data matching process when an abnormal end of the writing process occurs. In addition, the data matching apparatus 10 executes the data matching process when the writing process corresponding to the writing request from the predetermined application is abnormally terminated, and the writing corresponding to the writing request from the other application is performed. It may be set so that the data matching process is not executed when the process ends abnormally.

  As described above, the data matching apparatus 10 according to the first embodiment ensures consistency between redundant data stored in a plurality of storage units while suppressing a load related to a higher-level application and preventing a decrease in processing performance. can do.

[Second Embodiment]
FIG. 3 is a block diagram showing an outline of the data matching system 1 according to the second embodiment of the present invention. With reference to FIG. 3, the outline of the configuration of the data matching system 1 according to the second embodiment will be described.

[Data alignment system]
As shown in FIG. 3, the data matching system 1 includes a master 100, servers 200A, 200B, and 200C, disks 300A, 300B, and 300C, and clients 400A, 400B, and 400C. The master 100 is connected to the servers 200A, 200B, and 200C and the clients 400A, 400B, and 400C via the network, and transmits and receives information to and from each other. Further, the clients 400A, 400B, and 400C appropriately transmit a read command, a write command, or a data matching command to the servers 200A, 200B, and 200C based on the information transmitted from the master 100. The master 100 receives a request for reading or writing data from the clients 400A, 400B, and 400C, and transmits information for reading or writing or data alignment to the clients 400A, 400B, and 400C. Servers 200A, 200B, and 200C execute read processing, write processing, or data matching processing on disks 300A, 300B, and 300C based on instructions from clients 400A, 400B, and 400C.

  The master 100 is, for example, a distributed file system master. Servers 200A, 200B, and 200C are, for example, chunk servers of a distributed file system. Clients 400A, 400B, and 400C are, for example, users who use a distributed file system.

  The master 100 may be configured as an intercloud server of an intercloud system. In that case, the servers 200A, 200B, and 200C and the disks 300A, 300B, and 300C may be configured virtually by a cloud system. Further, the master 100, the servers 200A, 200B, and 200C and the disks 300A, 300B, and 300C may be constructed in one data center or may be constructed in a distributed manner in a plurality of data centers.

  The servers 200A, 200B, and 200C access the subordinate disks 300A, 300B, and 300C in accordance with commands from the master 100 and the clients 400A, 400B, and 400C, and read and write data. Each of the servers 200A, 200B, and 200C includes data access units 201A, 201B, and 201C for controlling access to the subordinate disks 300A, 300B, and 300C.

  Each of the servers 200A, 200B, and 200C stores the same data as redundant data so that the data is not lost even if a failure occurs in any of the servers. The clients 400A, 400B, and 400C can execute parallel processing on the same data by accessing any of the servers 200A, 200B, and 200C. 3 shows three servers 200A, 200B, and 200C, the number of servers is not limited to three. In the second embodiment, the number of disks and servers that store redundant data will be described as 3. In the case of a data matching system including 10 servers, for example, redundant data of the same data is appropriately selected from 10 servers. It may be stored in a disk under the selected three servers. The master 100 determines and controls which server's disk stores redundant data, and instructs the clients 400A, 400B, and 400C. However, the clients 400A, 400B, and 400C may be configured to designate a server that stores redundant data.

  The disks 300A, 300B, and 300C are storage devices such as a database that stores data. Access to the disks 300A, 300B, and 300C is controlled by the servers 200A, 200B, and 200C, respectively.

  Clients 400A, 400B, and 400C use data stored in disks 300A, 300B, and 300C. For example, the clients 400A, 400B, and 400C are user terminals that use an intercloud system. Each of the clients 400A, 400B, and 400C includes application execution units 401A, 401B, and 401C, writing units 402A, 402B, and 402C, reading units 403A, 403B, and 403C, and data matching devices 410A, 410B, and 410C. Each of the application execution units 401A, 401B, and 401C is a functional unit that executes a client application. The writing units 402A, 402B, and 402C, the reading units 403A, 403B, and 403C and the data matching devices 410A, 410B, and 410C are respectively written to the disks 300A, 300B, and 300C under the control of the application execution units 401A, 401B, and 401C. Send command, read command, data match command.

[Example of configuration of data matching device]
The clients 400A, 400B, and 400C include data matching devices 410A, 410B, and 410C, respectively. The data matching devices 410A, 410B, and 410C will be further described below. The data matching device 410 (hereinafter, “data matching device 410” indicates any one of the data matching devices 410A, 410B, 410C) is compared with the comparison unit 412 (hereinafter, “comparison unit 412” is the comparison unit 412A, 412B and 412C) and an identifying unit 413 (hereinafter, “identifying unit 413” represents any one of the identifying units 413A, 413B, and 413C).

  The comparison unit 412 performs comparison processing in response to a matching request from the client. For example, the comparison unit 412 executes the comparison process when the writing process requested by the client is not normally completed. In the master 100 of the second embodiment, the same data is stored in the servers 200A, 200B, and 200C in order to ensure data redundancy. However, there may be a discrepancy between the data stored in the server 200A and the data stored in the server 200B, such as when the writing process ends abnormally. In addition, when the writing is abnormally terminated, one of the servers cannot be accessed from the outside, and it may not be possible to know from the outside (client) whether or not the data has been properly written to other servers. is there. Therefore, the comparison unit 412 detects the write target data of the abnormally terminated write process from the accessible server, and executes a comparison process for comparing the respective data lengths.

  When the comparison unit 412 compares the redundant data stored in a plurality of servers and the mismatch is detected in the data length, the identification unit 413 compares the data length of the data with the shortest data length. In addition, data equalization processing is performed to align the lengths of other redundant data. Note that the processing by the comparison unit 412 and the identification unit 413 will be described in more detail later.

[Example of master configuration]
An example of the configuration of the master 100 will be described. As shown in FIG. 3, the master 100 includes a data arrangement control unit 110, a storage unit 120, and an input / output unit 130. The data arrangement control unit 110 controls the function and operation of each unit in the master 100. The storage unit 120 appropriately stores information used for processing in the master 100 and information generated as a result of the processing. The input / output unit 130 exchanges information between the outside and the master 100. Information received by the input / output unit 130 is appropriately transferred to the data arrangement control unit 110 and the storage unit 120.

  The data arrangement control unit 110 determines a place where data is written and read, and executes processing for storing in the storage unit 120 information for identifying redundant data and information indicating the location of redundant data. Further, the information stored in the storage unit 120 is updated according to the results of the read process, the write process, and the data matching process in the servers 200A, 200B, and 200C. Further, the data arrangement control unit 110 transmits information necessary for writing and reading data and data alignment to the clients 400A, 400B, and 400C in response to requests from the clients 400A, 400B, and 400C.

[Example of configuration of storage unit and stored information]
Next, the configuration of the storage unit 120 and an example of information stored in the storage unit 120 will be described. As illustrated in FIG. 3, the storage unit 120 includes a redundant data position table 121 and a processing target list (lease list) 122.

  The redundant data position table 121 stores an identifier (ID: Identifier) of data stored in each of the servers 200A, 200B, and 200C and information of the server that stores the data. FIG. 4A is a diagram for explaining an example of information stored in the redundant data position table 121. FIG. 4B is a diagram for explaining the correspondence between the information stored in the redundant data position table 121 and the disk storing the data.

  For example, it is assumed that the file name, that is, the data of the file ID “A” is stored in the disks 300A, 300B, and 300C in response to a write request from the client 400A. In this case, the data of the file ID “A” is first divided into fixed lengths to form a plurality of data blocks (see FIG. 4A). A handle name and a version number are assigned to each block. A combination of the handle name and the version number is the data ID of the block. For example, in the example of FIG. 4A, the data of the file ID “A” is divided into three fixed-length blocks. Then, the handle name “a1” and the version number “2” are assigned to the first block. Therefore, the data ID of the first block of the data of the file ID “A” is “a12”, and the data ID is stored in the redundant data position table 121 in association with the file ID “A”. Similarly, the handle name “a2” and the version number “3” are assigned to the second data block and stored in the redundant data position table 121. In addition, the handle name “a3” and the version number “3” are assigned to the third data block and stored in the redundant data position table 121.

  Of the data IDs assigned to each data block, for the handle name, a different handle name is assigned to each block so that each data block can be uniquely identified. Regarding the version number, the same version number may be assigned to a plurality of data blocks. Moreover, as long as a data block can be specified uniquely, ID of another format may be given.

  In this embodiment, redundant data of each data block is stored in three disks 300A, 300B, and 300C. Therefore, the redundant data position table 121 stores, for example, position information indicating the location where the redundant data is stored for each block of the data with the file ID “A”. In the example of FIG. 4B, the redundant data of the data block with the data ID “a33” in the data with the file ID “A” is transferred to the disk controlled by the servers with the server IDs “200A”, “200B”, and “200C”. It is shown that it is stored. That is, here, the location information specifies the server on which the data access unit that manages the data of the data block is operating.

  As position information indicating the location where redundant data is stored, for example, the IP address and host name of the server, the IP address and host name of the data access unit operating on the server, and the like can be used.

  The processing target list 122 is a list that identifies the server or data access unit that is executing the writing process and the processing target data at that time. FIG. 5 is a diagram illustrating an example of data stored in the processing target list 122.

  The processing target list 122 stores information for specifying the data block that is executing the writing process at that time, and information for specifying the functional unit that controls the writing process to the data block. For example, when the data access unit “201A” controls the writing process for the data block with the data ID “a12”, the processing target list 122 includes “a1”, which is the handle name of the data, and the data access The position information “201A” of the part “201A” is stored in association with each other. When the data access unit “201A” finishes writing the data block with the handle name “a1”, after a predetermined time has elapsed, the data placement control unit 110 changes the handle name “a1” from the processing target list 122. The information of the data access unit “201A” is deleted. Here, it is assumed that the processing target list 122 stores only the handle name of the processing target data and does not store the version number.

[Example of data placement control process flow during write process]
FIG. 6 is a flowchart illustrating an example of the flow of data arrangement control processing by the master 100 according to the second embodiment. With reference to FIG. 6, the flow of data arrangement control processing by the master 100 will be described.

  First, for example, the client 400A transmits a write request for requesting writing of data of the file ID “A” to the master 100. The write request includes a file ID of a file to which data is to be written and an offset “Z” indicating the position of the write data in the file. The write request is also transmitted by the identifying unit 413 described later. The processing in this case is similar to the following.

  When receiving a write request (step S601), the data arrangement control unit 110 specifies a position to write data based on information included in the write request (step S602). For example, when the offset “Z” included in the write request is a positive number, the data arrangement control unit 110 adds the offset “Z” and a predetermined block size (fixed length) to obtain a block Divide by size. The data arrangement control unit 110 determines that the block located at the place corresponding to the quotient obtained as a result is a block to which data is written. When the offset “Z” included in the write request is a negative number, the data arrangement control unit 110 determines that the end block of the file is a block in which data is written.

  Next, the data arrangement control unit 110 tries to acquire the data ID of the block to which data is written (step S603). That is, the data arrangement control unit 110 refers to the redundant data position table 121 and determines whether or not information about the data of the corresponding data ID is stored. If the information is not stored (No at Step S603), the data arrangement control unit 110 determines that the corresponding data is not yet stored on any disk, and creates a new handle name (Step S604). . Then, the data arrangement control unit 110 determines a disk (that is, a corresponding server or data access unit) that stores redundant data of the data (step S605). For example, when the number of redundant data determined to be created in advance is 3, the data placement control unit 110 creates a server that operates under “3-1” or more switches as the number of redundant data created “3”. "Only.

  Then, the data placement control unit 110 registers the data ID based on the handle name created in step S604 and the server location information selected in step S605 in the redundant data location table 121 in association with each other. When the registration to the redundant data position table 121 has failed (No at Step S606), the data arrangement control unit 110 transmits that the writing process has failed to the client 400A (request source) that has issued the write request. (Step S607). Then, the writing process is finished. If the registration to the redundant data position table 121 is successful (Yes in step S606), the data ID of the data writing block is acquired from the redundant data position table 121 in step S603, and the redundant data position table is stored in step S608. This is the same processing as when position information is acquired from 121 (described later).

  Returning to step S603, when the data placement control unit 110 acquires the data ID of the data writing block from the redundant data position table 121 (Yes in step S603), next, redundant data corresponding to the data ID is stored. Position information is acquired (step S608). That is, with reference to the redundant data position table 121, the position information stored in association with the data ID is acquired. When the position information cannot be acquired (No at Step S608), the process proceeds to Step S607, and the writing process failure is notified to the client 400A. Then, the process ends.

  If the position information can be acquired (Yes at Step S608), the data arrangement control unit 110 executes a write control process for the server or the data access unit indicated by the position information (Step S609). FIG. 7 is a flowchart illustrating an example of a flow of a write control process by the master 100 according to the second embodiment. With reference to FIG. 7, an example of the flow of the write control process of the data arrangement control unit 110 will be described.

  When the data placement control unit 110 acquires the data ID corresponding to the write request received from the client 400A and the position information indicating the write location (FIG. 6, step S608, positive), the handle name of the acquired data ID is Then, it is determined whether or not it is registered in the processing target list 122 (step S701). When the handle name of the data ID is registered in the processing target list 122, it means that the writing process of the data block corresponding to the data ID is being executed. Therefore, when the handle name of the data ID is registered in the processing target list 122 (Yes in step S701), the data arrangement control unit 110 ends the write control process as it is.

  If the data ID is not registered in the processing target list 122 (No at Step S701), it means that the writing process for the data block corresponding to the data ID is not being executed. Therefore, the data arrangement control unit 110 proceeds to the next process, and selects one data access unit (or corresponding server or disk) included in the position information acquired in step S608 (step S702). For example, the data arrangement control unit 110 randomly selects a data access unit. Further, for example, a data access unit having the smallest number of switches via the client 400A may be selected. Then, the data arrangement control unit 110 transmits a write control command to the data access unit corresponding to the selected position information (step S703). The write control instruction includes information for specifying a data block to be written. The write control command is a command to perform write control for the data handle to be written.

  Then, the data arrangement control unit 110 determines whether or not a command reception response has been received from the data access unit that is the transmission destination of the write control command (step S704). The command reception response does not mean that the write control process and the write process have been completed in response to the write control command, but means that the write control command has been received.

  If the data placement control unit 110 determines that the command reception response has not been received (No at Step S704), the data placement control unit 110 returns to Step S702 again to reselect the data access unit (Step S702), and the write control command. Is transmitted (step S703). As a case where the command reception response is not received, for example, a time-out or a case where the connection is cut off can be considered. On the other hand, if the data arrangement control unit 110 determines that the command reception response has been received (Yes in step S704), the data arrangement control unit 110 executes step from the position information stored in the redundant data position table 121 in association with the data ID. The position information corresponding to the data access unit selected in S702 is extracted, and the redundant data position table 121 is updated so that the position information comes to the top of the list (step S705). Then, the data placement control unit 110 adds, to the processing target list 122, the handle name of the write target data that is the target of the write control command and the position information of the data access unit that is the transmission destination of the write control command. Registration is performed (step S706). Then, the data arrangement control unit 110 updates the version number among the data IDs of the write target data registered in the redundant data position table 121 (step S707). The data arrangement control unit 110 notifies the updated version number to the data access unit included in the position information stored in association with the data ID (step S708). Thereby, the write control process by the data arrangement control unit 110 is completed.

  Returning to FIG. 6 again, the writing process will be described. When the write control process by the data arrangement control unit 110 in step S609 is completed, the data arrangement control unit 110 transmits the information registered in the redundant data position table 121 to the client 400A that is the transmission source of the write request (step S609). S610). That is, the data placement control unit 110 associates the data ID of the data block that is the target of the write request stored in the redundant data position table 121 with the data ID through the write control process of FIG. Position information is transmitted to the client 400A.

  Then, the data arrangement control unit 110 determines whether or not the transmission process to the request source has been successful based on whether or not a reception response indicating that the information from the client 400A has been received has been received (step S611). When it is determined that the transmission process is successful (Yes in step S611), the data arrangement control unit 110 ends the data arrangement control process. If it is determined that the transmission process has failed (No in step S611), the data arrangement control unit 110 determines whether or not the number of times the transmission process has been performed is N or less (N is a natural number of 1 or more). (Step S612). If it is determined that the number is N or less (step S612, affirmative), the data arrangement control unit 110 returns to step S610 and transmits information again. If it is determined that it is not less than N times (No at Step S612), the data arrangement control unit 110 ends the data arrangement control process without transmitting any more information. Thereby, the data arrangement control process by the data arrangement control unit 110 at the time of the writing process is completed.

[Example of flow of writing process by writing unit]
As shown in FIGS. 6 and 7, when the data arrangement control unit 110 executes the data arrangement control process and the write control process, the data necessary for executing the process is transmitted to the request source (client). The The client uses the received data to access the data access unit and execute data writing and reading. Next, an example of the flow of writing processing from the client side will be described.

  FIG. 8 is a flowchart illustrating an example of a flow of a writing process by the writing unit according to the second embodiment. FIG. 8 is a diagram illustrating an example of a flow of writing processing by the writing unit 402A included in the client 400A, for example. With reference to FIG. 8, an example of the flow of the writing process by the client 400A will be described.

  First, when the client 400A intends to execute a data write process, the application execution unit 401A generates a write request. The write request includes a file ID of data to be written and an offset indicating the position of the end of the file. The file ID and offset included in the write request are sent from the application execution unit 401A to the writing unit 402A together with the data to be written (step S801). The write request received by the writing unit 402A is transmitted from the writing unit 402A to the master 100 (see S601 in FIG. 6).

  In addition, the data arrangement control unit 110 of the master 100 that has received the write request stores the data ID of the data block to which data is to be written in response to the write request and the position information of the data block in the redundant data position table 121. The information is acquired by referring to the writing unit 402A. Thereby, the writing unit 402A acquires the data ID of the data block to be written and the position information of the data block (step S802).

  Next, the writing unit 402A obtains, from the acquired position information, information on a data access unit that controls write access to a server or a write target data block under the disk in which the write target data block is stored. Extract. Then, one server or data access unit closest to the writing unit 402A is selected from the extracted servers or data access units (step S803). Here, “closest” means, for example, that the number of switches interposed between the writing unit 402A and the server or the data access unit is the smallest, and further, the number of switches passing therethrough is the smallest. When there are a plurality of data access units, the value of “log (“ own IP address ”EXOR“ partner IP address ”)” is the smallest value, and there are a plurality of data access units with the same value. In this case, the previously acquired data access unit is selected from the acquired position information among the same data access units.

  Write unit 402A selects a data access unit, for example, data access unit 201A. Then, writing unit 402A transmits information for writing processing to selected data access unit 201A (step S804). For example, the writing unit 402A has a data ID of the writing target block, an offset of the writing target block, data to be actually written (obtained in step S801), position information (obtained in step S802), message ID, , Send. The message ID is an identifier generated and assigned by the writing unit 402A so as to be unique in the data matching system 1.

  Then, after transmitting the information for the writing process to the data access unit 201A, the writing unit 402A sends the write command associated with the message ID to the first server in the list (ie, the position information) The data is transmitted to a data access unit that performs write control (step S805). The write command instructs writing of data associated with the message ID.

  Then, writing unit 402A determines whether or not a command reception response has been received from the data access unit that is the transmission destination of the write command (step S806). For example, a predetermined time is set in advance, and if a command reception response is received within the predetermined time, it is determined that it has been received. If a command reception response is not received within the predetermined time, it is received. What is necessary is just to determine that it is not. If a command reception response is received (Yes at step S806), the writing unit 402A determines whether or not the application execution unit 401A can be notified that the command reception response from the data access unit 201A has been received. (Step S807). If the writing unit 402A determines that notification can be made (Yes in step S807), the writing unit 402A notifies the application execution unit 401A (step S808). Then, the writing process ends normally (step S809).

  On the other hand, when it is determined in step S807 that the writing unit 402A cannot notify that the instruction reception response has been received (No in step S807), the writing process is abnormally terminated without performing notification (step S810). ).

  In Step S806, when it is determined that the command reception response has not been received from the data access unit 201A (No in Step S806), the writing unit 402A notifies the application execution unit 401A that the writing command has failed. It is determined whether or not it can be performed (step S811). If it is determined that notification can be made (Yes at Step S811), the writing unit 402A notifies the application execution unit 401A that the write command has failed (Step S812). Then, the writing process ends abnormally (step S813). On the other hand, if it is determined that notification is not possible (No in step S811), the writing process is abnormally terminated without performing notification (step S810). This completes the writing process in the client 400A.

[Example of the flow of pre-write execution processing in the data access unit]
FIG. 9 is a flowchart illustrating an example of a flow of pre-write execution processing in the data access unit according to the second embodiment. With reference to FIG. 9, an example of the flow of the writing process in the data access unit will be described.

  First, for example, the data access unit 201A receives information for executing data writing processing from the writing unit 402A or another data access unit (step S901). The information to be received is, for example, information transmitted by the writing unit 402A in step S804 in FIG. In the present embodiment, information for writing is transmitted from the writing unit 402A to one data access unit, and transmitted from the data access unit to another data access unit.

  The data access unit 201A extracts position information from the received information, and determines whether or not a data access unit other than itself is included therein (step S902). If no other data access unit is included in the position information (No at Step S902), the data access unit 201A transmits a notification to the effect that the information has been successfully received to the client 400A that is the transmission source of the write request ( Step S903).

  On the other hand, when another data access unit is included in the position information (Yes at step S902), the data access unit 201A selects one other data access unit (step S904). At this time, the data access unit 201A may select the data access unit based on the same criteria as in step S803 in FIG. Then, the data access unit 201A creates position information excluding its own position information.

  Next, the data access unit 201A replaces the position information in the received data for writing processing with the position information created in step S904, and transmits it to the data access unit selected in step S904 (step S905). . Then, the data access unit 201A waits for a response indicating that the information has been received. If a response indicating that the information has been received is received (Yes at Step S906), the data access unit 201A proceeds to Step S903, and transmits a notification that the information reception has been successful to the transmission source client 400A. On the other hand, when the response indicating that the information has been received is not received (No at Step S906), the data access unit 201A transmits a notification that the information reception has failed to the transmission source client 400A (Step S907). ). As a result, the processing in the data access unit 201A ends.

[Example of flow of write execution processing in data access unit]
Next, the flow of write execution processing in the data access unit will be described with reference to FIG. FIG. 10 is a flowchart illustrating an example of the flow of the write execution process in the data access unit according to the second embodiment. In this embodiment, when writing redundant data to a plurality of servers, first, information for writing processing is transmitted to one server, and information is sequentially transmitted from the server to other corresponding servers. Then, when executing the writing, the writing unit transmits a writing command to the server that performs the writing first in accordance with a predetermined processing order. For other servers, a writing command is transmitted from the server that performs the writing first, not the writing unit. FIG. 10 shows a write execution process in a server that performs writing first (that is, a data access unit that performs write control).

  First, for example, the data access unit 201A receives a write command from the writing unit 402A (step S1001). Until then, the data access unit 201A has received data for executing the writing process, for example, by the process of step S901 shown in FIG. Then, in response to the write command, the data access unit 201A executes a process of writing the write target data at the designated position (step S1002). At this time, the data access unit 201A specifies the data to be written based on the message ID included in the write command received from the writing unit 402A. Next, the data access unit 201A transmits a write command to other data access units in the position information included in the data received in step S703 (FIG. 7) (step S1003). Then, the data access unit 201A determines whether or not the command has been transmitted (step S1004). If it is determined that the command could not be transmitted (No at Step S1004), the data access unit 201A does not perform any more processing and ends abnormally (Step S1008).

  On the other hand, when it is determined that the command has been transmitted (Yes at step S1004), the data access unit 201A determines whether there has been a command success response from the data access unit that is the transmission destination of the write command (step S1004). S1005). If it is determined that there has been a command success response from all the data access units (Yes at step S1005), the data access unit 201A transmits a notification to the effect that the writing has been completed to the transmission source client 400A (step S1006). . Then, the data access unit 201A normally ends the write execution process (step S1007). On the other hand, if it is determined that instruction success responses from all the data access units have not been received (No at step S1005), the data access unit 201A does not perform any further processing and ends abnormally (step S1008). ). This completes the write execution process.

[Processing Flow in Data Matching System According to Second Embodiment]
FIG. 11 is a diagram schematically showing an example of a normal processing flow in the data matching system 1 according to the second embodiment. With reference to FIG. 11, an example of a normal processing flow in the data matching system 1 according to the second embodiment will be described.

  As shown in FIG. 11, the client 400 (hereinafter, “client 400” indicates one of the clients 400A, 400B, and 400C) transmits a write request to the master 100 ((( 1)). The write request includes a file ID and an offset. The master 100 that has received the write request refers to the redundant data position table 121 based on the file ID and the offset, and extracts the data ID and position information of the requested target data for the writing process. If the corresponding data ID is not registered in the redundant data position table 121, a new handle name is created and registered ((2) in FIG. 11).

  Further, the master 100 refers to the processing target list 122 ((3) in FIG. 11), and if the handle name of the write target data is not registered, the master 100 selects one data access unit that performs the write processing control. ((4) in FIG. 11). Then, a write control command is transmitted to the selected data access unit ((5) in FIG. 11). When receiving a response from the data access unit ((6) in FIG. 11), the master 100 updates the redundant data position table 121 and the processing target list 122 ((7) in FIG. 11), and the updated version number Is notified to the data access unit ((8) in FIG. 11). Thereafter, the master 100 transmits to the client 400 the data ID of the target data to be written and the corresponding position information ((9) in FIG. 11).

  The client 400 includes the file ID and offset included in the write request (see (1) in FIG. 11), the data to be written, the data ID and position information received from the master 100 ((9) in FIG. 11), The message ID assigned to be unique in the client 400 is transmitted to the data access unit selected based on the same criteria as in step S803 in FIG. 8 ((10) in FIG. 11). The data access unit that has received the information refers to the location information, and if the other data access unit is included in the location information, deletes its own information from the location information, and then sends it to the other data access unit. The received information is transmitted ((11) in FIG. 11). Other data access units that have received the information also execute the same processing ((12) in FIG. 11).

  Thereafter, the client 400 transmits a write execution command to the data access unit (see (4) in FIG. 11) that performs write control ((13) in FIG. 11). The data access unit that has received the write execution command writes the data to the subordinate disk, and then transmits the write execution command to other data access units (if there are a plurality of data access units) included in the position information ( (14) of FIG. When the data is written to all the corresponding disks, a completion response is transmitted from the other data access unit to the data access unit that performs write control ((15) in FIG. 11). When the data access unit that performs the write control receives all the completion responses, it transmits the completion response to the client ((16) in FIG. 11). The writing process at the normal time is completed in this way.

[Example of flow of data arrangement control processing during read processing]
FIG. 12 is a flowchart illustrating an example of the flow of the data arrangement control process during the reading process by the data arrangement control unit 110 according to the second embodiment. With reference to FIG. 12, an example of the flow of the data arrangement control process during the reading process in the master 100 according to the second embodiment will be described.

  First, for example, the client 400A transmits a read request to the master 100. The read request is also transmitted by the comparison unit 412 described later. The processing in this case is similar to the following. The read request includes the file ID of the target file to be read and the offset “Z” indicating the end position of the target file to be read. Upon receiving the read request (step S1201), the data arrangement control unit 110 of the master 100 extracts the file ID and the offset “Z” included in the read request. Then, based on the extracted information, the data arrangement control unit 110 specifies from which block of the file the reading is to be performed (step S1202). This process is the same as step S602 in FIG.

  Next, the data arrangement control unit 110 tries to acquire the data ID of the block data corresponding to the identified block from the redundant data position table 121 (step S1203). When the data ID cannot be acquired (No at Step S1203), the data arrangement control unit 110 notifies the client 400A (request source) that has transmitted the read request that the read process has failed (Step S1204). ) Finish the process. On the other hand, if the data ID can be acquired (Yes at step S1203), the data arrangement control unit 110 then tries to acquire the position information corresponding to the data ID from the redundant data position table 121 (step S1205). ). When the position information cannot be acquired (No at Step S1205), the client 400A is notified that the reading process has failed (Step S1204), similarly to the case where the data ID cannot be acquired (No at Step S1203). ) Finish the process.

  If the position information can be acquired (Yes at Step S1205), the data arrangement control unit 110 uses the data ID (Step S1203) and the position information (Step S1205) acquired from the redundant data position table 121 as the client 400A (request source). ) (Step S1206). Then, the data arrangement control unit 110 determines whether or not the transmission process is successful (step S1207). For example, it may be determined whether or not it has succeeded by receiving a reception response from the client 400A.

  If it is determined that the transmission process is successful (Yes at step S1207), the data arrangement control unit 110 ends the process. On the other hand, if it is determined that the transmission process has failed (No in step S1207), the data arrangement control unit 110 determines whether the number of transmission process failures is N or less (N is a natural number equal to or greater than 1). (Step S1208). If it is determined that the number is N times or less (Yes at Step S1208), the data arrangement control unit 110 returns to Step S1206 and transmits information to the request source again. On the other hand, if it is determined that it is not less than N times (No in step S1208), the data arrangement control unit 110 ends the process as it is. Here, N may be determined in advance as the upper limit value of the number of retries.

[Example of the flow of read processing by the read unit]
FIG. 13 is a flowchart illustrating an example of a flow of reading processing by the reading unit according to the second embodiment. With reference to FIG. 13, an example of the flow of reading processing by the reading unit (for example, 403A) will be described.

  First, the reading unit 403A obtains a file ID for reading data from the application execution unit 401A (step S1301). Next, the reading unit 403A acquires the offset and data length of data read from the application execution unit 401A (step S1302). Next, the reading unit 403A acquires, from the data arrangement control unit 110, the data ID of the block holding the data to be read and the position information associated with the data ID based on the acquired file ID, offset, and data length. (Step S1303). The process of the data arrangement control unit 110 when acquiring the data ID and the position information associated with the data ID corresponds to the process of FIG.

  Next, reading unit 403A selects one of the closest data access units from the position information acquired from data arrangement control unit 110 (step S1304). At this time, the reading unit 403A may select the data access unit based on the same criteria as in step S803 in FIG.

  Then, reading unit 403A transmits a read command to the selected data access unit to read data (step S1305). The read command is a command for reading data designated by the application execution unit 401A in the processing.

  The reading unit 403A transmits the read data to the application execution unit 401A (step S1306). This completes the reading process.

[Example of flow of read execution processing in data access unit]
FIG. 14 is a flowchart illustrating an example of the flow of read execution processing in the data access unit according to the second embodiment. With reference to FIG. 14, an example of the flow of processing in the data access unit when there is a read command from the reading unit 403A as shown in FIG. 13 will be described.

  First, the data access unit, for example, the data access unit 201A receives a read command from the read unit 403A (step S1401). In response to the read command, the data access unit 201A reads the corresponding data from the subordinate disk (step S1402). The data access unit 201A transmits the read data to the reading unit 403A (step S1403). This completes the read execution process in the data access unit.

[Redundant data status at abnormal termination]
FIG. 15 is a diagram for explaining the state of redundant data when the data access unit abnormally terminates during the writing process. With reference to FIG. 15, the state of redundant data that may occur when the writing process ends abnormally will be described.

  For example, if the data access unit 201A does not receive an instruction success response from another data access unit in step S1005 of FIG. 10, normal writing of redundant data may not be completed in the other data access unit. Conceivable. If the data access unit that executes the write control for executing the processing of FIG. 10 stops for some reason after transmitting a write command to another data access unit (step S1003, FIG. 10), the data access The part becomes inaccessible from the outside. In addition, since the other data access units cannot send a response indicating whether or not the writing is successful to the data access unit, whether or not the redundant data writing is successful in these data access units, I can't know from outside.

  As shown in FIG. 15, consider a case where the data of the file “A” is divided into three blocks and the redundant data with the data ID “a33” is to be stored on three disks. In this case, if the server that attempted to execute writing first stops, it is unclear whether redundant data has been written to the disk corresponding to the server, and data cannot be accessed on the disk. . This state is shown in FIG. Thus, when the server stops and the success or failure of writing becomes unknown, the server location information stored corresponding to the data ID is deleted in the redundant data location table 121.

  Further, the data access unit that notifies the data access unit of the stopped server of the success or failure of writing cannot know whether or not the writing is successful. Therefore, the states of the write data controlled by these data access units are as shown in (2) and (3) of FIG.

[Example of data consistency processing flow]
FIG. 16 is a flowchart illustrating an example of the flow of data matching processing in the data matching apparatus 410 according to the second embodiment. An example of the flow of data matching processing in the data matching apparatus 410 according to the second embodiment will be described with reference to FIG.

  First, when the client 400 intends to execute the data matching process, a matching request is generated. The data matching apparatus 410 (comparison unit 412) receives the generated matching request (step S1601). For example, in the data access unit 201A, 201B, or 201C, the data write process does not end normally and any of the clients 400A, 400B, and 400C that is the request source of the write request Generated by the clients 400A, 400B, and 400C, for example, when a write completion notification is not transmitted. In the clients 400A, 400B, and 400C, the application execution units 401A, 401B, and 401C may determine whether to send a consistency request according to the importance of the write target data and the contents of the processing. The matching request includes a file ID to be matched. The matching request is also a request for aligning the data length of the last block of the specified file ID.

  The data matching apparatus 410 that has received the matching request corresponds to the data ID corresponding to the last block of the data blocks written with the file ID included in the matching request from the redundant data position table 121 and the data ID. An attempt is made to acquire attached position information (step S1602). If the data matching apparatus 410 cannot acquire the data ID and the position information (No at Step S1602), the data matching process is ended as it is. The acquisition of the data ID and position information may be performed in the same manner as the procedure shown in FIG.

  On the other hand, if the data matching apparatus 410 can acquire the data ID and the position information (Yes in step S1602), it next determines whether or not a plurality of data access units are included in the acquired position information. (Step S1603).

  If it is determined that a plurality of data access units are not included (No in step S1603), the data matching process is terminated as it is. On the other hand, if it is determined that a plurality of data access units are included (Yes in step S1603), the comparison unit 412 stores the matching request (specifically, the step stored in the disk controlled by each data access unit). The data length of the data block corresponding to the data ID specified in S1062) is acquired (step S1604). If the data length could not be acquired (No at step S1605), the process returns to step S1602. On the other hand, if the data length can be acquired (Yes in step S1605), the comparison unit 412 next determines whether or not the data lengths of the plurality of data blocks all match (step S1606). If the comparison unit 412 determines that the data lengths match (Yes in step S1606), the comparison unit 412 ends the data matching process. On the other hand, when it is determined that the data lengths do not match (No at Step S1606), the comparison unit 412 passes the process to the identification unit 413, and the identification unit 413 executes the identification process (Step S1607). . The identification process will be described later in detail. When the identification process is completed, the data matching process ends.

[Example of identification process flow]
FIG. 17 is a diagram illustrating an example of the flow of the identification process in the data matching apparatus 410 according to the second embodiment. With reference to FIG. 17, an example of the flow of the identification process by the identification unit 413 will be described. In the example of FIG. 17, it is assumed that the number of redundant data compared by the comparison unit 412 is two.

  First, when the comparison unit 412 determines that the data lengths of a plurality (two) of data do not match (No in step S1606 in FIG. 16), the identification unit 413 determines that the two data blocks compared by the comparison unit 412 The shorter one of the data lengths is selected (step S1701). Here, the two data blocks compared are referred to as “data A” and “data B”, respectively. The identifying unit 413 determines whether the selected data length is that of the data A (step S1702). If it is determined that the selected data length is that of data B (No at step S1702), the identifying unit 413 truncates the data A so that the data length of the data A matches the data B (step S1703). On the other hand, when it is determined that the selected data length is that of data A (Yes in step S1702), the identifying unit 413 truncates the length of data B so that the data length of data B matches the data A. (Step S1704). The identification unit 413 notifies the requester of the matching request that the identification process has been completed (step S1705).

  When the identification unit 413 performs the identification process, a process similar to the data arrangement control process illustrated in FIG. 6 is executed. That is, the identifying unit 413 transmits a command including a file ID and an offset designating a negative number to the data arrangement control unit 110 in order to acquire a target data block whose data length is uniform. If the position controller 110 designates a negative number for the offset, the data is determined to be in agreement with the designation of the last block of the file. Therefore, in response to the command, the data placement control unit 110 transmits the data ID and position information at the end of the specified file ID to the identification unit 413 and sends a control command to the data access unit. The subsequent processing is the same as that illustrated in FIG.

  Note that the reading process when the data matching process is performed is executed in the same manner as the processes shown in FIGS. However, if the disk is in an inaccessible state due to any abnormal termination or the like of the data access unit, the information of the data access unit is deleted from the position information in the redundant data position table 121. The content of the position information that can be acquired by the department has been changed from normal. As a result of the data matching process, the redundant data position table 121 and the processing target list 122 are updated as in the normal writing process.

[Effects of Second Embodiment]
As described above, the data matching system 1 according to the second embodiment is a system connected to one or more clients via a network, and stores a plurality of storage devices each storing redundant data that is a duplicate of the same data. And controlling access to redundant data stored in a plurality of storage devices, each of a plurality of servers connected to one of the plurality of storage devices, and each of the plurality of storage devices in response to a request from a client The comparison unit 412 that compares the data lengths of the redundant data to be stored and the comparison unit 412 determine that the data lengths of the redundant data are not the same, the shortest of the redundant data stored in the plurality of storage devices And an identification unit 413 that aligns the data length of the redundant data with the data length of the redundant data. For this reason, even if the writing process ends abnormally, the data lengths of the redundant data can be made uniform according to the selection on the higher application side. Therefore, it is possible to prevent inconsistency between redundant data stored in different storage devices and improve data reliability. In addition, the upper application can issue an alignment request after determining whether data alignment processing is necessary according to the characteristics of the target data of the abnormally terminated write processing, and automatically rewrites upon abnormal termination Compared with a coping method such as performing the above, it is possible to reduce the load on the host application and suppress the degradation of the processing performance.

  In the data matching system 1 according to the second embodiment, each of the plurality of storage units stores the redundant data obtained by dividing the same file data into fixed-length blocks and duplicating the same data block. The comparison unit 412 stores the identifier of the data block and the positional information of two or more storage units selected from the plurality of storage units specified as the storage destination of the data block in association with each other. With reference to the data position table 121, the data length of the last data block is compared among redundant data of two or more storage units stored in association with the same data block. For this reason, when the data lengths match between the redundant data of the block at the end of the file, the process of aligning the data lengths is not executed. Therefore, a response can be returned to the upper application in a shorter processing time compared to the case where a mismatch between redundant data is detected using a checksum or the like. Further, the host application can maintain the consistency of redundant data only by selecting whether or not to execute the data matching process, and the processing load is reduced. If the data matching process is automatically executed when the writing process is abnormally terminated, the consistency between redundant data can be maintained even when the application execution unit itself stops.

[Third Embodiment]
Although the embodiments of the present invention have been described so far, the present invention may be implemented in other embodiments besides the above-described embodiments. Other embodiments will be described below.

[System configuration]
Of the processes described in the above embodiment, all or part of the processes described as being performed automatically can be performed manually, or all of the processes described as being performed manually or A part can be automatically performed by a known method. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above-described document and drawings can be arbitrarily changed unless otherwise specified.

  In the description of the above embodiment, the client 400A has been described as including the application execution unit 401A, the writing unit 402A, the reading unit 403A, and the data matching device 410A. However, some of these functions are incorporated into the master 100. May be. For example, the data placement control unit 110 of the master 100 may be configured to transmit a write command, a read command, or a data matching command to the data access units 201A, 201B, and 201C. Also, some of the functions of the writing units 402A, 402B, 402C, the reading units 403A, 403B, 403C, or the data matching devices 410A, 410B, 410C are performed by the data access units 201A, 201B, 201C of the servers 200A, 200B, 200C. May be executed.

  Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or a part of the distribution / integration may be functionally or physically distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. For example, in the example illustrated in FIG. 3, the writing unit 402A, the reading unit 403A, and the data matching device 410A are illustrated as a part of the client 400A, but the writing unit 402A, the reading unit 403A, and the data A part of the function of the matching device 410A may be incorporated in the master 100. For example, the operations or functions described as the operations or functions of the client 400 </ b> A can be incorporated in the master 100.

[program]
FIG. 18 is a diagram showing that the information processing by the data matching program that is a program for executing a series of processes by the data matching system 1 is specifically realized by using a computer. As illustrated in FIG. 18, the computer 3000 includes, for example, a memory 3010, a CPU (Central Processing Unit) 3020, a hard disk drive 3080, and a network interface 3070. Each part of the computer 3000 is connected by a bus 3100.

  The memory 3010 includes a ROM 3011 and a RAM 3012 as illustrated in FIG. The ROM 3011 stores a boot program such as BIOS (Basic Input Output System).

  Here, as illustrated in FIG. 18, the hard disk drive 3080 stores, for example, an OS 3081, an application program 3082, a program module 3083, and program data 3084. In other words, the data matching program according to the disclosed technique is stored in, for example, the hard disk drive 3080 as the program module 3083 in which instructions executed by the computer are described. For example, the data placement control unit 110, the data matching device 410, and a program module 3083 in which procedures for executing information processing similar to the processing in the data access unit are described are stored in the hard disk drive 3080.

  Further, data used for information processing by the data matching program, such as data stored in the storage unit 120, is stored as program data 3084 in, for example, the hard disk drive 3080. The CPU 3020 reads the program module 3083 and program data 3084 stored in the hard disk drive 3080 to the RAM 3012 as necessary, and executes various procedures.

  Note that the program module 3083 and the program data 3084 related to the data matching program are not limited to being stored in the hard disk drive 3080. For example, the program module 3083 and the program data 3084 may be stored in a removable storage medium. In this case, the CPU 3020 reads data via a removable storage medium such as a disk drive. Similarly, the program module 3083 and the program data 3084 related to the update program may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). . In this case, the CPU 3020 reads various data by accessing another computer via the network interface 3070.

[Others]
The data matching program described in the present embodiment can be distributed via a network such as the Internet. The data alignment program is recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM (Compact Disk Read Only Memory), MO (Magnetooptic Disc), DVD (Digital Versatile Disc). It can also be executed by being read from a recording medium by a computer.

DESCRIPTION OF SYMBOLS 1 Data alignment system 10,410A, 410B, 410C Data alignment apparatus 11 Comparison part 12 Identity part 100 Master 110 Data arrangement control part 120 Storage part 121 Redundant data position table 122 Processing object list 130 Input / output part 200A, 200B, 200C Server 201A, 201B, 201C Data access unit 300A, 300B, 300C Disk 400A, 400B, 400C Client 401A, 401B, 401C Application execution unit 402A, 402B, 402C Writing unit 403A, 403B, 403C Reading unit 412A, 412B, 412C Comparison unit 413A, 413B, 413C Identification unit 3000 Computer 3010 Memory 3011 ROM
3012 RAM
3020 CPU
3070 Network interface 3080 Hard disk drive 3081 OS
3082 Application program 3083 Program module 3084 Program data 3100 Bus

Claims (6)

A comparison unit that compares the data length of redundant data that is a copy of the same data stored in each of the plurality of storage units when the writing process from the upper application ends abnormally;
When the comparison unit determines that the data length of the redundant data is not the same, the data length of the other redundant data is set to the data length of the shortest redundant data among the redundant data stored in the plurality of storage units. An identification unit to be aligned,
A data matching apparatus comprising: Each of the plurality of storage units stores the redundant data obtained by dividing the same file data into fixed-length blocks and replicating the same data block,
The comparing unit stores the identifier of the data block and the positional information of two or more storage units selected from the plurality of storage units specified as the storage destination of the data block in association with each other. 2. The data length of a last data block of the redundant data in the two or more storage units stored in association with the same data block is compared by referring to a data position table. The data alignment device described. A comparison step of comparing the data length of redundant data, which is a copy of the same data, stored in each of the plurality of storage units, when the writing process from the upper application ends abnormally;
When it is determined in the comparison step that the data length of the redundant data is not the same, the data length of the other redundant data is set to the data length of the shortest redundant data among the redundant data stored in the plurality of storage units. The same identification process,
A data alignment method comprising: The plurality of storage units each store the redundant data obtained by dividing the same file data into fixed-length blocks and replicating the same data block,
The comparison step includes redundantly storing the identifier of the data block and the positional information of two or more storage units selected from the plurality of storage units specified as the storage destination of the data block in association with each other Referencing the data position table, comparing the data length of the last block among the redundant data of the two or more storage units stored in association with the same data block in the redundant data position table. The data matching method according to claim 3, wherein: A comparison procedure for comparing the data length of redundant data, which is a copy of the same data stored in each of the plurality of storage units, when the writing process from the upper application ends abnormally,
When it is determined in the comparison procedure that the data length of the redundant data is not the same, the data length of the other redundant data is set to the data length of the shortest redundant data among the redundant data stored in the plurality of storage units. The same identification procedure,
A data alignment program for causing a computer to execute The plurality of storage units each store the redundant data obtained by dividing the same file data into fixed-length blocks and replicating the same data block,
The comparison procedure is a redundancy in which an identifier of the data block and position information of two or more storage units selected from the plurality of storage units specified as storage destinations of the data block are stored in association with each other. Referencing the data position table, comparing the data length of the last block among the redundant data of the two or more storage units stored in association with the same data block in the redundant data position table. 6. The data matching program according to claim 5, wherein

Images