JP6284395B2 - Data storage control device, data storage control method, and program - Google Patents

Description

  The present invention relates to a data storage system that stores data in a plurality of data storage devices, such as a distributed file system.

  Due to the widespread use of cloud services, storage with higher reliability and higher speed is required. In response to such a request, a distributed file system in which file system functions are distributed to a plurality of storage nodes has been proposed.

  In the distributed file system, to which storage node data is arranged is determined using a hash function or the like. As a result, reliability is improved by distributing a plurality of nodes, and performance is improved by parallelizing input / output from the plurality of nodes. As a prior art related to the distributed file system, there is a technique described in Patent Document 1.

JP 2009-259007 A

  In a distributed file system, there are a primary node that can write data from a client and a replica node that stores a copy of the data and can only read data from the client in multiple storage nodes where data is placed. To do.

  However, in the prior art, a predetermined algorithm determines which storage node becomes a primary node and which storage node becomes a replica node, and control such as designating a specific storage node as primary cannot be performed. .

  Therefore, for example, when the client moves to a point far away from the primary node once determined, even if there is a storage node (replica node) nearby, data writing to the remote primary node is possible May have a problem that it takes time for processing.

  The present invention has been made in view of the above points, and provides a technique that enables a data storage device to be designated as a primary in a data storage system that stores data in a plurality of data storage devices. Objective.

According to an embodiment of the present invention, a data storage control device in a data storage system for storing data in a plurality of data storage devices,
Storage means for storing an ordered set of the plurality of data storage devices;
Order calculating means for changing the order of elements in the ordered set using designation information for designating a specific data storage device of the plurality of data storage devices;
In the data storage system, a data storage control device, wherein a data storage device corresponding to a predetermined element in the ordered set whose order is changed by the order calculation means is used as the specific data storage device. Provided.

According to the embodiment of the present invention, there is provided a data storage control method executed by a data storage control device in a data storage system for storing data in a plurality of data storage devices,
The data storage control device includes storage means for storing an ordered set of the plurality of data storage devices, and the data storage control method includes:
Using designation information for designating a specific data storage device among the plurality of data storage devices, comprising an order calculation step of changing the order of elements in the ordered set;
In the data storage system, a data storage control method, wherein a data storage device corresponding to a predetermined element in the ordered set whose order is changed by the order calculation step is used as the specific data storage device. Provided.

  According to the embodiment of the present invention, there is provided a technique capable of designating a primary data storage device in a data storage system that stores data in a plurality of data storage devices.

1 is an overall configuration diagram of a system according to an embodiment of the present invention. 3 is a functional configuration diagram of a data storage device 30. FIG. 3 is a functional configuration diagram of the data storage control device 10. FIG. It is a figure which shows the example of the table which the data storage control apparatus 10 hold | maintains. It is a figure for demonstrating the process which determines the storage location of primary data. It is a figure for demonstrating the concept of a placement group. It is a figure for demonstrating a replication route determination process. It is a figure for demonstrating a replication route determination process. It is a sequence diagram which shows the example of the flow of a process in this Embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment. For example, the present embodiment is directed to a distributed file system, but the present invention is not limited to a distributed file system and can be applied.

(System configuration)
FIG. 1 shows an overall configuration diagram of a system according to an embodiment of the present invention. The system according to the present embodiment is a distributed file system in which file system functions are distributed to a plurality of storage nodes (the data storage device 30 according to the present embodiment). Currently, there are various distributed file systems, and the distributed file system of the present embodiment is based on Ceph, but this is only an example, and the technology of the present embodiment is limited to the types of distributed file systems. It is applicable. In the present embodiment, functions / processing related to the present invention are mainly described. For example, the process of mapping data to a plurality of storage nodes using a hash function or the like is a general process of a distributed file system, and is simply described in the present embodiment.

  As shown in FIG. 1, the system according to the present embodiment has a configuration in which a data storage control device 10, a client device 20, and a data storage device 30 are connected to a network 40. In FIG. 1, one data storage control device 10 and one client device 20 are shown, but this is an example, and there may be a plurality of each. Further, the function of the data storage control device 10 can be provided in the client device 20 or any one of the data storage devices 30. In this case, the data storage control device 10 may not be provided. When the function of the data storage control device 10 is provided in the client device 20 or the data storage device 30, the client device 20 or the data storage device 30 may be referred to as a data storage control device.

  The data storage device 30 is a storage node in the distributed file system. File data is stored in a plurality of data storage devices 30. In the distributed file system according to the present embodiment, there is one primary data storage device 30 and other replica data storage devices 30 for one piece of data stored in the data storage device 30 group. Hereinafter, the primary data storage device 30 is referred to as a primary storage device, the replica data storage device 30 is referred to as a replica storage device, and the data stored in the primary storage device is primary data and the data stored in the replica storage device. Is called replica data (or replica, copy, etc.).

  The data storage device 30 that can write to the file from the client device 20 is only the primary storage device. Data can be read from the primary storage device or the replica storage device. In addition, a copy of the data stored in the primary storage device is stored in each replica storage device.

  FIG. 2 shows a functional configuration diagram of the data storage device 30. As illustrated in FIG. 2, the data storage device 30 includes a data storage unit 31, a distributed file system processing unit 32, and a replication control unit 33. The data storage unit 31 is a device such as a hard disk or a memory that stores data. The distributed file system processing unit 32 is a processing unit for causing the data storage device 30 to function as a storage node in the distributed file system, and the processing unit itself can be realized by existing technology. The replication control unit 33 is a functional unit that transfers data along a path instructed by the data storage control device 10 in order to store a copy of data to be stored in another data storage device 30.

  The client device 20 is a PC, for example, and is loaded with software on the client side of the distributed file system.

  FIG. 3 shows a functional configuration diagram of the data storage control device 10 according to the present embodiment. As shown in FIG. 3, the data storage control device 10 includes a data storage device order calculation unit 11, a primary specifier determination unit 12, a table information storage unit 13, a replication path determination unit 14, a path determination information storage unit 15, and A communication processing unit 16 is included. The outline of each functional part is as follows.

  Based on the information stored in the table information storage unit 13, the data storage device sequence calculation unit 11, for example, for each file, which of the data storage devices 30 that stores the data of the file is the primary storage device, Which one is used as a replica storage device is determined. The primary specifier determination unit 12 determines a primary specifier to be described later, for example, in units of files, and stores the determined primary specifier in the table information storage unit 13. The replication path determination unit 14 is a functional unit that determines a storage path when storing a replica of data from the primary storage device to the replica storage device. The route determination information storage unit 15 stores geographical information, network information, and the like used when the replication route determination unit 14 determines a route. The communication processing unit 16 performs communication with other devices.

  FIG. 4 shows an example of a table stored in the table information storage unit 13. The table shown in FIG. 4A is a table that stores a primary designator for each file (identification information such as a file number). This table is called a primary specifier table. The table shown in FIG. 4B is a table that stores an ordered set of data storage devices 30 that store file data for each file. This table is called an ordered set table.

  The data storage control apparatus 10 according to the present embodiment can be realized by causing a computer to execute a program describing the processing contents described in the present embodiment. That is, the function of the data storage control device 10 is to execute a program corresponding to the process executed by the data storage control device 10 using hardware resources such as a CPU, memory, and hard disk built in the computer. Can be realized. Further, the program can be recorded on a computer-readable recording medium (portable memory or the like), stored, or distributed. It is also possible to provide the program through a network such as the Internet or electronic mail.

  Hereinafter, a process for determining a primary storage device that is a storage location of primary data and a process for determining a replication path will be described in detail.

(Process to determine primary storage device)
As an existing function, the distributed file system according to the present embodiment has a function of determining a data storage device group for storing data using a hash algorithm or the like (mapping data and data storage device). The mapping process may be performed by the client device 20 or the data storage control device 10. Here, the “data” to be stored may be data in file units, or may be data in units obtained by dividing file data as will be described later.

  An ordered set of data storage devices 30 is obtained by the above mapping process. In the obtained ordered set, the first data storage device 30 is used as a primary storage device, and the others are used as replica storage devices. However, with this method, the primary and replica cannot be interchanged unless the primary storage device fails.

  As described above, since data can be written only to the primary storage device from the client device 20, for example, when the client device 20 moves to a location far away from the primary storage device, Even if there is a data storage device 30 (replica storage device) in the vicinity, writing could not be performed using the replica storage device as the primary storage device. In such a case, when a large amount of data is written, it may take a long time.

  On the other hand, in the present embodiment, by using a primary designator, it is possible to change a replica storage device to a primary storage device without transferring data or the like. In the first description below, an example of mapping in units of files is shown as an example. The file number i is used as file identification information. A file to be stored is called a target file.

  It is assumed that O = {o1, o2, o3} is obtained as an ordered set of the data storage device 30 that stores data of the target file by using a hash function for the file number i and the like. In the present embodiment, the ordered set is stored in the table information storage unit 13 of the data storage control device 10 both when calculated by the client device 20 and when calculated by the data storage control device 10. Shall.

  As shown in FIG. 5A, each element in {o1, o2, o3} represents a data storage device 30. It is assumed that which data storage device 30 corresponds to the identification information such as o1 and information such as an IP address necessary for accessing the data storage device 30 is stored in the data storage control device 10 in advance.

  In this example in which the ordered set is {o1, o2, o3}, the leading element is o1, so the data storage device 30 of o1 becomes the primary storage device. Therefore, in this state, the client device 20 acquires the ordered set or the like from the data storage control device 10 at the time of access, and when writing the data of the target file, the o1 data storage device 30 (primary storage device). ) Only. Note that the client device 20 recognizes the primary storage device by the ordered set {o1, o2, o3}, and the processing itself of writing only to the primary storage device is the client of the distributed file system in this embodiment. This is a function that the existing software has as an existing function.

  Thereafter, it is assumed that a primary specifier r (not zero) is given to the target file and stored in the primary specifier table of the table information storage unit 13 in association with the file number i.

  The data storage device order calculation unit 11 in the data storage control device 10 reads the primary specifier r of the target file (in this example, an integer 1) from the primary specifier table and also sets the target file order set {o1, o2 , O3} is read from the ordered set table. Then, the data storage device order calculation unit 11 rotates (shifts, rotates) the ordered set {o1, o2, o3} by r (1) times, and obtains the shifted ordered set O ′ = {o2, o3, o1}. This is associated with the identification information of the target file and stored in the ordered set table.

  Since the leading element in the ordered set {o2, o3, o1} after the shift is o2, as shown in FIG. 5B, the data storage device of o2 becomes the primary storage device. That is, as shown in FIG. 5A, the data storage device (o1) at the base A is initially primary, but the data storage device (o2) at the base B is calculated by performing the calculation using the primary specifier r. ) Became primary.

  In the above example, it has been described that when a specific primary designator r is given, the primary storage device is changed to a specific data storage device. Here, the method of determining the primary specifier r is not limited to a specific method. For example, as described below, the primary storage device becomes the primary storage device so that the data storage device closest to the client device 20 becomes the primary storage device. The specifier r can be determined.

  For example, in the example of FIG. 5, the client device 20 (meaning a client device that accesses the target file) at a certain point in time is near the base A, and the data storage device (o1) at the base A is the primary storage device. .

  Thereafter, when the client device 20 moves to a location near the base B and accesses the target file, an access request including identification information of the target file is transmitted to the data storage control device 10. Further, the access request includes identification information such as identification information of the client device 20, and the distance between the client device 20 and each data storage device can be determined based on this identification information.

  In the data storage control device 10 that has received the access request, the primary designator determination unit 12 determines that the data storage device (o2) at the base B is the closest to the client device 20 among the data storage devices that store the data of the target file. The primary designator r is determined so that the data storage device (o2) becomes the primary storage device. For example, in order set {o1, o2, o3}, in order for the data storage device (o2) to become the primary storage device, it is only necessary that o2 is first, so r is determined as 1, and this is used as the identification information of the target file. Store in the primary specifier table in association. Thereafter, as described above, O ′ = {o2, o3, o1} is calculated, and the primary storage device of the target file is determined according to O ′ = {o2, o3, o1}.

<About Placement Group>
As described above, the unit of data mapped to the data storage device is not limited to a file. For example, the concept of Ceph's placement group (PG) may be applied, and mapping to the data storage device may be performed in units of dividing the file data.

  The concept of PG will be described with reference to FIG. As shown in FIG. 6, the file data d is divided into a plurality of file blocks (d1, d2, d3 in the example of FIG. 6) according to the size and the like. Each divided file block is mapped to a virtual container called a placement group (PG). In the example of FIG. 6, d1, d2, and d3 are mapped to PG1, PG2, and PG3, but this is only an example, and the mapping to the container is actually determined by a hash algorithm or the like. Finally, each PG is mapped to the data storage device by a hash function or the like. This mapping method may be the same as the method for mapping to the data storage device in units of files described above. That is, the above-described mapping method in units of files is equivalent to the case where the number of divisions is 1 when the concept of a placement group is introduced.

  When the concept of a placement group is introduced, the primary specifier table shown in FIG. 4A is a table in which a primary specifier is associated with each file block. Also, the ordered set table shown in FIG. 4B is a table in which the ordered set is associated with each PG. Further, since the client device 20 needs to know the relationship between the PG, the file block, and the file, mapping information of “file-file block-PG” is necessary. Alternatively, the data storage control device may hold it.

  The method for changing the primary storage device using the primary designator is as described above. In the file unit method described above, “file” may be replaced with “file block”.

  For example, when the primary storage device and the replica storage device exist in geographically separated locations, there is a possibility that the first half of the file data is accessed quickly but the second half is delayed. On the other hand, as described above, the problem can be solved by setting the primary designator r for each file block so that the same base becomes the primary storage device.

(Process to determine replication route)
Next, the replication path determination process performed in the replication path determination unit 14 of the data storage control device 10 will be described.

  For example, if the ordered set O = {o1, o2, o3} is calculated, the primary storage device is o1, and the others are replica storage devices. As described above, data can be written from the client device 20 only to the primary storage device, and a copy (replica data) of data (primary data) stored in the primary storage device is stored in the primary storage device. As a base point, it is transferred to each replica storage device and stored in each replica storage device. With regard to a path for storing data, for example, a path such as “o1-> o2-> o3”, “o1-> o2, o1-> o3”, and “o1-> o3-> o2” can be taken. . Such a route is called a replication route.

  Here, basically, the primary storage device that has received the data write request from the client device 20 can confirm the completion of data storage in all replica storage devices along with the data storage to itself, Thus, the client device 20 determines that the writing process has ended with this notification, and can proceed to the next process.

  Therefore, it is necessary to store the replica data in the replica storage device quickly. However, data transfer between devices may take a long time depending on the network status (bandwidth, etc.) between the devices. This problem becomes significant especially when the amount of data to be transferred is large.

  As will be described later, this embodiment employs a procedure different from the above, and notifies the client device 20 of the completion of storage before confirming the completion of data storage in the replica storage device. Even in such a case, it is necessary to store the replica data in the replica storage device quickly from the viewpoint of maintaining data consistency.

  Therefore, in the present embodiment, the replication route is determined in consideration of the network status and the like. An example of the determination will be described with reference to FIG. As shown in FIG. 7A, a primary storage device exists at point A (geographic position), and replica storage devices exist at points B and C. Assume that the network status (capacity in the example of FIG. 7) between the devices (points) is as shown in the figure. Here, it is assumed that replica data of size 10 that is assumed to be able to be quickly transferred in the network of the band 10 is transferred.

  At this time, data transfer from the primary storage device at the point A to the replica storage device at the point B can be performed quickly, but it is very difficult to transfer data from the primary storage device at the point A to the replica storage device at the point C. It takes time. On the other hand, data transfer from the replica storage device at point B to the replica storage device at point C can be performed quickly. Therefore, as shown in FIG. 7B, the replication path determination unit 14 determines a path A-> B-> C as the replication path.

  FIG. 8 shows another example. In the example shown in FIG. 8, the network bandwidth between the points is 10 in all cases, but A-> B and A-> C physically have only one line (band 10). In such a case, if replica data of size 10 is transmitted through the route of A-> B and A-> C, the result is that data of size 20 is transmitted by one line, and can be transmitted quickly. Can not. Therefore, in this case as well, as shown in FIG. 8B, the replication path determination unit 14 determines the path A-> B-> C as the replication path.

  In addition, since it is desirable to quickly receive a data storage completion notification from the replica storage device, for example, the ability to transfer replica data to a replica storage device closer to the distance from the primary storage device or to store data at high speed It is also possible to determine the route in consideration of the criterion of transferring replica data to a replica storage device having

  The route determination information storage unit 15 includes network information such as free bandwidth between devices, distance between points, physical line information, capacity / data storage speed / reliability / load status of each data storage device, and the like. The replication route determination unit 14 calculates the optimum route based on the information read from the route determination information storage unit 15.

(System operation sequence example)
Next, an example of an operation sequence of the system in the present embodiment will be described with reference to FIG. In the following example, it is assumed that there is already an ordered set of files (target files) to be accessed at a time point before step 101.

  When the client device 20 accesses (writes, etc.) the target file, the client device 20 transmits a file access request including the target file identification information to the data storage control device 10 (step 101). The primary designator determination unit 12 in the data storage control device 10 determines the distance between each data storage device included in the ordered set and the client device 20, determines that the closest data storage device is the primary, and the data The primary designator is determined so that the storage device is at the head of the ordered set (step 102).

  Next, the data storage device order calculation unit 11 shifts the ordered set from the existing ordered set for the target file and the primary specifier determined in step 102 to obtain a new ordered set (step 103). ). Thereby, the primary storage device is determined. In this example, the data storage control device 10 notifies the data storage device that has become the primary storage device for the target file (step 104).

  The data storage device 10 returns a response to the client device 20 (step 105). The response includes, for example, identification information of the primary storage device, identification information of the replica storage device, and the like.

  Thereafter, the client device 20 writes data to a file (step 106). Here, data is written to the primary storage device by the function of the distributed file system.

  Here, the primary storage device according to the present embodiment notifies the storage completion notification from the replica storage device before transferring the data copy to the replica storage device or after transferring the data copy to the replica storage device ( Before receiving the (ACK), the storage completion notification (ACK) is transmitted to the client device 20 when the storage of the data of the primary storage device itself is completed. In the example of FIG. 9, a storage completion notification (ACK) is transmitted to the client device 20 before transferring the data copy from the primary storage device to the replica storage device (step 107).

  When the distance between the replica storage device and the primary storage device is long, when the storage completion notification (ACK) is transmitted to the client device 20 after waiting for the storage completion notification (ACK) from the replica storage device, the storage to the client device 20 is performed. There is a possibility that the completion notification (ACK) transmission is delayed and the shift to the next processing on the client device 20 side may be delayed, but this embodiment solves this problem.

  In addition to the processing described above, the storage completion notification (ACK) may be transmitted to the client device 20 after receiving the storage completion notification (ACK) only from the replica storage device that is close to the primary storage device.

  In step 108 of FIG. 9, the primary storage device requests the data storage control device 10 for path information for transferring the replicated data. Upon receiving the request, the replication path determination unit 14 of the data storage control apparatus 10 determines the path by the method described above (step 109), and notifies the primary storage apparatus of the path information (step 110). Then, the copied data is transferred and stored along this path (steps 111 to 114).

  In the above example, the data storage control device 10 determines the route. However, the primary storage device itself acquires information necessary for route determination such as network information from the data storage control device 10, and the route determination processing is performed. It is good also as performing.

(Summary of the embodiment, effects, etc.)
As described above, according to the present embodiment, a data storage control device in a data storage system for storing data in a plurality of data storage devices, which stores an ordered set of the plurality of data storage devices. Means for changing the order of elements in the ordered set by using designation information for designating a specific data storage device among the plurality of data storage devices, and the data storage system The data storage control device is characterized in that a data storage device corresponding to a predetermined element in the ordered set whose order is changed by the order calculation means is used as the specific data storage device.

  The predetermined element in the ordered set is, for example, the first element in the ordered set. The designation information is an integer, and the order calculation means changes the order by shifting the order set stored in the storage means by the number of times of the integer.

  In the data storage system, for example, the data is written to the specific data storage device among the plurality of data storage devices, and the duplicate data storage device which is a data storage device other than the specific data storage device is used. Replicated data of the data is stored.

  The data storage control device uses the specific data storage device as a base point, a replication path for storing the replicated data in each replicated data storage device, and information on a network connecting the points where the respective data storage devices are installed A duplication path determination means for determining based on the above may be provided.

  In addition, the data storage control device may include the designation information so that a data storage device located closest to a client device that accesses the data among the plurality of data storage devices becomes the specific data storage device. There may be provided designation information determining means for determining.

  The data storage system is, for example, a distributed file system, and the data is file data or data obtained by dividing file data.

  As described above, in the conventional technology, detailed specification such as in which data storage device the primary data is placed cannot be performed. In addition, the relationship between primary data and replica data cannot be reversed.

  On the other hand, in the present embodiment, a specific data storage device can be designated as the primary. Thereby, for example, when a replica already exists in the vicinity from the client, it can be used as primary data. As a result, large-scale data of several terabytes can be moved in a pseudo high speed.

  The present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the scope of the claims.

DESCRIPTION OF SYMBOLS 10 Data storage control apparatus 20 Client apparatus 30 Data storage apparatus 40 Network 11 Data storage apparatus order calculation part 12 Primary specifier determination part 13 Table information storage part 14 Duplication path | route determination part 15 Path | route determination information storage part 16 Communication processing part 31 Data Storage unit 32 Distributed file system processing unit 33 Replication control unit

Claims (9)

A data storage control device in a data storage system for storing data in a plurality of data storage devices,
Storage means for storing an ordered set of the plurality of data storage devices;
Order calculating means for changing the order of elements in the ordered set using designation information for designating a specific data storage device of the plurality of data storage devices;
In the data storage system, a data storage device corresponding to a predetermined element in the ordered set whose order is changed by the order calculation means is used as the specific data storage device.   The data storage control device according to claim 1, wherein the predetermined element in the ordered set is the first element in the ordered set. The specification information is an integer, and the order calculation means changes the order by shifting the order set stored in the storage means by the number of times of the integer. The data storage control device according to 1 or 2. In the data storage system, the data is written to the specific data storage device of the plurality of data storage devices, and the data storage device other than the specific data storage device stores the data The data storage control device according to any one of claims 1 to 3, wherein duplicate data is stored. A replication path for determining a replication path for storing replicated data in each replicated data storage apparatus based on the information on the network connecting the points where the respective data storage apparatuses are installed, based on the specific data storage apparatus The data storage control device according to claim 4, further comprising a determination unit. A designation information determining unit configured to determine the designation information so that a data storage device located closest to a client device that accesses the data among the plurality of data storage devices becomes the specific data storage device; 6. The data storage control device according to claim 1, wherein the data storage control device is a data storage control device. The data storage system according to any one of claims 1 to 6, wherein the data storage system is a distributed file system, and the data is file data or data obtained by dividing file data. Control device. A data storage control method executed by a data storage control device in a data storage system for storing data in a plurality of data storage devices,
The data storage control device includes storage means for storing an ordered set of the plurality of data storage devices, and the data storage control method includes:
Using designation information for designating a specific data storage device among the plurality of data storage devices, comprising an order calculation step of changing the order of elements in the ordered set;
In the data storage system, a data storage device corresponding to a predetermined element in the ordered set whose order is changed by the order calculation step is used as the specific data storage device. The program for functioning a computer as each means in the data storage control apparatus of any one of Claims 1 thru | or 7.

Images