CN106657356A - Data writing method and device for cloud storage system, and cloud storage system - Google Patents
Data writing method and device for cloud storage system, and cloud storage system Download PDFInfo
- Publication number
- CN106657356A CN106657356A CN201611248364.7A CN201611248364A CN106657356A CN 106657356 A CN106657356 A CN 106657356A CN 201611248364 A CN201611248364 A CN 201611248364A CN 106657356 A CN106657356 A CN 106657356A
- Authority
- CN
- China
- Prior art keywords
- data
- written
- memory
- cloud storage
- storage system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data writing method and device for a cloud storage system. A storage node of the cloud storage system at least includes high-speed storage equipment and high-capacity storage equipment. The method comprises the following steps of: receiving a writing request sent by a client side node, and writing to-be-written data in the high-speed storage equipment after an erasure code is processed at the client side node; after completing writing, sending completion response to the client side node; and, receiving a persistent request sent by the client side node, and persistently writing the to-be-written data in the high-capacity storage equipment. Writing of data is completed through two steps of writing the high-speed storage equipment and the high-capacity storage equipment; the data reliability and consistency of the cloud storage system can be ensured; the writing operation delay can be reduced while the high storage capacity is provided; furthermore, processing of an erasure code is completed at the client side node; the pressure of a service side can be lightened; the network communication data size can be reduced; and furthermore, the invention further provides the cloud storage system having the above technological advantages.
Description
Technical field
The present invention relates to technical field of data storage, more particularly to a kind of method for writing data of cloud storage system, dress
Put and cloud storage system.
Background technology
As the rapid development of mobile Internet, Internet of Things etc., personal data, family data, business data etc. are all in
Existing explosive growth, the mass data epoch quietly arrive.Simultaneously the own Jing of these data is from simple relational data to non-relation
Type data conversion, such as picture, audio frequency, video, document, traditional own Jing of storage management mode cannot meet depositing for this kind of data
How storage demand, effectively store, manage, safeguarding this kind of non-relational data by more and more in the industry number institute emphasis is closed
Note.
Cloud storage is referred to by functions such as cluster application, grid or distributed file systems, a large amount of various in network
Different types of storage device gathers collaborative work by application software, common externally to provide data storage and Operational Visit
One system of function, it is ensured that the security of data, and save memory space.In simple terms, cloud storage is exactly by storage resources
A kind of emerging scheme for people's access is put on cloud.User can at any time, Anywhere, through any web-enabled
Device is connected on cloud and easily accesses data.Just as cloud form wide area network as internet, cloud storage to user come
Say, do not refer to some specific equipment, and refer to a set being made up of various storage devices and server
Body.User uses cloud storage, is not to use some storage device, but the one kind brought using whole cloud storage system
Data access service.
Cloud storage technology is developed rapidly so that cloud storage becomes a new Data Storage Solution.Nearly two
Year, domestic cloud storage technology also achieves vigorous growth, and each manufacturer releases one after another the cloud storage product of oneself, cloud storage neck
Domain presents scene of flourishing life.But on the whole, domestic cloud storage technology is also in developing stage.Existing memory node
During writing to big data, due to the restriction of the performance of mass-memory unit, client appearance is easily caused
There is the phenomenon for postponing, affecting storage performance in interim card, write operation.
The content of the invention
It is an object of the invention to provide a kind of method for writing data of cloud storage system, device and cloud storage system, with
Solving the problems, such as easily to occur in existing cloud stocking system Caton phenomenon, write operation, to postpone more, storage performance relatively low.
To solve above-mentioned technical problem, the present invention provides a kind of method for writing data of cloud storage system, the cloud storage
Including at least a high-speed processing apparatus and a mass-memory unit on the memory node of system, methods described includes：
The write request that client node sends is received, and the client node is carried out to be written after correcting and eleting codes process
Data are written to the high-speed processing apparatus；
After after having write, to the client node response is sent completely；
The persistence request that the client node sends is received, the data persistence to be written is written to described big
Capacity storage device.
Alternatively, it is described the data persistence to be written is written into the mass-memory unit after also wrap
Include：
Deletion is stored on the high-speed processing apparatus corresponding versioned data.
Alternatively, also include：
Each memory node detects the running status of other memory nodes by heartbeat mechanism；
When the state for detecting one or more memory nodes occurs abnormal, compare between each node of normal condition
The version number of data in the high-speed processing apparatus, using the minimum data of version as reference data, will be big in other nodes
Deleted in the data of the version number of the reference data.
Alternatively, the high-speed processing apparatus are solid state hard disc or nonvolatile memory；The mass-memory unit
For mechanical hard disk or disk array.
Present invention also offers a kind of data transfer apparatus of cloud storage system, on the memory node of the cloud storage system
Including at least a high-speed processing apparatus and a mass-memory unit, described device includes：
First writing module, for receiving the write request of client node transmission, and the client node is entangled
Delete the data to be written after code is processed and be written to the high-speed processing apparatus；
Respond module, after after having write, to the client node response is sent completely；
Second writing module, for receiving the persistence request that the client node sends, by the data to be written
Persistence is written to the mass-memory unit.
Present invention also offers a kind of cloud storage system, including：
The storage cluster of client node and multiple memory nodes composition；Each memory node in the storage cluster
It is upper to include at least a high-speed processing apparatus and a mass-memory unit；
Wherein, the client node is used for the write request of receive user, and data to be written are carried out with correcting and eleting codes process, and
Data is activation after correcting and eleting codes are processed is to specified memory node；
The memory node receives the write request that the client node sends, and the data to be written are written into institute
High-speed processing apparatus are stated, after the completion of write, to the client node response is sent completely；
The client node sends persistence request after response is completed described in receiving to the memory node；
The memory node receives the persistence request, and the data persistence to be written is written into the Large Copacity
Storage device.
Alternatively, the memory node is additionally operable to detect the running status of other memory nodes by heartbeat mechanism；Work as inspection
When the state for measuring one or more memory nodes occurs abnormal, compare the high speed between each node of normal condition and deposit
The version number of data in storage equipment, using the minimum data of version as reference data, will be more than the base value in other nodes
According to the data of version number deleted.
Alternatively, the client node generates initial data specifically for the data to be written are divided into fragment
Block；The redundant data in the original data block is calculated, the redundant data is extended, is encoded, and be stored in different
Position.
Alternatively, the memory node is additionally operable to for the data persistence to be written to be written to the massive store
After in equipment, deletion is stored on the high-speed processing apparatus corresponding versioned data.
Alternatively, the high-speed processing apparatus are solid state hard disc or nonvolatile memory；The mass-memory unit
For mechanical hard disk or disk array.
The method for writing data and device of cloud storage system provided by the present invention, on the memory node of cloud storage system
Including at least a high-speed processing apparatus and a mass-memory unit, please by receiving writing for client node transmission
Ask, and client node is carried out the data to be written after correcting and eleting codes process and be written to high-speed processing apparatus；After after having write,
Response is sent completely to client node；The persistence request that client node sends is received, data persistence is written into and is write
Enter to mass-memory unit.The present invention completes writing for data by writing high-speed processing apparatus, writing the step of mass-memory unit two
Enter, the data reliability and uniformity of cloud storage system can either be ensured, can reduce while big memory capacity is provided again
The delay of write operation.And the process of correcting and eleting codes is completed in client node, the pressure of service end can be mitigated, reduced network and lead to
The data volume of letter.Additionally, a kind of present invention also offers cloud storage system with above-mentioned technological merit.
Description of the drawings
For the clearer explanation embodiment of the present invention or the technical scheme of prior art, below will be to embodiment or existing
The accompanying drawing to be used needed for technology description is briefly described, it should be apparent that, drawings in the following description are only this
Some bright embodiments, for those of ordinary skill in the art, on the premise of not paying creative work, can be with root
Other accompanying drawings are obtained according to these accompanying drawings.
Fig. 1 is a kind of flow process of specific embodiment of the method for writing data of cloud storage system provided by the present invention
Figure；
Fig. 2 is the flow process of another kind of specific embodiment of the method for writing data of cloud storage system provided by the present invention
Figure；
Fig. 3 is correcting and eleting codes principle schematic；
Fig. 4 is the structured flowchart of the data transfer apparatus of cloud storage system provided in an embodiment of the present invention；
Fig. 5 is the schematic diagram of cloud storage system provided in an embodiment of the present invention；
Fig. 6 is the processing procedure schematic diagram of cloud storage system write operation provided in an embodiment of the present invention.
Specific embodiment
In order that those skilled in the art more fully understand the present invention program, with reference to the accompanying drawings and detailed description
The present invention is described in further detail.Obviously, described embodiment is only a part of embodiment of the invention, rather than
Whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art are not making creative work premise
Lower obtained every other embodiment, belongs to the scope of protection of the invention.
A kind of flow chart such as Fig. 1 of the specific embodiment of the method for writing data of cloud storage system provided by the present invention
It is shown, set including at least a high-speed processing apparatus and a massive store on the memory node of the cloud storage system
Standby, methods described includes：
Step S101：The write request that client node sends is received, and the client node is carried out into correcting and eleting codes process
Data to be written afterwards are written to the high-speed processing apparatus；
It should be noted that provided by the present invention kind of cloud storage system can be specifically included：One or more clients
Node, for receiving and processes the write request of user.
The storage cluster of multiple memory node compositions, for data storage.One is included at least wherein on each memory node
Individual high-speed processing apparatus and a mass-memory unit.
Client node carries out correcting and eleting codes coding and packet to the data that user submits to；For each packet, client
End node is encapsulated as write request and is sent to the memory node specified.Memory node receives the write request that client sends,
And be written into data and be written to high-speed processing apparatus can read, in the way of non-covered, versioned.
Step S102：After after having write, to the client node response is sent completely；
Step S103：The persistence request that the client node sends is received, the data persistence to be written is write
Enter to the mass-memory unit.
Client node receive all memory nodes complete response after, notify that user's write operation is completed, and to
Each memory node sends persistence request.Each memory node receives the persistence request of client node, and data persistence is write
Enter in mass-memory unit.
The embodiment of the present invention may be used also after the data persistence to be written is written into the mass-memory unit
To further include：Delete corresponding versioned data on high-speed processing apparatus.
The method for writing data of cloud storage system provided by the present invention, includes at least on the memory node of cloud storage system
One high-speed processing apparatus and a mass-memory unit, by receiving the write request that client node sends, and by visitor
Family end node carries out the data to be written after correcting and eleting codes process and is written to high-speed processing apparatus；After after having write, to client
Node is sent completely response；The persistence request that client node sends is received, data persistence is written into and is written to great Rong
Amount storage device.The present invention, can either by writing high-speed processing apparatus, writing the write that the step of mass-memory unit two completes data
Ensure the data reliability and uniformity of cloud storage system, prolonging for write operation can be reduced while big memory capacity is provided again
Late.And the process of correcting and eleting codes is completed in client node, the pressure of service end can be mitigated, reduce the data of network service
Amount.
The flow chart of another kind of specific embodiment of the method for writing data of cloud storage system provided by the present invention is such as
Shown in Fig. 2, the method includes：
Step S201：Client node carries out correcting and eleting codes coding and packet to the data that user submits to；For each data
Packet, client node is encapsulated as write request and is sent to the memory node specified.
Step S202：Each memory node receives the write request of client node, by data therein can read, non-cover
Lid, the mode of versioned are written to high-speed processing apparatus；After the completion of write, memory node to client node is sent completely sound
Should.
Step S203：Client node receive all memory nodes complete response after, notified user's write operation
Complete, and persistence request is sent to each memory node.
Step S204：Each memory node receives the persistence request of client node, and data persistence is written into great Rong
In amount storage device, and delete corresponding versioned data on high-speed processing apparatus.
Correcting and eleting codes (erasure coding, EC) are a kind of data guard methods, and data are divided into fragment by it, redundancy
Data block extension, coding, and store it in different positions, such as disk, memory node or other geographical position.Mesh
Before, a common use case of correcting and eleting codes is object-based cloud storage.
With reference to Fig. 3, the basic thought of correcting and eleting codes is to calculate the original data element of k blocks by certain coding, obtains m
Block check element.System is respectively stored in this k+m block numbers evidence on k+m memory node.When arbitrarily less than equal to m storage section
When point breaks down, system can go out original k block number evidences by the data recovery on remaining node.
On the basis of any of the above-described embodiment, the method for writing data of cloud storage system provided by the present invention can be with
Further include：Each memory node detects the running status of other memory nodes by heartbeat mechanism；When detect one or
When the state of multiple memory nodes occurs abnormal, number in the high-speed processing apparatus is compared between each node of normal condition
According to version number, using the minimum data of version as reference data, will be in other nodes more than the version number of the reference data
Data deleted.
Anomaly can be specially node failure or node is restarted, and be enabled to by above-mentioned detection and recovery operation
Data between all memory nodes all in identical version, so as to ensure that system in data uniformity.
Used as a kind of specific embodiment, high-speed processing apparatus can be specially solid state hard disc or nonvolatile memory；
Mass-memory unit can be specially mechanical hard disk or disk array.High-speed processing apparatus have high-performance, low latency etc. excellent
Point, therefore use as caching；And mass-memory unit can reach very big capacity and higher cost performance, thus be accordingly used in
Persistent storage data.
The data transfer apparatus of cloud storage system provided in an embodiment of the present invention are introduced below, cloud described below
The data transfer apparatus of storage system can be mutually to should refer to the method for writing data of above-described cloud storage system.
Fig. 4 is the structured flowchart of the data transfer apparatus of cloud storage system provided in an embodiment of the present invention, cloud storage system
Memory node on including at least a high-speed processing apparatus and a mass-memory unit, with reference to Fig. 4 cloud storage systems
Data transfer apparatus can include：
First writing module 100, for receiving the write request of client node transmission, and the client node is carried out
Data to be written after correcting and eleting codes process are written to the high-speed processing apparatus；
Respond module 200, after after having write, to the client node response is sent completely；
Second writing module 300, for receiving the persistence request that the client node sends, by the number to be written
The mass-memory unit is written to according to persistence.
The data transfer apparatus of the cloud storage system of the present embodiment are used to realize the data write of aforesaid cloud storage system
The data of the visible cloud storage system hereinbefore of specific embodiment in method, therefore the data transfer apparatus of cloud storage system
The embodiment part of wiring method, for example, the first writing module 100, respond module 200, the second writing module 300 is used respectively
Step S101 in the method for writing data for realizing above-mentioned cloud storage system, S102, S103, so, its specific embodiment can
With reference to the description of corresponding various pieces embodiment, will not be described here.
Client can specifically include correcting and eleting codes module and message processing module.
Wherein, correcting and eleting codes module is used for according to correcting and eleting codes algorithm, and the data that will be write are divided into K block initial data, and
Calculated based on K blocks initial data and obtain M block redundant datas.
The communication that message processing module is used between client and memory node.Client is first by K blocks initial data and M
Block redundant data is separately sent to (K+M) individual memory node, and then wait each memory node completes response；Receiving all (K
+ M) individual memory node complete response after, client confirm write operation complete, then to each memory node send persistence please
Ask.
First writing module of memory node writes data into high speed storing and sets after the write request for receiving client
Standby, respond module is sent completely response after write is completed to client；Second writing module is receiving the lasting of client
After changing request, mass-memory unit is write data into, and it is deleted from high-speed processing apparatus.
Wherein, the first writing module is specifically for data are written at a high speed can read, in the form of non-covered, versioned
Storage device；After the completion of persistence write, cache module will delete corresponding data.
Second writing module notifies that cache module deletes corresponding number specifically for writing data into large capacity equipment
According to.
Further, the data transfer apparatus of cloud storage system provided by the present invention can also include：State-detection and
Recovery module：Each memory node detects the running status of other memory nodes by heartbeat mechanism.When one or more nodes
State change (node failure, node are restarted) when, mutually compare between each node of normal condition and deposit at a high speed
The version number of data in storage equipment, the data using that minimum node of version number are used as reference data.Other nodes are by deleting
Except the data more than the version number, data recovery is made to the state consistent with benchmark.
Additionally, present invention also offers a kind of cloud storage system, as shown in figure 5, the system includes：
The storage cluster of client node and multiple memory nodes composition；Each memory node in the storage cluster
It is upper to include at least a high-speed processing apparatus and a mass-memory unit；
Wherein, the client node is used for the write request of receive user, and data to be written are carried out with correcting and eleting codes process, and
Data is activation after correcting and eleting codes are processed is to specified memory node；
The memory node receives the write request that the client node sends, and the data to be written are written into institute
High-speed processing apparatus are stated, after the completion of write, to the client node response is sent completely；
The client node sends persistence request after response is completed described in receiving to the memory node；
The memory node receives the persistence request, and the data persistence to be written is written into the Large Copacity
Storage device.
Wherein, the memory node is additionally operable to detect the running status of other memory nodes by heartbeat mechanism；Work as detection
When occurring abnormal to the state of one or more memory nodes, the high speed storing is compared between each node of normal condition
The version number of data in equipment, using the minimum data of version as reference data, will be more than the reference data in other nodes
The data of version number deleted.
Used as a kind of specific embodiment, the client node by the data to be written specifically for being divided into piece
Section, generates original data block；The redundant data in the original data block is calculated, the redundant data is extended, is encoded,
And it is stored in different positions.
The memory node is additionally operable to that the data persistence to be written is being written in the mass-memory unit
Afterwards, delete and be stored on the high-speed processing apparatus corresponding versioned data.
Used as a kind of specific embodiment, the high-speed processing apparatus are solid state hard disc or nonvolatile memory；It is described
Mass-memory unit is mechanical hard disk or disk array.
With reference to the processing procedure schematic diagram of Fig. 6 write operations, the process of data write is further elaborated on below.
The process is specifically included：
Client node carries out correcting and eleting codes coding and packet to the data that user submits to；For each packet, client
End node is encapsulated as write request and is sent to the memory node specified.
Each memory node receives the write request of client node, by data therein can read, non-covered, versioned
Mode is written to high-speed processing apparatus；After the completion of write, memory node to client node is sent completely response.
Client node receive all memory nodes complete response after, notify that user's write operation is completed, and to
Each memory node sends persistence request.
Each memory node receives the persistence request of client node, and data persistence is written into mass-memory unit
In, and delete corresponding versioned data on high-speed processing apparatus.
The present invention can either be protected by writing high-speed processing apparatus, writing the write that the step of mass-memory unit two completes data
The data reliability and uniformity of card cloud storage system, can reduce prolonging for write operation while big memory capacity is provided again
Late.And the process of correcting and eleting codes is completed in client node, the pressure of service end can be mitigated, reduce the data of network service
Amount.
Each embodiment is described by the way of progressive in this specification, and what each embodiment was stressed is and other
The difference of embodiment, between each embodiment same or similar part mutually referring to.For dress disclosed in embodiment
For putting, because it is corresponded to the method disclosed in Example, so description is fairly simple, related part is referring to method part
Illustrate.
Professional further appreciates that, with reference to the unit of each example of the embodiments described herein description
And algorithm steps, can with electronic hardware, computer software or the two be implemented in combination in, in order to clearly demonstrate hardware and
The interchangeability of software, according to function has generally described the composition and step of each example in the above description.These
Function is performed with hardware or software mode actually, depending on the application-specific and design constraint of technical scheme.Specialty
Technical staff can use different methods to realize described function to each specific application, but this realization should not
Think beyond the scope of this invention.
The step of method described with reference to the embodiments described herein or algorithm, directly can be held with hardware, processor
Capable software module, or the combination of the two is implementing.Software module can be placed in random access memory (RAM), internal memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In field in known any other form of storage medium.
Method for writing data above to cloud storage system provided by the present invention, device and cloud storage system are carried out
It is discussed in detail.Specific case used herein is set forth to the principle and embodiment of the present invention, above example
Illustrate that being only intended to help understands the method for the present invention and its core concept.It should be pointed out that for the common skill of the art
For art personnel, under the premise without departing from the principles of the invention, some improvement and modification can also be carried out to the present invention, these change
Enter and modify to also fall into the protection domain of the claims in the present invention.
Claims (10)
1. a kind of method for writing data of cloud storage system, it is characterised in that on the memory node of the cloud storage system at least
Comprising a high-speed processing apparatus and a mass-memory unit, methods described includes：
The write request that client node sends is received, and the client node is carried out the data to be written after correcting and eleting codes process
It is written to the high-speed processing apparatus；
After after having write, to the client node response is sent completely；
The persistence request that the client node sends is received, the data persistence to be written is written into the Large Copacity
Storage device.
2. the method for writing data of cloud storage system as claimed in claim 1, it is characterised in that it is described will be described to be written
Data persistence is written to after the mass-memory unit also to be included：
Deletion is stored on the high-speed processing apparatus corresponding versioned data.
3. the method for writing data of cloud storage system as claimed in claim 1 or 2, it is characterised in that also include：
Each memory node detects the running status of other memory nodes by heartbeat mechanism；
When the state for detecting one or more memory nodes occurs abnormal, between each node of normal condition institute is compared
The version number of data in high-speed processing apparatus is stated, using the minimum data of version as reference data, institute will be more than in other nodes
The data for stating the version number of reference data are deleted.
4. the method for writing data of cloud storage system as claimed in claim 3, it is characterised in that the high-speed processing apparatus are
Solid state hard disc or nonvolatile memory；The mass-memory unit is mechanical hard disk or disk array.
5. a kind of data transfer apparatus of cloud storage system, it is characterised in that on the memory node of the cloud storage system at least
Comprising a high-speed processing apparatus and a mass-memory unit, described device includes：
First writing module, for receiving the write request of client node transmission, and carries out correcting and eleting codes by the client node
Data to be written after process are written to the high-speed processing apparatus；
Respond module, after after having write, to the client node response is sent completely；
Second writing module, for receiving the persistence request that the client node sends, by the lasting data to be written
Change is written to the mass-memory unit.
6. a kind of cloud storage system, it is characterised in that include：
The storage cluster of client node and multiple memory nodes composition；Each memory node in the storage cluster is up to
Less comprising a high-speed processing apparatus and a mass-memory unit；
Wherein, the client node is used for the write request of receive user, and data to be written are carried out with correcting and eleting codes process, and will entangle
The data is activation after code is processed is deleted to specified memory node；
The memory node receives the write request that the client node sends, and the data to be written are written into the height
Fast storage device, after the completion of write, to the client node response is sent completely；
The client node sends persistence request after response is completed described in receiving to the memory node；
The memory node receives the persistence request, and the data persistence to be written is written into the massive store
Equipment.
7. cloud storage system as claimed in claim 6, it is characterised in that the memory node is additionally operable to be examined by heartbeat mechanism
Survey the running status of other memory nodes；When the state for detecting one or more memory nodes occurs abnormal, in normal
Compare the version number of data in the high-speed processing apparatus between each node of state, using the minimum data of version as base value
According to the data of version number in other nodes more than the reference data are deleted.
8. cloud storage system as claimed in claims 6 or 7, it is characterised in that the client node is specifically for will be described
Data to be written are divided into fragment, generate original data block；The redundant data in the original data block is calculated, by the redundancy
Data are extended, encode, and are stored in different positions.
9. cloud storage system as claimed in claim 8, it is characterised in that the memory node is additionally operable to will be described to be written
After data persistence is written in the mass-memory unit, deletion is stored on the high-speed processing apparatus corresponding version
This change data.
10. cloud storage system as claimed in claim 9, it is characterised in that the high-speed processing apparatus are solid state hard disc or non-
Volatile memory；The mass-memory unit is mechanical hard disk or disk array.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611248364.7A CN106657356A (en) | 2016-12-29 | 2016-12-29 | Data writing method and device for cloud storage system, and cloud storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611248364.7A CN106657356A (en) | 2016-12-29 | 2016-12-29 | Data writing method and device for cloud storage system, and cloud storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106657356A true CN106657356A (en) | 2017-05-10 |
Family
ID=58836335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611248364.7A Pending CN106657356A (en) | 2016-12-29 | 2016-12-29 | Data writing method and device for cloud storage system, and cloud storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106657356A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107357536A (en) * | 2017-07-28 | 2017-11-17 | 郑州云海信息技术有限公司 | Distributed memory system data modification write method and system |
| CN108595109A (en) * | 2017-12-29 | 2018-09-28 | 郑州云海信息技术有限公司 | A method of the multi-client based on Ceph distributed memory systems is write |
| CN109144770A (en) * | 2018-08-02 | 2019-01-04 | 联想(北京)有限公司 | The data processing method and device of storage system |
| CN109213420A (en) * | 2017-06-29 | 2019-01-15 | 杭州海康威视数字技术股份有限公司 | Date storage method, apparatus and system |
| CN109558079A (en) * | 2018-11-19 | 2019-04-02 | 郑州云海信息技术有限公司 | A kind of method and device of distributed memory system processing operation request |
| CN109634526A (en) * | 2018-12-11 | 2019-04-16 | 浪潮(北京)电子信息产业有限公司 | A kind of data manipulation method and relevant apparatus based on object storage |
| WO2019170004A1 (en) * | 2018-03-09 | 2019-09-12 | 杭州海康威视系统技术有限公司 | Data storage system, method and device |
| CN111198783A (en) * | 2018-11-16 | 2020-05-26 | 阿里巴巴集团控股有限公司 | Data access method, device, system, equipment and storage medium |
| CN113467719A (en) * | 2021-06-25 | 2021-10-01 | 阿里巴巴新加坡控股有限公司 | Data writing method and device |
| CN114928621A (en) * | 2022-02-15 | 2022-08-19 | 杭州瑞网广通信息技术有限公司 | Data writing method and device of cloud storage system |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1496510A (en) * | 2001-03-07 | 2004-05-12 | ���Ĺ��ʹ�˾ | Disk writes in distributed shared disk system |
| EP1638003A2 (en) * | 2004-09-08 | 2006-03-22 | Fujitsu Limited | Apparatus and method for retrieving data from a data storage system |
| US20080147972A1 (en) * | 2005-10-26 | 2008-06-19 | International Business Machines Corporation | System, method and program for managing storage |
| CN103154909A (en) * | 2010-10-06 | 2013-06-12 | 马维尔国际贸易有限公司 | Distributed cache coherency protocol |
| CN103503414A (en) * | 2012-12-31 | 2014-01-08 | 华为技术有限公司 | Computing storage integration cluster system |
| CN104317926A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Persistent data storage method, persistent data query method, persistent data storage device and persistent data query system |
| CN105589664A (en) * | 2015-12-29 | 2016-05-18 | 四川中电启明星信息技术有限公司 | Virtual storage high-speed transmission method |
| CN105791353A (en) * | 2014-12-23 | 2016-07-20 | 深圳市腾讯计算机系统有限公司 | Distributed data storage method and system based on erasure code |
-
2016
- 2016-12-29 CN CN201611248364.7A patent/CN106657356A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1496510A (en) * | 2001-03-07 | 2004-05-12 | ���Ĺ��ʹ�˾ | Disk writes in distributed shared disk system |
| EP1638003A2 (en) * | 2004-09-08 | 2006-03-22 | Fujitsu Limited | Apparatus and method for retrieving data from a data storage system |
| US20080147972A1 (en) * | 2005-10-26 | 2008-06-19 | International Business Machines Corporation | System, method and program for managing storage |
| CN103154909A (en) * | 2010-10-06 | 2013-06-12 | 马维尔国际贸易有限公司 | Distributed cache coherency protocol |
| CN103503414A (en) * | 2012-12-31 | 2014-01-08 | 华为技术有限公司 | Computing storage integration cluster system |
| CN104317926A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Persistent data storage method, persistent data query method, persistent data storage device and persistent data query system |
| CN105791353A (en) * | 2014-12-23 | 2016-07-20 | 深圳市腾讯计算机系统有限公司 | Distributed data storage method and system based on erasure code |
| CN105589664A (en) * | 2015-12-29 | 2016-05-18 | 四川中电启明星信息技术有限公司 | Virtual storage high-speed transmission method |
Non-Patent Citations (1)
| Title |
|---|
| 孔晨燕 等: "HDFS平台下基于纠删码的一种数据放置策略", 《浙江师范大学学报（自然科学版）》 * |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109213420A (en) * | 2017-06-29 | 2019-01-15 | 杭州海康威视数字技术股份有限公司 | Date storage method, apparatus and system |
| CN107357536A (en) * | 2017-07-28 | 2017-11-17 | 郑州云海信息技术有限公司 | Distributed memory system data modification write method and system |
| CN107357536B (en) * | 2017-07-28 | 2020-07-07 | 郑州云海信息技术有限公司 | Data modification and writing method and system for distributed storage system |
| CN108595109A (en) * | 2017-12-29 | 2018-09-28 | 郑州云海信息技术有限公司 | A method of the multi-client based on Ceph distributed memory systems is write |
| WO2019170004A1 (en) * | 2018-03-09 | 2019-09-12 | 杭州海康威视系统技术有限公司 | Data storage system, method and device |
| CN109144770A (en) * | 2018-08-02 | 2019-01-04 | 联想(北京)有限公司 | The data processing method and device of storage system |
| CN109144770B (en) * | 2018-08-02 | 2020-11-20 | 联想(北京)有限公司 | Data processing method and device of storage system |
| CN111198783A (en) * | 2018-11-16 | 2020-05-26 | 阿里巴巴集团控股有限公司 | Data access method, device, system, equipment and storage medium |
| CN109558079A (en) * | 2018-11-19 | 2019-04-02 | 郑州云海信息技术有限公司 | A kind of method and device of distributed memory system processing operation request |
| CN109634526A (en) * | 2018-12-11 | 2019-04-16 | 浪潮(北京)电子信息产业有限公司 | A kind of data manipulation method and relevant apparatus based on object storage |
| CN109634526B (en) * | 2018-12-11 | 2022-04-22 | 浪潮(北京)电子信息产业有限公司 | Data operation method based on object storage and related device |
| CN113467719A (en) * | 2021-06-25 | 2021-10-01 | 阿里巴巴新加坡控股有限公司 | Data writing method and device |
| CN113467719B (en) * | 2021-06-25 | 2024-03-29 | 阿里巴巴创新公司 | Data writing method and device |
| CN114928621A (en) * | 2022-02-15 | 2022-08-19 | 杭州瑞网广通信息技术有限公司 | Data writing method and device of cloud storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106657356A (en) | Data writing method and device for cloud storage system, and cloud storage system | |
| CN104239493B (en) | cross-cluster data migration method and system | |
| WO2017050014A1 (en) | Data storage processing method and device | |
| US20170075741A1 (en) | Prioritizing Data Reconstruction in Distributed Storage Systems | |
| US8621143B2 (en) | Elastic data techniques for managing cache storage using RAM and flash-based memory | |
| WO2016086819A1 (en) | Method and apparatus for writing data into shingled magnetic record smr hard disk | |
| WO2019001521A1 (en) | Data storage method, storage device, client and system | |
| EP3037988A1 (en) | Configuration method and device for hash database | |
| CN111400083B (en) | Data storage method and system and storage medium | |
| CN103516549B (en) | A kind of file system metadata log mechanism based on shared object storage | |
| US10838825B2 (en) | Implementing snapshot sets for consistency groups of storage volumes | |
| US11947842B2 (en) | Method for writing data in append mode, device and storage medium | |
| CN107329708A (en) | A kind of distributed memory system realizes data cached method and system | |
| CN109582213A (en) | Data reconstruction method and device, data-storage system | |
| CN111177143B (en) | Key value data storage method and device, storage medium and electronic equipment | |
| CN110196818A (en) | Data cached method, buffer memory device and storage system | |
| CN106445409A (en) | Distributed block storage data writing method and device | |
| CN110147203A (en) | A kind of file management method, device, electronic equipment and storage medium | |
| CN109918352B (en) | Memory system and method of storing data | |
| CN107797770A (en) | A kind of synchronous method and device of Disk State information | |
| CN107301021B (en) | Method and device for accelerating LUN (logical Unit number) by utilizing SSD (solid State disk) cache | |
| US10592138B1 (en) | Avoiding storage device overlap in raid extent sub group and keeping relationship balance on mapped raid system and method | |
| CN112748849B (en) | Method, apparatus and computer program product for storing data | |
| CN109086462A (en) | The management method of metadata in a kind of distributed file system | |
| CN103150268A (en) | Block-level data capture method in CDP (Continuous Data Protection) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170510 |