CN107070987A - Data capture method and system for distributed objects storage system - Google Patents
Data capture method and system for distributed objects storage system Download PDFInfo
- Publication number
- CN107070987A CN107070987A CN201710118848.8A CN201710118848A CN107070987A CN 107070987 A CN107070987 A CN 107070987A CN 201710118848 A CN201710118848 A CN 201710118848A CN 107070987 A CN107070987 A CN 107070987A
- Authority
- CN
- China
- Prior art keywords
- data
- metadata
- sublist
- request
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a kind of data capture method and system for distributed objects storage system.Wherein, this method includes：According to preset rules, send first to server and ask, wherein, first asks the data for the first metadata sublist in acquisition request distributed objects storage system；The data for the first metadata sublist that the reception server is returned；The second request of the data for asking other metadata sublists, and the data that the reception server is returned sequentially are sent, until server returns to end mark.The present invention is solved when obtaining mass data, the low technical problem of resource utilization.
Description
Technical field
The present invention relates to data processing field, in particular to a kind of data for distributed objects storage system
Acquisition methods and system.
Background technology
Distributed objects storage is the major programme for solving the problems, such as mass data storage at present, in order to meet data management
Demand, distributed objects storage system typically can all provide traversal function to user, and system can be obtained by the traversal function
The metadata of middle stored file or object.One typical implementation method is exactly：A large amount of metadata sublists are pre-established, this
A little metadata sublists are stored on the different nodes in system (such as storage device), in order to avoid the data volume stored on individual node
More than the upper limit of single storage device in system；Metadata is distributed in different metadata sublists according to specific rule
Go, wherein, what the data inside metadata sublist were ordered into, be unordered between metadata sublist.In this scenario, quilt completely
Perception of the client to metadata sublist is shielded, when entering row-column list List operations, system will be from all metadata sublists
Pulling data, and merger sequence is carried out, result will be pulled and be sent to client.In such scheme, because data volume is huge,
After pulling data, it is ranked up, rather than the data to client full dose are returned in single query, it is orderly at this
Under conditions of, last data that client can be inquired about according to last time come determine one mark, then allow system from that
Continue returned data after mark.
But, the above method is when full dose is traveled through, in order to store the metadata of mass data, it may be desirable to substantial amounts of member
Data sublist, to substantial amounts of metadata sublist inquire about and merger sequence is relatively time consuming, and may need to repeat substantial amounts of
Useless sequence.Such as, there is N number of metadata sublist in system, subscription client can only once receive Y bars record.When client is needed
When traveling through all records, process will be so：Client first sends a List operation to system and (limits and return to Y bars note
Record), system is the preceding Y bars record of each metadata sublist inquiry into N number of metadata sublist, to this N*Y bar record ordering,
Y bars record is returned, then remaining (N-1) * Y bars are recorded and abandoned, that is, wastes ratio and is up to (N-1)/N.Client can use this
The mode of sample travels through all records.Metadata is traveled through, is operation (such as active and standby file ratio commonly used during daily O＆M
To, file clean-up etc.), so such poorly efficient operation scheme produces considerable restraint to the Expansion of cluster.Traveled through
During operation, optionally, using the scheme of sequence, the global order for the data list that can ensure to return by the program, still,
The program greatly consumes internal bandwidth, internal memory or CPU, and the efficiency for obtaining data is low.
For above-mentioned when obtaining mass data, the problem of resource utilization is low not yet proposes effective solution party at present
Case.
The content of the invention
The embodiments of the invention provide a kind of data capture method and system for distributed objects storage system, so that
It is few to solve when obtaining mass data, the low technical problem of resource utilization.
There is provided a kind of data acquisition for distributed objects storage system for one side according to embodiments of the present invention
Method, including：According to preset rules, send first to server and ask, wherein, described first asks to be used for acquisition request distribution
The data of first metadata sublist in formula object storage system；Receive the first metadata sublist that the server is returned
Data；The second request of the data for asking other metadata sublists is sequentially sent, and receives the number that the server is returned
According to until server returns to end mark.
Further, the preset rules include：Metadata sublist is not included in first sent to server is asked
During numbering, then it represents that first request needs to carry out across sublist sequence to metadata sublist data；When what is sent to server
When including metadata sublist numbering in the first request, then it represents that first request to metadata sublist data without carrying out across son
List sorting.
Further, the second request of the data for asking other metadata sublists is sequentially sent, and receives the clothes
The data that business device is returned, include until server returns to end mark：P second request is sent to the server, wherein,
The P is the natural number more than 1；Receive the number for the metadata sublist that the server is returned based on the P second request
According to；After the data of the metadata sublist are performed with scheduled operation, sending (P+1) individual second to the server please
Ask, until getting the data in whole metadata sublists.
Further, it is described to send P second request to the server, wherein, the P is the natural number more than 1；
Receiving the data for the metadata sublist that the server is returned based on the P second request also includes：To the server
The second request for obtaining data in the P metadata sublist is sent, without the sublist in the server returned data
End mark when, the above one request obtain data in last data as start mark, continue to service end request should
The data of metadata sublist, the end mark until returning to the metadata sublist in the data of acquisition.Further, the data
Acquisition methods apply to the acquisition of full dose data.
Further, described according to preset rules, before sending the first request to server, in addition to：To server
Ask the sublist quantity of the metadata.
Another aspect according to embodiments of the present invention, additionally provides a kind of data for distributed objects storage system and obtains
Method is taken, including：The first request that client is sent according to preset rules is received, wherein, described first asks to obtain for request
Take the data of the first metadata sublist in distributed objects storage system；The first metadata sublist is returned to the client
Data；Receive the second request for being used to ask the data of other metadata sublists that the client is sequentially sent；To institute
State after the data that client returns to other metadata sublists, return to end mark.
Further, receive the data for asking other metadata sublists that the client is sequentially sent second please
After asking, including：Obtain the start mark carried in second request；In the indicated metadata sublist of the described second request
In, the data since the metadata sublist is obtained the corresponding next data of the start mark.
There is provided a kind of data acquisition for distributed objects storage system for another aspect according to embodiments of the present invention
System, the system includes：Client；Server；Wherein, the client sends first according to preset rules to the server
Request, wherein, described first asks the data for the first metadata sublist in acquisition request distributed objects storage system；
In the case that the server receives first request, the number of the first metadata sublist is returned to the client
According to；The client receives the data for the first metadata sublist that the server is returned, and sequentially sends for asking it
Second request of the data of his metadata sublist, and the data that the server is returned are received, terminate mark until server is returned
Note.
Further, after the data of the first metadata sublist are returned to the client, the server is also
For：Receive second request；The server is returned after the data of other metadata sublists are returned to the client
Return the end mark.
In embodiments of the present invention, it can send first according to preset rules to server and ask, wherein, the first request
For the data of the first metadata sublist in acquisition request distributed objects storage system, and first yuan that the reception server is returned
The data of data sublist, then, client sequentially send the second request of the data for asking other metadata sublists, and connect
The data that server is returned are received, until server returns to end mark.The embodiment of the present invention, according to preset rules, to server
Send the request of data for obtaining metadata sublist, it is to avoid in across the sublist sequence of service end, so as to obtain multiple metadata sublists
Data, improve data acquisition efficiency and resource utilization, solve when obtaining mass data, the low technology of resource utilization is asked
Topic.
Brief description of the drawings
Accompanying drawing described herein is used for providing a further understanding of the present invention, constitutes the part of the application, this hair
Bright schematic description and description is used to explain the present invention, does not constitute inappropriate limitation of the present invention.In the accompanying drawings：
Fig. 1 is a kind of flow of data capture method for distributed objects storage system according to embodiments of the present invention
Figure one；
Fig. 2 is the stream of the data capture method according to embodiments of the present invention optionally with distributed objects storage system
Journey figure two；
Fig. 3 is a kind of data acquisition side of client for distributed objects storage system according to embodiments of the present invention
The flow chart of method；
Fig. 4 is a kind of data of service end optionally with distributed objects storage system according to embodiments of the present invention
The flow chart of acquisition methods；
Fig. 5 is according to embodiments of the present invention a kind of optionally with the data-acquisition system of distributed objects storage system
Schematic diagram.
Embodiment
In order that those skilled in the art more fully understand the present invention program, below in conjunction with the embodiment of the present invention
Accompanying drawing, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is only
The embodiment of a part of the invention, rather than whole embodiments.Based on the embodiment in the present invention, ordinary skill people
The every other embodiment that member is obtained under the premise of creative work is not made, should all belong to the model that the present invention is protected
Enclose.
It should be noted that term " first " in description and claims of this specification and above-mentioned accompanying drawing, "
Two " etc. be for distinguishing similar object, without for describing specific order or precedence.It should be appreciated that so using
Data can exchange in the appropriate case, so as to embodiments of the invention described herein can with except illustrating herein or
Order beyond those of description is implemented.In addition, term " comprising " and " having " and their any deformation, it is intended that cover
Lid is non-exclusive to be included, for example, the process, method, system, product or the equipment that contain series of steps or unit are not necessarily limited to
Those steps or unit clearly listed, but may include not list clearly or for these processes, method, product
Or the intrinsic other steps of equipment or unit.
Part term in the present invention is made explanations：
Distributed memory system：It is to disperse data to be stored in many independent equipment.Using expansible system knot
Structure, storage load is shared using many storage servers, and storage information is positioned using location server.
Object：Attribute information comprising file data and correlation, can carry out self-management.
There is provided the implementation of a kind of method of data acquisition for distributed objects storage system according to embodiments of the present invention
Example, it is necessary to explanation, the step of the flow of accompanying drawing is illustrated can such as one group computer executable instructions calculating
Performed in machine system, and, although show logical order in flow charts, but in some cases, can with different from
Order herein performs shown or described step.
Fig. 1 is a kind of flow of data capture method for distributed objects storage system according to embodiments of the present invention
Figure one, as shown in figure 1, this method comprises the following steps：
Step S102, according to preset rules, sends first to server and asks, wherein, first asks to be used for acquisition request
The data of first metadata sublist in distributed objects storage system；
Step S104, the data for the first metadata sublist that the reception server is returned；
Step S106, sequentially sends the second request of the data for asking other metadata sublists, and the reception server
The data of return, until server returns to end mark.
By above-described embodiment, client can send first to server and ask according to preset rules, wherein, first
Ask the data for the first metadata sublist in acquisition request distributed objects storage system, and the reception server return the
The data of one metadata sublist, then, client sequentially send the second request of the data for asking other metadata sublists,
And the data that the reception server is returned, until server returns to end mark.According to embodiments of the present invention, can be according to according to pre-
If regular, the data for obtaining metadata sublist are sent to server, the data of multiple metadata sublists in server are sequentially obtained,
Client can effectively obtain the data of multiple metadata sublists, and resource utilization is improved, and solve when obtaining mass data, money
The low technical problem of source utilization rate.
The client of above-described embodiment can include intelligent terminal, and the intelligent terminal can include smart mobile phone, PC etc., visitor
Family end can be the device that user uses.
The server of above-described embodiment can be the place of data storage (e.g., file, metadata), in the server, can
So that substantial amounts of data storage to be got up, and by substantial amounts of data storage in each storage device of server, server can be with
Including multiple storage devices, each storage device, which can be corresponded to, can include one on a memory node, each memory node
Or multiple metadata sublists.
Optionally, can be stored with multiple data item in metadata sublist, and the data item in each metadata sublist can
To arrange order, and each metadata sublist can not sort., can be by metadata when being sorted to metadata sublist
Table is marked, wherein, the mark can include numbering, letter, symbol etc..It is preferred that, mark can be numbering.Wherein, it is first
The numbering of data sublist is derived from ordered set, such as natural number.
Optionally, according to preset rules, before sending the first request to server, in addition to：Should to service end request
The sublist quantity of metadata.In embodiments of the present invention, the sublist quantity of the metadata can be set to be N number of.N can be more than
Natural number equal to 1.Wherein, multiple metadata sublists can be included on each memory node in server.Optionally, in step
Suddenly in the technical scheme that S102 is provided, according to preset rules, send first to server and ask, wherein, first asks to be used to ask
Seek the data for obtaining the first metadata sublist in distributed objects storage system.
Optionally, can be the first metadata in the above-mentioned N number of metadata sublist of acquisition request for the first request
The data of table, the data can be the data item of above-described embodiment, in the data in needing processing server, can first send
Solicited message, with obtain the data that each storage device can be stored in manageable data, each server be it is limited,
The data of client process are also limited, when client sends the first request, carry and can locate in being asked first
The parameter of the quantity of the data of reason.
Wherein, preset rules can include：Numbered when not including metadata sublist in the first request sent to server
When, then it represents that the first request needs to carry out across sublist sequence to metadata sublist data；When the first request sent to server
In when including metadata sublist numbering, then it represents that request is without to across the sublist sequence of metadata sublist data progress.Sending
After first request, judge whether include the number information of metadata sublist in first request, if so, avoiding the need for first number
It is ranked up again according to sublist, acquisition request the first metadata sublist can be carried in being asked first according to number information
Data, if not having, need to be ranked up each metadata sublist, to facilitate client to obtain in each metadata sublist
Data.
In the technical scheme that step S104 is provided, the data for the first metadata sublist that the reception server is returned.Service
The data of first metadata sublist in ranked metadata sublist are sent to client by device.Server can first number
It is sent to according to the number information of sublist in client, client can receive the number information of the metadata sublist.
In the technical scheme that step S106 is provided, second of the data for asking other metadata sublists is sequentially sent
Request, and the data that the reception server is returned, until server returns to end mark.
Optionally, above-mentioned embodiment can be in multiple metadata sublists, after the second request is received, server
The data of each metadata sublist can be sent in client.Optionally, the data of each metadata sublist can be advance
It is labeled, you can be ranked up to the data in metadata sublist；, can be sequentially by metadata after the second request is received
Data in sublist are sent in client.
Optionally, in the data in sending metadata sublist, the data of predetermined quantity are sent every time.Wherein, predetermined number
Amount can be predetermined quantity, and the predetermined quantity can be determined according to the disposal ability of client, i.e. client one
It is secondary can be with the quantity of processing data, when receiving data, it is not possible to beyond the quantity, server is when sending data, hair every time
The data sent can be in the quantity, and client, can be by the parameter notification server of predetermined quantity when sending request；Every time
Data sending request is received, the data of predetermined quantity can be sent in client, after request is received next time,
The data of predetermined quantity can be obtained again, and optionally, the quantity that the data bulk of this acquisition and last time get data can
To be identical.
Another optional embodiment, sequentially sends the second request of the data for asking other metadata sublists,
And the data that the reception server is returned, include until server returns to end mark：P second request is sent to server,
Wherein, P is the natural number more than 1；The data for the metadata sublist that the reception server is returned based on P second request；Right
The data of metadata sublist are performed after scheduled operation, (P+1) individual second request are sent to server, until getting whole
Data in metadata sublist.
By above-mentioned embodiment, the data in metadata sublist can be constantly got, in the data to getting
After being handled, the data of next metadata sublist can be obtained.Wherein, P is the numerical value not limited, and is asked sending second
When, can be since the metadata sublist after first asks, the end of data until getting whole metadata sublists.
For above-described embodiment, scheduled operation can include disposal data, for example, file or data are cut into fixation
The data of fragment.Can be the data of the predetermined quantity of above-mentioned embodiment for the first data, the data can be from transmission
To the second request metadata sublist in get.
Another optional embodiment, P second request is sent to server, wherein, P is the natural number more than 1；
The data for the metadata sublist that the reception server is returned based on P second request also include：Sent to server for obtaining
Second request of data in the P metadata sublist, when in server returned data without the end mark of the sublist, the above
One request obtains last data in data and, as start mark, continues to ask the data of the metadata sublist to service end,
End mark until returning to the sublist in the data of acquisition.
Optionally, above-mentioned embodiment can be that the data of multiple (such as N number of) metadata sublists are handled, Ke Yi
After the number information for receiving the metadata sublist that server is sent, other metadata is sequentially obtained according to the number information
The data of table, send P second request, the whole numbers for the metadata sublist that detection service device is returned in the above-described embodiment
According to when in the data of return without end mark (data of the metadata sublist also without all acquisition), can choosing and return
Last data is used as start mark in the data returned.Until the total data of metadata sublist is got, and according to above-mentioned
Embodiment, sequentially gets the data of other metadata sublists, the data of the N number of metadata sublist whole until getting.
Another optional embodiment, also records last in the data for having last time to receive predetermined quantity in the second request
The mark of one data.
Optionally, the mark of above-mentioned embodiment can include it is a variety of, e.g., separator, lines and letter etc., wherein,
The mark is transportable, i.e., when sending second each time and asking, can carry the data for the predetermined quantity that last time receives
In last data label information, the rearmost position of transmitted data is asked with secondary response on notification server second,
The position can have label information, by the label information, and what server can be quickly inquires the predetermined number transmitted by last time
The position of last data in the data of amount.Optionally, it can be recorded after the second request is got according to label information
Last data position, since label information record last data next data, again by other
The data of metadata sublist are sent in client.
It is preferred that, the data capture method in above-mentioned embodiment can apply to the acquisition of full dose data.In server
The full dose that can be stored with data, client will can be distinguished in the full dose data stored in obtaining server in full dose data
It is stored in different metadata sublists.
By above-mentioned embodiment, client can get the label information of multiple metadata sublists, and according to the mark
Note information constantly gets the data of predetermined quantity in metadata sublist, it is to avoid server unnecessary inquiry and sequence,
Improve the efficiency of data transfer.
Fig. 2 is the stream of the data capture method according to embodiments of the present invention optionally with distributed objects storage system
Journey figure two, as shown in Fig. 2 this method comprises the following steps：
Step S201, receives the first request that client is sent according to preset rules, wherein, first asks to obtain for request
Take the data of the first metadata sublist in distributed objects storage system；
Step S203, the data of the first metadata sublist are returned to client；
Step S205, receives the second request for being used to ask the data of other metadata sublists that client is sequentially sent；
Step S207, after the data of other metadata sublists are returned to client, returns to end mark.
By above-mentioned embodiment, server can receive the first request that client is sent according to preset rules, wherein,
First asks the data for the first metadata sublist in acquisition request distributed objects storage system, and returns to the to client
The data of one metadata sublist, after client sends the second request, being used for of can receiving that client sequentially sends asks it
Second request of the data of his metadata sublist, then, after the data of other metadata sublists are returned to client, is returned
End mark.According to above-mentioned embodiment, server can send out the data of multiple metadata sublists according to the request of client
It is sent in client, the data in other metadata sublists can be directly sent according to the second request, server is reduced
Work, improves client and the efficiency of server transmission data, improves the level of resources utilization of server and client side.
Optionally, the second request for being used to ask the data of other metadata sublists that client is sequentially sent is received
Afterwards, including：Obtain the start mark carried in the second request；In the indicated metadata sublist of the second request, from starting mark
Remember that corresponding next data start to obtain the data in metadata sublist.
Optionally, after the second request is received, the data of the corresponding metadata sublist of the second request can be sent
Into client, in second request, the data message of manageable metadata sublist is sent to server by client
In, to allow server to send corresponding data according to the second request.
Optionally, while the data of the metadata sublist obtained are returned to client, method also includes：To client
Inform whether the data in the metadata sublist that the second request is indicated obtain completion.
Wherein, after the data of predetermined quantity are got, the whether whole quilts of the data that can be inquired about in the metadata sublist
Acquisition is finished, so, after the completion of the data for informing the client metadata sublist are all acquired, and can send an end
Mark so that client obtains the data of next metadata sublist after the end mark is received, until getting whole
The data of metadata sublist.Wherein, the end mark can be announcement information, for example, notification frame, can be accused in the notification frame
Know that the data in user data table are all obtained to finish.So, client without sending solicited message again.
Here is the specific embodiment according to the present invention.
Fig. 3 is a kind of data acquisition side of client for distributed objects storage system according to embodiments of the present invention
The flow chart of method, as shown in figure 3, this method includes：
Step S301, judges whether metadata sublist has had numbering.
If so, step S303 is performed, if it is not, performing step S317.
In distributed memory system, client can receive the numbering of metadata sublist by parameter, to specific member
Data sublist carries out traversing operation.
Step S303, client request obtains Y data item of the first metadata sublist in full dose data.
Step S305, the Y data item from metadata sublist since mark is asked to service end.
When distributed storage service end (i.e. the server of above-described embodiment) system receives request, member is extracted from parameter
The numbering of data sublist, is then inquired about to the corresponding metadata sublist of the numbering and (does not carry out across sublist sequence), and handle
As a result client is returned to.
Step S307, client is handled Y data item.
After Y are disposed by client, then using this Y last as tag entry, continue to ask to service end
Y item of the metadata sublist after tag entry.
Step S309, judges whether it has been last data item in metadata sublist.
If so, step S311 is performed, if it is not, performing step S313.
Optionally, if a metadata sublist has returned to all data item, client continues to use next height
The numbering of table, carries out associative operation.
Step S311, first data item of the next metadata sublist of acquisition request.
Step S313, mark this Y data item got last.
After the flag, step S305 is returned to.
Step S315, judges whether it is last metadata sublist.
If so, step S321 is performed, if it is not, returning to step S305.
Step S317, obtains the Y item datas in full dose data successively.
Step S319, judges whether the total data in full dose data obtains and finishes.
If so, step S321 is performed, if it is not, return to step S317.
Step S321, client end operation.
This scheme is that the application scenarios insensitive to global order are optimized, and passes through the label information of exposure sublist
(as number) give mode of client, it is to avoid service end unnecessary inquiry and sequence, so as to improve efficiency.
Fig. 4 is a kind of data of service end optionally with distributed objects storage system according to embodiments of the present invention
The flow chart of acquisition methods, as shown in figure 4, this method comprises the following steps：
Step S402, receives the first request operation that client request obtains the first metadata sublist.
Step S404, sends the data of the first metadata sublist to client.
Step S406, receives the second request operation that client obtains the data of other metadata sublists.
Wherein, the corresponding metadata sublist of acquisition data item and label information, the mark are carried in the second request operation
Remember the information record position of data to be obtained.
Step S408, operates the data in other corresponding metadata sublists to be sent to client the second request, until
Data in whole metadata sublists are sent.
Optionally, when data are sent, service end (i.e. above-mentioned server) can inform the client metadata
Whether the total data in sublist, which is acquired, finishes.
Pass through above-mentioned embodiment, it is to avoid unnecessary inquiry and sequence, accelerate the speed of data transfer, save
Resource, reduces waste of the system in internal memory, CPU or internal bandwidth, it is no longer necessary to ensure the overall situation of whole metadata sublist
Sequentially, but allow data of the client one by one to metadata sublist to handle.
Fig. 5 is according to embodiments of the present invention a kind of optionally with the data-acquisition system of distributed objects storage system
Schematic diagram, as shown in figure 5, the system includes：Client 51；Server 53；Wherein, client 51 according to preset rules to clothes
Business device 53 sends first and asked, wherein, first asks to be used for the first metadata in acquisition request distributed objects storage system
The data of table；In the case where server 53 receives the first request, the data of the first metadata sublist are returned to client 51；
The data for the first metadata sublist that the reception server 53 of client 51 is returned, sequentially send for asking other metadata sublists
Data second request, and the reception server 53 return data, until server 53 return end mark.
By above-described embodiment, first can be sent to server 53 by client 51 and asked, wherein, the first request is used
The data of the first metadata sublist, the first request is received in server 53 in acquisition request distributed objects storage system
In the case of, the data of the first metadata sublist are returned to client 51, then client 51 can be returned with the reception server 53
The data of first metadata sublist, sequentially send the second request of the data for asking other metadata sublists, and receive clothes
The data that business device 53 is returned, until server 53 returns to end mark..
In embodiments of the present invention, can be according to preset rules, the data for sending acquisition metadata sublist to server please
Ask, it is to avoid in across the sublist sequence of service end, so as to obtain the data of multiple metadata sublists, improve data acquisition efficiency and resource
Utilization rate, is solved when obtaining mass data, the low technical problem of resource utilization.
Further, after the data of the first metadata sublist are returned to client 51, server 53 is additionally operable to：Receive
Second request；Server 53 returns to end mark after the data of other metadata sublists are returned to client 51.
Optionally, preset rules include：Numbered when not including metadata sublist in the first request sent to server 53
When, then it represents that the first request needs to carry out across sublist sequence to metadata sublist data；When first sent to server 53 please
When including metadata sublist numbering in asking, then it represents that request to metadata sublist data without carrying out across sublist sequence.
It is preferred that, data-acquisition system can apply to the acquisition of full dose data.
By above-mentioned embodiment, server 53 can receive the first request of the transmission of client 51, and to client 51
The data of the first metadata sublist are returned, after client sends the second request, the use that client 51 is sequentially sent can be received
In the second request of the data for asking other metadata sublists, then, the number of other metadata sublists is being returned to client 51
After, end mark is returned.According to above-mentioned embodiment, server 53 can be according to the request of client 51 by multiple first numbers
It is sent to according to the data of sublist in client 51, the data in other metadata sublists can be sent according to the second request, it is to avoid
Across sublist sequence, so as to reduce the computation burden of server 53, improves the effect that client 51 transmits data with server 53
Rate, improves the level of resources utilization of server 53 and client 51.
The embodiments of the present invention are for illustration only, and the quality of embodiment is not represented.
In the above embodiment of the present invention, the description to each embodiment all emphasizes particularly on different fields, and does not have in some embodiment
The part of detailed description, may refer to the associated description of other embodiment.
In several embodiments provided herein, it should be understood that disclosed technology contents, others can be passed through
Mode is realized.Wherein, device embodiment described above is only schematical, such as division of described unit, Ke Yiwei
A kind of division of logic function, can there is other dividing mode when actually realizing, such as multiple units or component can combine or
Person is desirably integrated into another system, or some features can be ignored, or does not perform.Another, shown or discussed is mutual
Between coupling or direct-coupling or communication connection can be the INDIRECT COUPLING or communication link of unit or module by some interfaces
Connect, can be electrical or other forms.
The unit illustrated as separating component can be or may not be it is physically separate, it is aobvious as unit
The part shown can be or may not be physical location, you can with positioned at a place, or can also be distributed to multiple
On unit.Some or all of unit therein can be selected to realize the purpose of this embodiment scheme according to the actual needs.
In addition, each functional unit in each embodiment of the invention can be integrated in a processing unit, can also
That unit is individually physically present, can also two or more units it is integrated in a unit.Above-mentioned integrated list
Member can both be realized in the form of hardware, it would however also be possible to employ the form of SFU software functional unit is realized.
If the integrated unit is realized using in the form of SFU software functional unit and as independent production marketing or used
When, it can be stored in a computer read/write memory medium.Understood based on such, technical scheme is substantially
The part contributed in other words to prior art or all or part of the technical scheme can be in the form of software products
Embody, the computer software product is stored in a storage medium, including some instructions are to cause a computer
Equipment (can for personal computer, server or network equipment etc.) perform each embodiment methods described of the invention whole or
Part steps.And foregoing storage medium includes：USB flash disk, read-only storage (ROM, Read-Only Memory), arbitrary access are deposited
Reservoir (RAM, Random Access Memory), mobile hard disk, magnetic disc or CD etc. are various can be with store program codes
Medium.
Described above is only the preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, under the premise without departing from the principles of the invention, some improvements and modifications can also be made, these improvements and modifications also should
It is considered as protection scope of the present invention.
Claims (10)
1. a kind of data capture method for distributed objects storage system, it is characterised in that including：
According to preset rules, send first to server and ask, wherein, described first asks to be used for acquisition request distributed objects
The data of first metadata sublist in storage system；
Receive the data for the first metadata sublist that the server is returned；
The second request of the data for asking other metadata sublists is sequentially sent, and receives the number that the server is returned
According to until server returns to end mark.
2. data capture method according to claim 1, it is characterised in that the preset rules include：
When not including metadata sublist numbering in the first request sent to server, then it represents that the first request needs pair
Metadata sublist data carry out across sublist sequence；
When including metadata sublist numbering in the first request sent to server, then it represents that first request is without to member
Data sublist data carry out across sublist sequence.
3. data capture method according to claim 1, it is characterised in that sequentially send for asking other metadata
Second request of the data of table, and the data that the server is returned are received, include until server returns to end mark：
P second request is sent to the server, wherein, the P is the natural number more than 1；
Receive the data for the metadata sublist that the server is returned based on the P second request；
After the data of the metadata sublist are performed with scheduled operation, sending (P+1) individual second to the server please
Ask, until getting the data in whole metadata sublists.
4. data capture method according to claim 3, it is characterised in that
It is described to send P second request to the server, wherein, the P is the natural number more than 1；
Receiving the data for the metadata sublist that the server is returned based on the P second request also includes：
The second request for obtaining data in the P metadata sublist is sent to the server,
When in the server returned data without the end mark of the metadata sublist, the request of the above one is obtained in data most
Latter data continue to ask the data of the metadata sublist to service end, returned until in the data of acquisition as start mark
Return the end mark of the sublist.
5. the data capture method according to claim 1 or claim 2, it is characterised in that the data acquisition side
Method applies to the acquisition of full dose data.
6. the data capture method according to claim 1, it is characterised in that described according to preset rules, to service
Device is sent before the first request, in addition to：The sublist quantity of the metadata is asked to server.
7. a kind of data capture method for distributed objects storage system, it is characterised in that including：
The first request that client is sent according to preset rules is received, wherein, described first asks to be used for acquisition request distribution
The data of first metadata sublist in object storage system；
The data of the first metadata sublist are returned to the client；
Receive the second request for being used to ask the data of other metadata sublists that the client is sequentially sent；
After the data of other metadata sublists are returned to the client, end mark is returned.
8. data capture method according to claim 7, it is characterised in that what the reception client was sequentially sent is used for
After the second request of data for asking other metadata sublists, including：
Obtain the start mark carried in second request；
In the indicated metadata sublist of the described second request, obtained since the corresponding next data of the start mark
Data in the metadata sublist.
9. a kind of data-acquisition system for distributed objects storage system, it is characterised in that the system includes：
Client；
Server；
Wherein, the client sends first to the server according to preset rules and asked, wherein, described first asks to be used for
The data of first metadata sublist in acquisition request distributed objects storage system；
In the case where the server receives first request, the first metadata sublist is returned to the client
Data；
The client receives the data for the first metadata sublist that the server is returned, and sequentially sends for asking it
Second request of the data of his metadata sublist, and the data that the server is returned are received, terminate mark until server is returned
Note.
10. data-acquisition system according to claim 9, it is characterised in that returning to described first to the client
After the data of metadata sublist, the server is additionally operable to：
Receive second request；
The server returns to the end mark after the data of other metadata sublists are returned to the client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710118848.8A CN107070987B (en) | 2017-03-01 | 2017-03-01 | Data acquisition method and system for distributed object storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710118848.8A CN107070987B (en) | 2017-03-01 | 2017-03-01 | Data acquisition method and system for distributed object storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107070987A true CN107070987A (en) | 2017-08-18 |
| CN107070987B CN107070987B (en) | 2020-02-14 |
Family
ID=59621805
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710118848.8A Active CN107070987B (en) | 2017-03-01 | 2017-03-01 | Data acquisition method and system for distributed object storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107070987B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107566514A (en) * | 2017-09-22 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of method that metadata request quantity is controlled in distributed memory system |
| CN108735224A (en) * | 2018-04-11 | 2018-11-02 | 北京小唱科技有限公司 | Audio modification method based on distributed frame and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105912609A (en) * | 2016-04-06 | 2016-08-31 | 中国农业银行股份有限公司 | Data file processing method and device |
| US20160321376A1 (en) * | 2015-04-28 | 2016-11-03 | Microsoft Technology Licensing, Llc | Linked data processor for database storage |
| CN106202261A (en) * | 2016-06-29 | 2016-12-07 | 浪潮(北京)电子信息产业有限公司 | The distributed approach of a kind of data access request and engine |
-
2017
- 2017-03-01 CN CN201710118848.8A patent/CN107070987B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160321376A1 (en) * | 2015-04-28 | 2016-11-03 | Microsoft Technology Licensing, Llc | Linked data processor for database storage |
| CN105912609A (en) * | 2016-04-06 | 2016-08-31 | 中国农业银行股份有限公司 | Data file processing method and device |
| CN106202261A (en) * | 2016-06-29 | 2016-12-07 | 浪潮(北京)电子信息产业有限公司 | The distributed approach of a kind of data access request and engine |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107566514A (en) * | 2017-09-22 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of method that metadata request quantity is controlled in distributed memory system |
| CN108735224A (en) * | 2018-04-11 | 2018-11-02 | 北京小唱科技有限公司 | Audio modification method based on distributed frame and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107070987B (en) | 2020-02-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103902646B (en) | Distributed task managing system and method | |
| CN107992356A (en) | Block chain affairs block processes method, electronic device and readable storage medium storing program for executing | |
| CN104090901B (en) | A kind of method that data are processed, device and server | |
| CN104111958B (en) | A kind of data query method and device | |
| CN110110006A (en) | Data managing method and Related product | |
| CN104750690B (en) | A kind of inquiry processing method, apparatus and system | |
| CN107888716A (en) | A kind of sort method of domain name resolution server, terminal device and storage medium | |
| CN104679778A (en) | Search result generating method and device | |
| CN102236581A (en) | Mapping reduction method and system thereof for data center | |
| CN105608117A (en) | Information recommendation method and apparatus | |
| CN104809130A (en) | Method, equipment and system for data query | |
| CN110175281A (en) | A kind of user data processing, exchange method, apparatus and system | |
| CN106033428B (en) | The selection method of uniform resource locator and the selection device of uniform resource locator | |
| CN106708996A (en) | Method and system for full text search of relational database | |
| CN107329991A (en) | A kind of list access to web page and methods of exhibiting and its system | |
| CN108734566A (en) | Collage-credit data querying method, terminal device and medium | |
| CN108681568A (en) | A kind of search ordering method, device, equipment and computer readable storage medium | |
| CN106446289B (en) | Information query method and device based on Pinpoint | |
| CN108154024A (en) | A kind of data retrieval method, device and electronic equipment | |
| CN107070987A (en) | Data capture method and system for distributed objects storage system | |
| CN104182546B (en) | The data query method and device of database | |
| CN107291826A (en) | File search processing method and processing device | |
| CN114070791A (en) | Data flow rate limiting processing method and device | |
| CN107943588A (en) | Data processing method, system, computer equipment and readable storage medium storing program for executing | |
| CN110119396A (en) | Data managing method and Related product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210809 Address after: 518000 b1-1302, building B, Kexing Science Park, No. 15, Keyuan Road, Science Park community, Yuehai street, Nanshan District, Shenzhen, Guangdong Patentee after: Shenzhen aijieyun Technology Co.,Ltd. Address before: Floor 5, building a, Guangqi Cultural Plaza, No. 2899 Xietu Road, Xuhui District, Shanghai 200030 Patentee before: WANGSU SCIENCE & TECHNOLOGY Co.,Ltd. |