CN106686113A - Intelligent pre-reading implementation method for distributed file system - Google Patents
Intelligent pre-reading implementation method for distributed file system Download PDFInfo
- Publication number
- CN106686113A CN106686113A CN201710039359.3A CN201710039359A CN106686113A CN 106686113 A CN106686113 A CN 106686113A CN 201710039359 A CN201710039359 A CN 201710039359A CN 106686113 A CN106686113 A CN 106686113A
- Authority
- CN
- China
- Prior art keywords
- metadata
- client
- file system
- request
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
- H04L67/5681—Pre-fetching or pre-delivering data based on network characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses an intelligent pre-reading implementation method for a distributed file system, and belongs to the field of distributed file systems. The method specifically comprises the following steps that S1, a customer transmits a file system operating request, and the request is transmitted to a meta-data service through the client; S2, the meta-data service builds a pre-reading meta-data buffer memory according to the heat degree of meta-data of a plurality of clients; S3, the meta-data service intelligent pre-judges other client buffer memories, and pushes the meta-data to all client buffer memories; S4, other customers transmits the same file system operating request, and directly hits the local meta-data buffer memory of the client. According to the invention, the plurality of clients visit a file system at the same time, and the method increases the meta-data buffer memory through the intelligent pushing of meta-data, reduces the number of direct interaction between the clients and the meta-data service, and improves the file operating performance.
Description
Technical field
The present invention relates to distributed file system field, specifically a kind of distributed file system intelligently pre-reads realization
Method.
Background technology
Metadata has become a kind of trend with the structure of data separating.Meta data server is responsible for maintaining file system
Global name space, manages the mapping relations of Documents Logical position and disk physical location, is file distribution physical memory resources,
Metadata record is in proprietary metadata storage devices.Metadata is the data of file system management file attribute information, bag
Include the information such as filename, creation time, authority, data storage location.
The metadata management of distributed file system, can be divided into symmetrical, asymmetric two class.Symmetrically refer to all clients
Using consistent caching, file operation each time is required for asking metadata, then could complete file behaviour according to metadata information
Deal with, therefore metadata request directly affects the performance of file operation.
The data that distributed type file system client side reads need to be obtained from service end through network, the number at this moment needing
According to may not be in the caching of service end, then be accomplished by being read from disk, and disk be to postpone very big to set at a slow speed
Standby, the waiting time is long.
It is existing to pre-read, by the way that the data that will be accessed are read in advance client-cache, when subsequent client is accessed
Data can in the buffer be obtained, although save the synchronous expense that data are read from disk, reach relatively low access delay.But
Be client and Metadata Service direct interaction number of times it is still more, the operating characteristics of file also has room for promotion.
The content of the invention
The technical assignment of the present invention is to provide a kind of distributed file system and intelligently pre-reads implementation method.
The technical assignment of the present invention realizes that in the following manner a kind of distributed file system intelligently pre-reads realization side
Method, concrete grammar is as follows：
S1, client send file system operation request, and by client Metadata Service is transmit a request to；
S2, Metadata Service are set up and pre-read metadata cache according to the metadata temperature of multiple client；
S3, Metadata Service intelligence anticipation other client-caches, and push metadata and cache to all clients；
S4, other clients send the request of identical file system operation, directly hit the local metadata cache of client.
Further, preferred method is, when miss local metadata cache,
S1, Metadata Service, from storage pool storage device, read metadata to internal memory；Then according to client request response visitor
Ask at family end；
S2, client receive the response of service end, return client's request.
Further, preferred method is that described service end includes Metadata Service and storage pool.
Further, preferred method is, when client sends file system operation request to browse catalogue, concrete steps
It is as follows：
S1, client send and browse catalog request, and customer end A sends and browses catalog request to Metadata Service；
S2, Metadata Service statistics metadata temperature, set up pre- read buffer；
S3, Metadata Service are pushed and pre-read metadata to other customer end Bs；
S4, customer end B browse catalog request and directly hit the local metadata cache of client；
S5, customer end B the are miss local metadata cache of client, Metadata Service, from storage pool storage device, reads first number
According to internal memory；Then according to the request of Client B requests acknowledged client end；
S6, customer end B receive the response of service end, return client's request.
Further, preferred method is, described metadata temperature, according to access temperature, the access time of metadata
And visit capacity.
A kind of intelligent pre- read apparatus of distributed file system, including client modules, metadata service module and storage pool
Module, also including the pre- read through model of intelligence increased in metadata service module；
Client modules, for sending a request to metadata service module；
Metadata service module, for the cache management of metadata；
The pre- read through model of intelligence, pre-reads metadata cache and pushes metadata for foundation；
Storage pool module, for receiving the access request of metadata service module, and returned data reading of content.
Further, preferred structure is, the pre- read through model of described intelligence, also including intelligent anticipation module, for intelligence
Anticipation, decides whether to push metadata to client.
Further, preferred structure is, described storage pool module, including storage device.
Compared to the prior art a kind of distributed file system of the present invention intelligently pre-reads implementation method, and beneficial effect is such as
Under：
1st, by intelligent intended client metadata, metadata is pushed to other clients；
2nd, improve the cache hit rate of asymmetric metadata cache, temperature, visit capacity are accessed according to client, in Metadata Service
Realize intellectual learning；
3rd, the present invention accesses file system simultaneously to multiple client, by metadata intelligently pushing, improves metadata cache, subtracts
Few client and the direct interaction times of Metadata Service, lift file operation performance.
Description of the drawings
Fig. 1 is the schematic flow sheet that a kind of distributed file system intelligently pre-reads implementation method.
Specific embodiment
It is existing to pre-read, it is ensured that read operation every time does not access disk, and the data volume pre-read by dynamic adjustment, carries significantly
The stability that high application is read.And pre-read by two-stage, client and service end have all cached a number of data, visitor
The family end caching horse back data to be used, the service end caching data to be used in future.So, client-cache can resist net
The shake that network brings, particularly when pressure is big, network delay sometimes can be long, causes to answer by caching substantial amounts of data
It is unaffected within the specific limits with program.On the other hand, the data of service end caching can resist the shake that disk brings, right
In postponing big to access the access that disk operating does not interfere with client yet individually.
A kind of distributed file system of the present invention intelligently pre-reads implementation method, improves the caching life of asymmetric metadata cache
Middle rate, increases intelligent buffer management, according to temperature, visit capacity, the access time of metadata in Metadata Service cache management
Deng metadata information, metadata cache is pushed to other distributed type file system client sides, improve the metadata hit of client
Rate.
A kind of distributed file system intelligently pre-reads implementation method, and concrete grammar is as follows：
S1, client send file system operation request, and by client Metadata Service is transmit a request to；
S2, Metadata Service are set up and pre-read metadata cache according to the metadata temperature of multiple client；Metadata temperature, according to
According to the access temperature of metadata, access time and visit capacity.
S3, Metadata Service intelligence anticipation other client-caches, and push metadata and cache to all clients；
S4, other clients send the request of identical file system operation, directly hit the local metadata cache of client.
S5, when miss local metadata cache, Metadata Service from storage pool storage device, reads metadata and arrives
Internal memory；Then according to the request of client request acknowledged client end；
S6, client receive the response of service end, return client's request, and described service end includes Metadata Service and storage
Pond.
Embodiment 1,
When client sends file system operation request to browse catalogue, comprise the following steps that：
S1, client send and browse catalog request, and customer end A sends and browses catalog request to Metadata Service；
S2, Metadata Service statistics metadata temperature, set up pre- read buffer；
S3, Metadata Service are pushed and pre-read metadata to other customer end Bs；
S4, customer end B browse catalog request and directly hit the local metadata cache of client；
S5, customer end B the are miss local metadata cache of client, Metadata Service, from storage pool storage device, reads first number
According to internal memory；Then according to the request of Client B requests acknowledged client end；
S6, customer end B receive the response of service end, return client's request.
Embodiment 2,
When client sends file system operation request to browse catalogue, comprise the following steps that：
S1, client send and browse catalog request, and customer end A sends and browses catalog request to Metadata Service；
S2, Metadata Service statistics metadata temperature, set up pre- read buffer；
S3, Metadata Service decide whether that push pre-reads metadata to other customer end Bs by intelligent anticipation；
S4, customer end B browse catalog request and directly hit the local metadata cache of client；
S5, customer end B the are miss local metadata cache of client, Metadata Service, from storage pool storage device, reads first number
According to internal memory；Then according to the request of Client B requests acknowledged client end；
S6, customer end B receive the response of service end, return client's request.
Present invention additionally comprises, a kind of intelligent pre- read apparatus of distributed file system, including client modules, Metadata Service
Module and storage pool module, also including the pre- read through model of intelligence increased in metadata service module；
Client modules, for sending a request to metadata service module；
Metadata service module, for the cache management of metadata；
The pre- read through model of intelligence, pre-reads metadata cache and pushes metadata for foundation；The pre- read through model of intelligence, it is also pre- including intelligence
Sentence module, for intelligent anticipation, decide whether to push metadata to client.
Storage pool module, it is described for receiving the access request of metadata service module, and returned data reading of content
Storage pool module, including storage device.
The present invention is mainly proposed in the asymmetric metadata cache of many clients, the heat that Metadata Service is accessed by client
The intellectual learnings such as degree, time, metadata is pushed in advance to other client metadata cachings by intellectual learning, improves metadata life
Middle rate.Temperature, visit capacity are accessed according to client, in Metadata Service intellectual learning is realized；By intelligent intended client unit
Data, push metadata to other clients, lift AS13000 product competitiveness.
The present invention accesses file system simultaneously to multiple client, by metadata intelligently pushing, improves metadata cache,
Client and the direct interaction times of Metadata Service are reduced, file operation performance is lifted.
By specific embodiment above, the those skilled in the art can readily realize the present invention.But should
Work as understanding, the present invention is not limited to above-mentioned several specific embodiments.On the basis of disclosed embodiment, the technology
The technical staff in field can the different technical characteristic of combination in any, so as to realize different technical schemes.
Claims (8)
1. a kind of distributed file system intelligently pre-reads implementation method, it is characterised in that concrete grammar is as follows：
S1, client send file system operation request, and by client Metadata Service is transmit a request to；
S2, Metadata Service are set up and pre-read metadata cache according to the metadata temperature of multiple client；
S3, Metadata Service intelligence anticipation other client-caches, and push metadata and cache to all clients；
S4, other clients send the request of identical file system operation, directly hit the local metadata cache of client.
2. a kind of distributed file system according to claim 1 intelligently pre-reads implementation method, it is characterised in that when not ordering
In local metadata cache when,
S1, Metadata Service, from storage pool storage device, read metadata to internal memory；Then according to client request response visitor
Ask at family end；
S2, client receive the response of service end, return client's request.
3. a kind of distributed file system according to claim 2 intelligently pre-reads implementation method, it is characterised in that described
Service end includes Metadata Service and storage pool.
4. a kind of distributed file system according to claim 1 and 2 intelligently pre-reads implementation method, it is characterised in that when
When client sends file system operation request to browse catalogue, comprise the following steps that：
S1, client send and browse catalog request, and customer end A sends and browses catalog request to Metadata Service；
S2, Metadata Service statistics metadata temperature, set up pre- read buffer；
S3, Metadata Service are pushed and pre-read metadata to other customer end Bs；
S4, customer end B browse catalog request and directly hit the local metadata cache of client；
S5, customer end B the are miss local metadata cache of client, Metadata Service, from storage pool storage device, reads first number
According to internal memory；Then according to the request of Client B requests acknowledged client end；
S6, customer end B receive the response of service end, return client's request.
5. a kind of distributed file system according to claim 1 intelligently pre-reads implementation method, it is characterised in that described
Metadata temperature, according to the access temperature of metadata, access time and visit capacity.
6. intelligent pre- read apparatus of a kind of distributed file system, it is characterised in that including client modules, metadata service module
With storage pool module, also including in metadata service module increase the pre- read through model of intelligence；
Client modules, for sending a request to metadata service module；
Metadata service module, for the cache management of metadata；
The pre- read through model of intelligence, pre-reads metadata cache and pushes metadata for foundation；
Storage pool module, for receiving the access request of metadata service module, and returned data reading of content.
7. a kind of intelligent pre- read apparatus of distributed file system according to claim 6, it is characterised in that described intelligence
Pre- read through model, also including intelligent anticipation module, for intelligent anticipation, decides whether to push metadata to client.
8. a kind of intelligent pre- read apparatus of distributed file system according to claim 6, it is characterised in that described storage
Pond module, including storage device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710039359.3A CN106686113A (en) | 2017-01-19 | 2017-01-19 | Intelligent pre-reading implementation method for distributed file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710039359.3A CN106686113A (en) | 2017-01-19 | 2017-01-19 | Intelligent pre-reading implementation method for distributed file system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106686113A true CN106686113A (en) | 2017-05-17 |
Family
ID=58859784
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710039359.3A Pending CN106686113A (en) | 2017-01-19 | 2017-01-19 | Intelligent pre-reading implementation method for distributed file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106686113A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107506154A (en) * | 2017-09-26 | 2017-12-22 | 郑州云海信息技术有限公司 | A kind of read method of metadata, device and computer-readable recording medium |
| CN108319634A (en) * | 2017-12-15 | 2018-07-24 | 创新科存储技术（深圳）有限公司 | The directory access method and apparatus of distributed file system |
| CN108920616A (en) * | 2018-06-28 | 2018-11-30 | 郑州云海信息技术有限公司 | A kind of metadata access performance optimization method, system, device and storage medium |
| CN109033363A (en) * | 2018-07-26 | 2018-12-18 | 广东浪潮大数据研究有限公司 | A kind of method, device and equipment of customer in response end request |
| CN111787062A (en) * | 2020-05-28 | 2020-10-16 | 北京航空航天大学 | Adaptive Fast Incremental Read-Ahead Method for Wide Area Network File System |
| CN113127420A (en) * | 2021-03-30 | 2021-07-16 | 山东英信计算机技术有限公司 | Metadata request processing method, device, equipment and medium |
| CN113326003A (en) * | 2021-05-25 | 2021-08-31 | 北京计算机技术及应用研究所 | Intelligent acceleration method for metadata access of distributed storage system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101510219A (en) * | 2009-03-31 | 2009-08-19 | 成都市华为赛门铁克科技有限公司 | File data accessing method, apparatus and system |
| CN102546751A (en) * | 2011-12-06 | 2012-07-04 | 华中科技大学 | Hierarchical metadata cache control method of distributed file system |
| CN103279568A (en) * | 2013-06-18 | 2013-09-04 | 无锡紫光存储系统有限公司 | System and method for metadata management |
| CN103312776A (en) * | 2013-05-08 | 2013-09-18 | 青岛海信传媒网络技术有限公司 | Method and device for caching contents of videos by edge node server |
-
2017
- 2017-01-19 CN CN201710039359.3A patent/CN106686113A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101510219A (en) * | 2009-03-31 | 2009-08-19 | 成都市华为赛门铁克科技有限公司 | File data accessing method, apparatus and system |
| CN102546751A (en) * | 2011-12-06 | 2012-07-04 | 华中科技大学 | Hierarchical metadata cache control method of distributed file system |
| CN103312776A (en) * | 2013-05-08 | 2013-09-18 | 青岛海信传媒网络技术有限公司 | Method and device for caching contents of videos by edge node server |
| CN103279568A (en) * | 2013-06-18 | 2013-09-04 | 无锡紫光存储系统有限公司 | System and method for metadata management |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107506154A (en) * | 2017-09-26 | 2017-12-22 | 郑州云海信息技术有限公司 | A kind of read method of metadata, device and computer-readable recording medium |
| CN108319634A (en) * | 2017-12-15 | 2018-07-24 | 创新科存储技术（深圳）有限公司 | The directory access method and apparatus of distributed file system |
| CN108319634B (en) * | 2017-12-15 | 2021-08-06 | 深圳创新科技术有限公司 | Directory access method and device for distributed file system |
| CN108920616A (en) * | 2018-06-28 | 2018-11-30 | 郑州云海信息技术有限公司 | A kind of metadata access performance optimization method, system, device and storage medium |
| CN109033363A (en) * | 2018-07-26 | 2018-12-18 | 广东浪潮大数据研究有限公司 | A kind of method, device and equipment of customer in response end request |
| CN111787062A (en) * | 2020-05-28 | 2020-10-16 | 北京航空航天大学 | Adaptive Fast Incremental Read-Ahead Method for Wide Area Network File System |
| CN111787062B (en) * | 2020-05-28 | 2021-11-26 | 北京航空航天大学 | Wide area network file system-oriented adaptive fast increment pre-reading method |
| CN113127420A (en) * | 2021-03-30 | 2021-07-16 | 山东英信计算机技术有限公司 | Metadata request processing method, device, equipment and medium |
| CN113127420B (en) * | 2021-03-30 | 2023-03-14 | 山东英信计算机技术有限公司 | Metadata request processing method, device, equipment and medium |
| CN113326003A (en) * | 2021-05-25 | 2021-08-31 | 北京计算机技术及应用研究所 | Intelligent acceleration method for metadata access of distributed storage system |
| CN113326003B (en) * | 2021-05-25 | 2024-03-26 | 北京计算机技术及应用研究所 | Intelligent acceleration method for metadata access of distributed storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106686113A (en) | Intelligent pre-reading implementation method for distributed file system | |
| CN104935648B (en) | The CDN system and file of a kind of high performance-price ratio push away in advance, the method for fragment cache memory | |
| CN104252536B (en) | A kind of internet log data query method and device based on hbase | |
| CN102014158B (en) | Cloud storage service client high-efficiency fine-granularity data caching system and method | |
| CN102088389B (en) | Distributed content access scheduling device and content reading method | |
| EP2369494A1 (en) | Web application based database system and data management method therof | |
| CN105183839A (en) | Hadoop-based storage optimizing method for small file hierachical indexing | |
| CN109240946A (en) | The multi-level buffer method and terminal device of data | |
| CN101916289A (en) | Construction Method of Digital Library Storage System Supporting Massive Small Files and Dynamic Backup Numbers | |
| CN106528451B (en) | Cloud storage framework and construction method for secondary cache prefetching of small files | |
| CN111159176A (en) | Method and system for storing and reading mass stream data | |
| CN107368608A (en) | The HDFS small documents buffer memory management methods of algorithm are replaced based on ARC | |
| CN103167036A (en) | Raster Data Access Method Based on Distributed Multi-Level Cache System | |
| CN103049393B (en) | Memory headroom management method and device | |
| CN104281673B (en) | A kind of caching structure system of database and corresponding construction method | |
| US20090119361A1 (en) | Cache management for parallel asynchronous requests in a content delivery system | |
| CN107506154A (en) | A kind of read method of metadata, device and computer-readable recording medium | |
| CN106951182A (en) | A kind of block device caching method and device | |
| US9178931B2 (en) | Method and system for accessing data by a client from a server | |
| CN102917005A (en) | Method and device supporting massive memory access to transactions | |
| CN107066503A (en) | The method and device of magnanimity metadata burst distribution | |
| CN101377788B (en) | Method and system of caching management in cluster file system | |
| CN108156428A (en) | A kind of video monitoring system | |
| CN104283981A (en) | CDN transparent cache method based on hot spot mechanism | |
| CN109634520B (en) | Storage system based on HDFS optical disc library |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170517 |
|
| RJ01 | Rejection of invention patent application after publication |