CN101271429A - Data storage method and device - Google Patents
Data storage method and device Download PDFInfo
- Publication number
- CN101271429A CN101271429A CNA2007100909194A CN200710090919A CN101271429A CN 101271429 A CN101271429 A CN 101271429A CN A2007100909194 A CNA2007100909194 A CN A2007100909194A CN 200710090919 A CN200710090919 A CN 200710090919A CN 101271429 A CN101271429 A CN 101271429A
- Authority
- CN
- China
- Prior art keywords
- piece
- data
- sub
- storage
- data block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention discloses a data storage method and a device for increasing the efficiency of data storage. The method includes: distributing a data block to the obtained data to be stored; dividing the data block into a plurality of sub-blocks according to the preset partitioning rules and storing the data in the sub-blocks. The device includes a distribution unit, a partition unit and a storage unit; the distribution unit is used for distributing the data block to the obtained data to be stored; the partition unit is used for dividing the data block into a plurality of sub-blocks according to the preset partitioning rules; the storage unit is used for storing the data in the sub-blocks. The present invention can effectively increase the efficiency of the data storage.
Description
Technical field
The present invention relates to data processing field, relate in particular to a kind of date storage method and device.
Background technology
(Internet), incomplete epoch of radio network technique in the internet, the network bandwidth is very restricted, the literal that the medium of client access, forwarding are brief often, network application system often use the general commercial database to realize the storage of medium, fast access.
But constantly perfect along with network technology, the medium that transmitted no longer are confined to brief word content, but all greatly abundant content of multimedia of the content and the form of expression, comprise sound, image, film, commercial file etc., the general commercial database is used for storing the inefficiency of these multi-medium datas, for those systems that efficient is had relatively high expectations and inapplicable.
The piece storage is complied with and is given birth to as a kind of solution that improves data storage efficiency.A kind of date storage method based on piece storage is in the prior art: the storage of all data is that the data block with fixed length is that unit stores.
In the data storage scheme in the prior art, generally data file is divided into the N piece, every size is alignd with the size of disk cluster as far as possible, when preserving data, saves the data in M the piece according to the size of data.
But in the scheme of prior art, the size of each piece is according to the size of disk cluster relative fixed all, may cause the memory by using rate low or increase the rough sledding of data storage complexity when the storage data.
For example, it is 1024 bytes that certain block storage system is set block size, and the size of data X is 1025 bytes, and then X will be stored in two data blocks.A piece is preserved the data of 1024 bytes, and piece is preserved the data of 1 byte, and the remaining space of then preserving in the data block of 1 byte data will be wasted.The if block storage system is used littler data block size definition, improves space utilization efficient, and will impact the efficient of big data storage, and make the data block index file huger, thus the overall operation efficiency of reduction block storage system.
Summary of the invention
The technical matters that the embodiment of the invention will solve provides a kind of date storage method and device, can improve the efficient of data storage.
The date storage method that the embodiment of the invention provides comprises: be the data allocations data block of the needs that get access to storage; According to the division rule that presets described data block is divided into the plurality of sub piece; With described data storage in described sub-piece.
The data storage device that the embodiment of the invention provides comprises: allocation units, division unit and storage unit; Described allocation units are used to the data allocations data block of the needs storage that gets access to; Described division unit is used for according to the division rule that presets described data block being divided into the plurality of sub piece; Described storage unit is used for described data storage in described sub-piece.
As can be seen from the above technical solutions, the embodiment of the invention has following beneficial effect:
In the embodiment of the invention, before the storage data, data block is divided into the plurality of sub piece once more, data are cut apart storage according to the size of sub-piece, thus operating factor of memory space improved, and then improved the efficient of data storage.
Description of drawings
Fig. 1 is a date storage method overview flow chart in the embodiment of the invention;
Fig. 2 is a data storage device embodiment synoptic diagram in the embodiment of the invention;
Fig. 3 is the date storage method first embodiment process flow diagram in the embodiment of the invention;
Fig. 4 is the date storage method second embodiment process flow diagram in the embodiment of the invention.
Embodiment
The embodiment of the invention provides a kind of date storage method and device, is used to improve the efficient of data storage.
See also Fig. 1, the date storage method overall procedure comprises in the embodiment of the invention:
101, distribute data piece;
After getting access to the data that need storage, from data block, select corresponding data block allocations to give these data.
102, data block is divided into the plurality of sub piece;
According to the division rule that presets described data block is divided into the plurality of sub piece, concrete division methods describes in detail in the back.
103, with data storage in sub-piece.
With described data storage in described sub-piece.
See also Fig. 2, data storage device embodiment comprises in the embodiment of the invention:
Wherein, division unit 202 comprises:
Acquiring unit 2021, computing unit 2022, performance element 2023 and identify unit 2024;
Identify unit 2024 is used to each the sub-piece that marks off that sub-bulk state sign is set.
Below in conjunction with date storage method embodiment in the embodiment of the invention is described in detail:
See also Fig. 3, date storage method first embodiment comprises in the embodiment of the invention:
301, data file is divided into the plurality of data piece;
Data file in the data-storage system is divided into the plurality of data piece.
302, for each data block status indicator is set;
After data file is divided into the plurality of data piece, for each data block is provided with status indicator, for example a data file is divided into N data block, then set up a concordance list that comprises N data bulk state sign, wherein preserve the status indicator of N data block, if 1, then representing has data in this data block, if 0, then representing does not have data in this data block.
303, according to status indicator traversal queries freed data blocks;
All data block status indicator concordance lists are traveled through, and concrete ergodic algorithm does not limit, and can be that update time is preferential, or storage time is preferential, or sequence number is preferential.
Inquiring first data block status indicator is 0, and promptly Kong Xian data block is then given the data that need storage with this data block allocations.
304, obtain firmware information;
Comprise many storages or handle firmware in data-storage system, CPU etc. for example obtains the facility information of these firmwares.
305, computing block storage size;
For the operational efficiency that makes firmware is the highest, need the physical block storage size, the piece storage size was fixed when in general the firmware operational efficiency was the highest, can calculate according to actual conditions, and calculation process and mode are prior art, do not give unnecessary details.
306, divide sub-piece and status indicator is set;
According to top definite piece storage size data block is divided into the plurality of sub piece, makes the size of each sub-piece consistent with the piece storage size of determining.
Divide and for sub-piece its status indicator is set after the sub-piece, the tentation data piece is the 4K byte, is divided into the sub-piece of 16 256 bytes, in the sub-block size of the integer data representation data block of 4 bytes of the first use of the piece of data block and idle condition separately thereof.4 bytes, promptly integer data amount to 32 bit, and preceding 16 bit represent the size of data block, and back 16 bit represent the idle condition of the sub-piece of notebook data piece.
The status indicator of each sub-piece can be created as a sub-bulk state identification index table equally, comprises the status indicator of all sub-pieces in this data block.
307, obtain size of data;
When data need be stored, obtain the size of the data that need storage.
308, data are cut apart storage;
Suppose that needing the size of data of storage is 280 bytes, then need to be stored in two sub-pieces the data of 256 bytes of storage in one of them sub-piece, the data of 24 bytes of storage in the sub-piece of another one, as if the size of data that needs storage is 5 bytes, then only need be stored in the sub-piece to get final product.
309, revise sub-bulk state sign;
After being stored into data in certain height piece, status indicator that then will this sub-piece correspondence is revised as 1, and is promptly busy.
310, when the data in certain sub-piece are deleted, refuse new data and deposit in.
Wherein Ju Jue mode has following two kinds:
One, do not revise its status indicator, promptly when the data in this sub-piece were deleted, the status indicator that still keeps this sub-piece was busy;
Two, revise status indicator, promptly when the data in this sub-piece are deleted, the status indicator of this sub-piece are revised as the free time, but delete the list item of this sub-piece from sub-bulk state identification index table, promptly this sub-piece can not deposited in new data again.
What take in the present embodiment is the second way.
311, judge whether that all sub-blocks of data are all deleted, if then execution in step 312, if not, then execution in step 310;
In the present embodiment, whether the standard of judgement is the free time for the status indicator of all the sub-pieces in this data block.
312, revise the data block status indicator.
If the status indicator of all sub-pieces is the free time in certain data block, think that then this data block is for idle, the status indicator of then revising this data block in the data block status indicator concordance list discharges the sub-piece of dividing in advance simultaneously for idle, repartitions when waiting for storage next time data.
In the foregoing description, what data block was divided into that sub-piece takes is to divide according to the situation that satisfies the firmware top efficiency, so can improve the runnability of entire system, be understandable that, divide sub-piece other mode equally in addition, introduce the embodiment of another division methods below, see also Fig. 4, date storage method second embodiment comprises in the embodiment of the invention:
401, data file is divided into the plurality of data piece;
Data file in the data-storage system is divided into the plurality of data piece.
402, for each data block status indicator is set;
After data file is divided into the plurality of data piece, for each data block is provided with status indicator, for example a data file is divided into N data block, then set up a concordance list that comprises N data bulk state sign, wherein preserve the status indicator of N data block, if 1, then representing has data in this data block, if 0, then representing does not have data in this data block.
403, according to status indicator traversal queries freed data blocks;
All data block status indicator concordance lists are traveled through, and concrete ergodic algorithm does not limit, and can be that update time is preferential, or storage time is preferential, or sequence number is preferential.
Inquiring first data block status indicator is 0, and promptly Kong Xian data block is then given the data that need storage with this data block allocations.
404, obtain the division parameter;
Dividing parameter is what set in advance, can comprise a lot of contents according to practical situations, for example divide according to 2 power power, or divide according to linear power etc.
405, divide sub-piece and status indicator is set;
According to top definite division parameter data block is divided into the plurality of sub piece, for example, the size of a data block is 4K, can be divided into the sub-piece of 2 2048 bytes, also can be divided into the sub-piece of 4 1024 bytes, also can be divided into the sub-piece of 8 512 bytes, also can be divided into the sub-piece of 16 256 bytes;
Or a data block directly is divided into 20 parts or other number part.
Divide and for sub-piece its status indicator is set after the sub-piece, the tentation data piece is the 4K byte, is divided into the sub-piece of 16 256 bytes, in the sub-block size of the integer data representation data block of 4 bytes of the first use of the piece of data block and idle condition separately thereof.4 bytes, promptly integer data amount to 32 bit, and preceding 16 bit represent the size of data block, and back 16 bit represent the idle condition of the sub-piece of notebook data piece.
The status indicator of each sub-piece can be created as a sub-bulk state identification index table equally, comprises the status indicator of all sub-pieces in this data block.
406, obtain size of data;
When data need be stored, obtain the size of the data that need storage.
407, data are cut apart storage;
Suppose that needing the size of data of storage is 280 bytes, then need to be stored in two sub-pieces the data of 256 bytes of storage in one of them sub-piece, the data of 24 bytes of storage in the sub-piece of another one, as if the size of data that needs storage is 5 bytes, then only need be stored in the sub-piece to get final product.
408, revise sub-bulk state sign;
After being stored into data in certain height piece, status indicator that then will this sub-piece correspondence is revised as 1, and is promptly busy.
409, when the needs of the data in certain sub-piece are deleted, delete the data in this sub-piece;
410, revise sub-bulk state sign.
Status indicator that will this sub-piece after the data of deletion in this sub-piece is revised as the free time, when then store data next time, this sub-piece still can be used to store data, promptly realize the recycling of sub-piece, can make the data that deposit in continuously adjacent as far as possible on physics or logical place like this, help data query.
In addition, also provide a kind of computer-readable recording medium in the embodiment of the invention, store the software of describing in the foregoing description on the described medium, comprise during this running software: be the data allocations data block of the needs that get access to storage; According to the division rule that presets described data block is divided into the plurality of sub piece; With described data storage in described sub-piece.
Aforementioned calculation machine readable storage medium storing program for executing can be a disk, CD, and flash memory can also be other storage medium.
More than a kind of date storage method and device that the embodiment of the invention provided are described in detail, according to thought of the present invention, the part that all can change in specific embodiments and applications, in sum, this description should not be construed as limitation of the present invention.
Claims (10)
1. a date storage method is characterized in that, comprising:
Data allocations data block for the storage of the needs that get access to;
According to the division rule that presets described data block is divided into the plurality of sub piece;
With described data storage in described sub-piece.
2. date storage method according to claim 1 is characterized in that, comprises before the step of described data allocations data block for the storage of the needs that get access to:
Data file in the storage system is divided into several data blocks;
For each data block is provided with the data block status indicator.
3. date storage method according to claim 2 is characterized in that, the step of described data allocations data block for the storage of the needs that get access to comprises:
All data blocks are traveled through;
Selection mode is designated idle data block, with described data block as the storage data block.
4. according to each described date storage method in the claim 1,2 or 3, it is characterized in that the division rule that described basis presets comprises the step that described data block is divided into the plurality of sub piece:
Obtain system firmware information;
According to the piece storage size under the described firmware information calculating optimum efficient;
Described data block is divided into the identical sub-piece of several piece storage sizes big or small and under the described optimum efficiency also sub-bulk state sign is set for each sub-piece.
5. according to each described date storage method in the claim 1,2 or 3, it is characterized in that the division rule that described basis presets comprises the step that described data block is divided into the plurality of sub piece:
Described data certainly are divided into 2 power power height piece and sub-bulk state sign are set for each sub-piece.
6. date storage method according to claim 4 is characterized in that, described the step of described data storage in described sub-piece is comprised:
Calculate the sub-piece number that needs use according to the size of described data and the size of described sub-piece;
Selecting a corresponding number status indicator is idle sub-piece;
Described data are cut apart storage according to the size of described sub-piece;
The described status indicator that stores the sub-piece of data is set to non-NULL.
7. date storage method according to claim 6 is characterized in that, described will comprising after the step of described data storage in described sub-piece:
When the data in the sub-piece of deletion, refuse new data and deposit described sub-piece in;
Whether the data in the judgment data piece in all sub-pieces all delete, if then the status indicator of described data block is set to the free time.
8. date storage method according to claim 6 is characterized in that, described will comprising after the step of described data storage in described sub-piece:
When the data in the sub-piece of deletion, the status indicator of described sub-piece is set to the free time;
Allow new data to deposit described sub-piece in.
9. a data storage device is characterized in that, comprising:
Allocation units, division unit and storage unit;
Described allocation units are used to the data allocations data block of the needs storage that gets access to;
Described division unit is used for according to the division rule that presets described data block being divided into the plurality of sub piece;
Described storage unit is used for described data storage in described sub-piece.
10. data storage device according to claim 9 is characterized in that, described division unit comprises:
Acquiring unit, computing unit, performance element and identify unit;
Described acquiring unit is used to obtain system firmware information, and sends described information to computing unit;
Described computing unit is used for according to the piece storage size under the described firmware information calculating optimum efficient;
Described performance element is used for described data block is divided into several identical sub-pieces of piece storage size big or small and under the described optimum efficiency;
Described identify unit is used to each the sub-piece that marks off that sub-bulk state sign is set.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2007100909194A CN101271429A (en) | 2007-03-23 | 2007-03-23 | Data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2007100909194A CN101271429A (en) | 2007-03-23 | 2007-03-23 | Data storage method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101271429A true CN101271429A (en) | 2008-09-24 |
Family
ID=40005413
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNA2007100909194A Pending CN101271429A (en) | 2007-03-23 | 2007-03-23 | Data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101271429A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104885052A (en) * | 2012-12-19 | 2015-09-02 | 高通股份有限公司 | Virtual boundary codes in a data image of a read-write memory device |
| WO2015196470A1 (en) * | 2014-06-27 | 2015-12-30 | 华为技术有限公司 | Method for writing data into flash memory device, flash memory device and storage system |
| CN106021460A (en) * | 2012-12-28 | 2016-10-12 | 华为技术有限公司 | Data processing method and device |
| CN106775449A (en) * | 2016-11-16 | 2017-05-31 | 无锡港湾网络科技有限公司 | Disaster prevention system data distribution formula storage method |
| CN107766388A (en) * | 2016-08-22 | 2018-03-06 | 平安科技（深圳）有限公司 | The processing method and processing device of attendance data |
| CN111190908A (en) * | 2018-11-15 | 2020-05-22 | 华为技术有限公司 | Data management method, device and system |
| CN111597127A (en) * | 2020-05-19 | 2020-08-28 | 深圳忆联信息系统有限公司 | Hybrid SLC caching method and device, computer equipment and storage medium |
| CN111597239A (en) * | 2020-04-10 | 2020-08-28 | 中科驭数（北京）科技有限公司 | Data alignment method and device |
| US10877680B2 (en) | 2012-12-28 | 2020-12-29 | Huawei Technologies Co., Ltd. | Data processing method and apparatus |
| CN113595557A (en) * | 2021-09-30 | 2021-11-02 | 阿里云计算有限公司 | Data processing method and device |
-
2007
- 2007-03-23 CN CNA2007100909194A patent/CN101271429A/en active Pending
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104885052A (en) * | 2012-12-19 | 2015-09-02 | 高通股份有限公司 | Virtual boundary codes in a data image of a read-write memory device |
| CN104885052B (en) * | 2012-12-19 | 2017-12-05 | 高通股份有限公司 | The virtual boundary code read and write in the data mapping of storage device |
| CN106021460A (en) * | 2012-12-28 | 2016-10-12 | 华为技术有限公司 | Data processing method and device |
| US10877680B2 (en) | 2012-12-28 | 2020-12-29 | Huawei Technologies Co., Ltd. | Data processing method and apparatus |
| CN106021460B (en) * | 2012-12-28 | 2020-09-08 | 华为技术有限公司 | Data processing method and device |
| WO2015196470A1 (en) * | 2014-06-27 | 2015-12-30 | 华为技术有限公司 | Method for writing data into flash memory device, flash memory device and storage system |
| KR101730151B1 (en) | 2014-06-27 | 2017-04-25 | 후아웨이 테크놀러지 컴퍼니 리미티드 | Method for writing data into flash memory apparatus, flash memory apparatus, and storage system |
| US10203899B2 (en) | 2014-06-27 | 2019-02-12 | Huawei Technologies Co., Ltd. | Method for writing data into flash memory apparatus, flash memory apparatus, and storage system |
| CN107766388B (en) * | 2016-08-22 | 2020-09-01 | 平安科技（深圳）有限公司 | Attendance data processing method and device |
| CN107766388A (en) * | 2016-08-22 | 2018-03-06 | 平安科技（深圳）有限公司 | The processing method and processing device of attendance data |
| CN106775449A (en) * | 2016-11-16 | 2017-05-31 | 无锡港湾网络科技有限公司 | Disaster prevention system data distribution formula storage method |
| CN111190908A (en) * | 2018-11-15 | 2020-05-22 | 华为技术有限公司 | Data management method, device and system |
| CN111190908B (en) * | 2018-11-15 | 2023-09-22 | 华为技术有限公司 | Data management method, device and system |
| CN111597239A (en) * | 2020-04-10 | 2020-08-28 | 中科驭数（北京）科技有限公司 | Data alignment method and device |
| CN111597239B (en) * | 2020-04-10 | 2021-08-31 | 中科驭数（北京）科技有限公司 | Data alignment method and device |
| CN111597127A (en) * | 2020-05-19 | 2020-08-28 | 深圳忆联信息系统有限公司 | Hybrid SLC caching method and device, computer equipment and storage medium |
| CN111597127B (en) * | 2020-05-19 | 2023-07-04 | 深圳忆联信息系统有限公司 | Method, device, computer equipment and storage medium for hybrid SLC cache |
| CN113595557A (en) * | 2021-09-30 | 2021-11-02 | 阿里云计算有限公司 | Data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101271429A (en) | Data storage method and device | |
| KR102462781B1 (en) | KVS tree database | |
| JP6025149B2 (en) | System and method for managing data | |
| CN110268394B (en) | Method, system and machine readable storage medium for storing and manipulating key value data | |
| CN106776967B (en) | Method and device for storing massive small files in real time based on time sequence aggregation algorithm | |
| CN102467570B (en) | Connection query system and method for distributed data warehouse | |
| US10019459B1 (en) | Distributed deduplication in a distributed system of hybrid storage and compute nodes | |
| CN101692252B (en) | Method for distributing and reclaiming idle blocks of file | |
| CN101727465B (en) | Methods for establishing and inquiring index of distributed column storage database, device and system thereof | |
| CN110149803A (en) | Date storage method, system and terminal device | |
| CN104794177B (en) | A kind of date storage method and device | |
| TWI738037B (en) | Partition splitting method, device, electronic equipment and readable storage medium | |
| CN105117417A (en) | Read-optimized memory database Trie tree index method | |
| JP2014225260A (en) | Database control part, method and program for managing dispersion type data storage | |
| JP2021501389A (en) | Data hierarchy storage and hierarchy search method and device | |
| CN103678172A (en) | Local data cache management method and device | |
| DE102017118341B4 (en) | Repartitioning of data in a distributed computer system | |
| CN116089414B (en) | Time sequence database writing performance optimization method and device based on mass data scene | |
| CN102184229A (en) | Memory data storage method of huge RFID (Radio Frequency Identification Device) label information | |
| CN102054000A (en) | Data querying method, device and system | |
| US10223256B1 (en) | Off-heap memory management | |
| CN105468541B (en) | A kind of buffer memory management method towards lucidification disposal intelligent terminal | |
| CN110941619A (en) | Method for defining graph data storage model and structure for multiple use scenarios | |
| CN112527836B (en) | Big data query method based on T-BOX platform | |
| CN103778219B (en) | HBase-based method for updating incremental indexes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Open date: 20080924 |