CN109739646A - A kind of data processing method and device - Google Patents
A kind of data processing method and device Download PDFInfo
- Publication number
- CN109739646A CN109739646A CN201811621057.8A CN201811621057A CN109739646A CN 109739646 A CN109739646 A CN 109739646A CN 201811621057 A CN201811621057 A CN 201811621057A CN 109739646 A CN109739646 A CN 109739646A
- Authority
- CN
- China
- Prior art keywords
- data
- data cell
- cell
- period
- access equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The invention discloses a kind of data processing method and devices.This method comprises: data access equipment obtains at least one data cell that the central processor CPU of the data access equipment is handled within the period 1；The data access equipment determines the cold and hot data target value of each data cell at least one described data cell；The data access equipment is by the cold and hot maximum T data cell of data target value at least one described data cell as first set；For the first data cell at least one described data cell, the data access equipment is if it is determined that first data cell is respectively positioned on the first set in continuous N number of period before the period 1, and be not stored in the cpu cache, then by first data cell storage into the cpu cache；First data cell is any data unit at least one described data cell.
Description
Technical field
The present invention relates to data processing field more particularly to a kind of data processing method and devices.
Background technique
In data storage, searching system, there are two types of common datas, wherein dsc data is in following a period of time, it will
The data cell frequently handled by central processing unit (CPU, Central Processing Unit)；Cold data is in future one
In the section time, the data cell that will not be frequently handled by CPU, general data amount is larger, such as historical data.Dsc data is preferential
It is put into caching, the swapping in and out that can undoubtedly reduce data in caching operates, and then improves the performance and efficiency of CPU processing, needs
Identify cold and hot data.
Therefore, how determining cold and hot data and improving data processing accuracy rate is a urgent problem to be solved.
Summary of the invention
The embodiment of the present application provides a kind of data processing method and device, solve how to determine in the prior art it is cold and hot
The problem of data and raising data processing accuracy rate.
The embodiment of the present invention provides a kind of data processing method, this method comprises:
The central processor CPU that data access equipment obtains the data access equipment is handled at least within the period 1
One data cell；
The data access equipment determines the cold and hot data target of each data cell at least one described data cell
Value；The attribute information value of the data cell determines according to the cold and hot data target value of each data cell；The attribute letter
Breath includes following one or more: the access of the predicted value, data cell of the auto regressive moving average arma modeling of data cell
Spatial cache size needed for frequency, the storage access price of data cell and data cell；
The data access equipment cold and hot maximum T data sheet of data target value at least one data cell by described in
Member is used as first set；
The data access equipment deposits the data cell for meeting the first Rule of judgment at least one described data cell
In the caching for storing up the CPU；First Rule of judgment is continuous N number of period of the data cell before the period 1
It is inside respectively positioned on the first set, and is not stored in the caching of the CPU, the N is the integer greater than 0.
Optionally, the data access equipment determines the cold and hot number of each data cell at least one described data cell
According to index value, comprising:
For each data cell at least one described data cell, the data access equipment determines the data cell
Attribute information value each single item and the corresponding weighted value of this；
The data access equipment by each single item of the attribute information value of the data cell and the corresponding weighted value of this,
Substitute into the operation result of preset formula, the cold and hot data target value as the data cell.
Optionally, the data access equipment determines each single item of the attribute information value of the data cell, comprising:
If the attribute information value of the data cell includes the storage access price of data cell, the data access equipment
According to the data cell within the period 1, accessed number and duration accessed every time determine the data cell
Storage access price.
Optionally, the data access equipment determines each single item of the attribute information value of the data cell, comprising:
If the attribute information value of the data cell includes the visiting frequency of the data cell, the data access equipment root
The number and the duration of the period 1 being accessed within the period 1 according to the data cell, determine the data cell
The visiting frequency of the data cell.
Optionally, the data access equipment will meet the data of the second Rule of judgment at least one described data cell
Unit release；Second Rule of judgment is not located in the first set for data cell, and before the period 1
Continuous N number of period in be not located in the first set.
The embodiment of the present invention provides a kind of data processing equipment, which includes:
Obtain module, at least one data cell handled within the period 1 for obtaining central processor CPU；
Processing module, for determining the cold and hot data target value of each data cell at least one described data cell；
The attribute information value of the data cell determines according to the cold and hot data target value of each data cell；The attribute information packet
It includes following one or more: the predicted value of the auto regressive moving average arma modeling of data cell, the data access equipment
Spatial cache needed for the visiting frequency of central processing unit data cell, the storage access price of data cell and data cell is big
It is small；The data cell for meeting the first Rule of judgment at least one described data cell is stored into the caching of the CPU；Institute
Stating the first Rule of judgment is that data cell is respectively positioned on the first set in continuous N number of period before the period 1,
And be not stored in the caching of the CPU, the N is the integer greater than 0.
Optionally, the processing module, is specifically used for:
For each data cell at least one described data cell, the every of the attribute information value of the data cell is determined
One and the corresponding weighted value of this；And by each single item of the attribute information value of the data cell and the corresponding power of this
Weight values substitute into the operation result of preset formula, the cold and hot data target value as the data cell.
Optionally, the processing module, is specifically used for:
If the attribute information value of the data cell includes the storage access price of data cell, existed according to the data cell
In the period 1, accessed number and duration accessed every time determine the storage access price of the data cell.
Optionally, the processing module, is specifically used for:
If the attribute information value of the data cell includes the visiting frequency of the data cell, according to the data cell in institute
The number being accessed in the period 1 and the duration of the period 1 are stated, determines the visit of the data cell of the data cell
Ask frequency.
Optionally, the processing module is also used to that the second Rule of judgment will be met at least one described data cell
Data cell release；Second Rule of judgment is not located in the first set for data cell, and in the period 1
It is not located in the first set in continuous N number of period before.
The embodiment of the present invention according to the attribute information value of each data cell at least one of, whether determine the data cell
, by CPU processing but not in the data cell of cpu cache, to ensure that accuracy in N number of period；If so, data access equipment will
The data cell is stored to cpu cache, so that the cold and hot biggish data cell of data target value is preferentially processed, is improved slow
Hit rate and hot spot data access efficiency are deposited, disposed of in its entirety performance is improved.
Detailed description of the invention
Fig. 1 is a kind of architecture diagram for data processing method respective modules that the embodiment of the present invention proposes；
Fig. 2 is the knot of the cold and hot data identifier internal module in a kind of data processing method that the embodiment of the present invention proposes
Composition；
Fig. 3 is a kind of step flow chart of data processing method provided in an embodiment of the present invention；
Fig. 4 is a kind of execution flow chart of data processing method provided in an embodiment of the present invention；
Fig. 5 is a kind of specific execution flow chart of data processing method provided in an embodiment of the present invention；
Fig. 6 is a kind of structural schematic diagram of data processing equipment provided in an embodiment of the present invention.
Specific embodiment
In order to better understand the above technical scheme, below in conjunction with Figure of description and specific embodiment to above-mentioned
Technical solution is described in detail, it should be understood that the specific features in the embodiment of the present application and embodiment are to the application skill
The detailed description of art scheme, rather than the restriction to technical scheme, in the absence of conflict, the embodiment of the present application
And the technical characteristic in embodiment can be combined with each other.
In data-storage system, there are two types of common datas, wherein dsc data is in following a period of time, it will by
The data cell that central processor (CPU, Central Processing Unit) is frequently handled；Cold data is when one section following
In, the data cell that will not be frequently handled by CPU, general data amount is larger, such as historical data.
If cooperating caching mechanism on the basis of to cold and hot data, the process performance of data-storage system can be optimized.Tool
Body caching mechanism is, using memory cache dsc data, dsc data to be preferentially put into caching, can undoubtedly reduce data changing in caching
Enter the operation that swaps out, and then improves the performance and efficiency of CPU processing.Accordingly, it is determined that cold and hot data are for meaning in distributed system
It is extremely great, for example, the distributed cache system of computing engines (spark).
The embodiment of the present invention proposes a kind of data processing method.As shown in Figure 1, the one kind proposed for the embodiment of the present invention
The architecture diagram of data processing method respective modules.For example, inquiry request is structured query language (structured
Query language, SQL) inquiry request.It should be noted that the distribution that the corresponding distributed system of Fig. 1 is spark is slow
Deposit system, but only illustrate the framework of each module of realization data processing method as example, it is not limited to other and has used the present invention
A kind of framework of the data processing method proposed in embodiment.
Inquiry request resolver 101, for the query statement of middle rank, high-level programming language to be resolved to machine language.
Logic plan generator 102 is instructed for generating corresponding logical operation according to machine language.
Logic planning optimizer 103, for optimizing the logical operation generated instruction.
Physics plan generator 104, instruction morphing for the logical operation after optimizing is machine operational order, machine behaviour
It include the physical address that CPU needs to handle data in instructing.
Cold and hot data identifier 105, CPU needs to handle the cold data and dsc data of data for identification, real for the present invention
Apply the corresponding specific module of a method.
Cache manager 106 caches for CPU management, determines cold data, the swapping in and out of dsc data.
Operations actuator 107, for executing machine operational order to the data in caching.
Raw storage section 108, for storing all data of machine operational order needs.
Data cached memory block 109, for storing the data for currently needing to operate.
Below with reference to Fig. 2, the internal module of cold and hot data identifier is introduced, as shown in Fig. 2, proposing for the embodiment of the present invention
A kind of data processing method in cold and hot data identifier internal module structure chart.In Fig. 2, history table 1051 is looked into
Ask the internal module that processing module 1052, data identification module 1053 and caching record sheet 1054 are data identifier 105.
History table 1051, for recording processing condition data.
Query processing module 1052, for receiving the machine operational order of the transmission of physics plan generator 104, according to operation
The operation note of actuator 107 identifies real data disposition, and more new engine operational order；It is also used to extraction machine behaviour
Following information in instructing: data table name (including database name), column, querying condition (optional) are identified as data cell, and
CPU handles the processing history of each data cell in record certain period of time.Processing history includes: data cell history processing
Number is counted with configurable time granularity according to time series；The data cell processing time per treatment, to locate every time
The reason returned data time subtracts the calculating of processing time started；The memory space of the buffering data units occupies.
Data identification module 1053 periodically carries out the data cell handled by CPU according to the processing history of record
The determination of cold and hot data, and notify cache manager 106 to update caching, while saving caching record after update.
Caching record table 1054, for recording cpu cache information.
As shown in figure 3, being a kind of step flow chart of data processing method provided in an embodiment of the present invention.
Step 301: data access equipment obtains the central processor CPU of the data access equipment within the period 1
At least one data cell of reason.
Step 302: each data cell is cold and hot at least one determining described data cell of the data access equipment
Data target value.
Step 303: the data access equipment cold and hot maximum T of data target value at least one data cell by described in
A data cell is as first set.
Step 304: the data access equipment will meet the number of the first Rule of judgment at least one described data cell
According to unit storage into the caching of the CPU.
First Rule of judgment is described in data cell is respectively positioned in continuous N number of period before the period 1
First set, and be not stored in the caching of the CPU, the N is the integer greater than 0.
In step 301, each data cell is the memory space for having default size.At least the one of data access equipment acquisition
A data cell is all data cells that CPU is handled within the period 1, and the period 1 is a preset period, citing
For, the period 1 is 1 minute.
In step 302, the attribute information value of the data cell is true according to the cold and hot data target value of each data cell
Fixed；The attribute information includes following one or more: the autoregressive moving average (Autoregressive of data cell
Moving average, ARMA) predicted value of model, the central processing unit data cell of the data access equipment access frequency
Spatial cache size needed for degree, the storage access price of data cell and data cell.Cold and hot finger data scale value is for characterizing
One data cell belongs to dsc data or cold data.
A kind of optional implementation of cold and hot finger data scale value for determining data cell is, at least one described data
Each data cell in unit, the data access equipment determine the attribute information value of the data cell each single item and this
Corresponding weighted value；
The data access equipment by each single item of the attribute information value of the data cell and the corresponding weighted value of this,
Substitute into the operation result of preset formula, the cold and hot data target value as the data cell.
For example,
Wherein, Score (t) is the cold and hot data target value of the data cell,For auto regressive moving average ARMA
The predicted value of model,For the visiting frequency of the central processing unit data cell of data access equipment,At CPU
Duration cost is managed, O is the storage size of the cpu cache.t1、t2、t3, α, β, γ be configurable parameter, can be according to historical record
The data set of formation carries out tune ginseng to configurable parameter by deep learning model, accurately identifies hot number using account of the history
According to.t1Typically much deeper than t2、t3, α, β, γ representative give data processing cycle or time trend factor, recent processing frequency because
The shared weight of element, processing cost factor in marking, at obvious data periodicity or time trend
Reason then α need to configure it is larger, if there is obvious Recent data processing locality then β need to configure it is larger, if different numbers
Huge according to processing cost difference, γ need to configure larger.
Optionally, data access equipment caches effect for different access scene optimization by providing different parameter configurations,
Expand the scope of application of the data processing method.
The specific each single item of attribute information optionally determines that method is as follows:
(1) predicted value of the auto regressive moving average arma modeling of data cell.
It determines that method is to be fitted using ARIMA model row, obtains prediction model:
WhereinFor predicted value,For periodic portions,For trend part,For residual error portion
Point,Reflect the processed history cycle of data cell and development trend.
(2) visiting frequency of the central processing unit data cell of data access equipment: for nearest a period of time range t2It is interior
Average treatment frequency, determine method are as follows:
If the attribute information value of the data cell includes the visiting frequency of the data cell, the data access equipment
According to number of processes and the duration of the period 1 of the data cell within the period 1, the data cell is determined
The visiting frequency of the data cell.
For example,
WhereinFor (t-t2, t] and the sum of the number of processes counting of data cell in the time.
Reflect the recent processing frequency of data cell.
(3) the determination method of CPU handling duration cost are as follows:
If the attribute information value of the data cell includes the storage access price of data cell, the data access equipment
According to the data cell within the period 1, the duration handled by the CPU number of processing and every time by the CPU,
Determine the storage access price of the data cell.
For example,
WhereinFor (t-t3, t] and the sum of the processing time of data cell in the time,For (t-
t3, t] and the sum of the number of processes counting of data cell in the time.
Ct3(t) time overhead that the processing of data cell needs is reflected.
(4) storage size of cpu cache is that the data cell stored in cpu cache takes up space corresponding value.Citing
For, every 1 Mbytes of respective value is 1.
In step 303, the data access equipment is maximum by cold and hot data target value at least one described data cell
T data cell as first set.
Optionally, the data access equipment will at least one described data cell according to each data cell described the
The sequence of the cold and hot data target value from high to low in one period, by preceding T data cell, as first set；The T
For the integer greater than 0；The K data cells of the data access equipment by the period 1 in the cpu cache,
As second set；The K is the integer greater than 0；The data access equipment is according to the first set and second collection
It closes, determines dsc data and cold data.
Optionally, the data access equipment by the first set subtract the second set as a result, as third collection
It closes；
The data access equipment by the second set subtract the first set as a result, as the 4th set；
The data access equipment by within continuous X period 1 in P data sheet of the third set
Member, as the dsc data；The P is natural number；X is positive integer；
The Q data sheet that the data access equipment will be gathered the described 4th within the continuous X period 1
Member, as the cold data；The Q is natural number.
Continuous N number of period before the period 1, adjacent with the period 1 in step 304, centre is without time interval.Separately
Outside, each period is respectively positioned on first set expression, and the cold and hot data target value of the data cell in each period is at preceding T.
Optionally, the data access equipment obtains the second data cell in the cpu cache；Second data sheet
Member is any data unit at least one described data cell；
The data access equipment is not if it is determined that second data cell is located in the first set, and described
It is not located in the first set in continuous N number of period before one period, then discharges second data cell.
Another mode of the embodiment of the present invention replaces multiple cold datas with multiple dsc datas, specifically, if the P
More than or equal to the Q, Q data cell replaces the Q before data access equipment sequence in the P data cell
A data cell；Q data cell is the cold and hot number of the P data cell according to each data cell before the sequence
The preceding Q data cell arranged in descending order according to index value；If the P is less than the Q, the data access equipment institute
It states P data cell and replaces in the Q data cell P data cell before backward；P data cell is institute before the backward
State the last P that Q data cell arranges in descending order according to the cold and hot data target value according to each data cell
A data cell.
As shown in figure 4, being a kind of execution flow chart of data processing method provided in an embodiment of the present invention.
Step 401: data access equipment determines each data cell in all data cells that CPU in the period 1 is accessed
Cold and hot data target value.
Step 402: data access equipment determines and refers to by cold and hot data according to the cold and hot data target value of each data cell
The preceding T data cell of scale value sequence.
It should be noted that this T data cell is first set, the data sheet stored in cpu cache in the period 1
Member is second set.
Step 403: data access equipment determines third set and the 4th set according to first set and second set.
It is specific to determine method, it is described in detail in step 301~304, details are not described herein again.
Step 404: data access equipment determines in third set with the presence or absence of the continuous X period 1 in third set
Element and the 4th set in the presence or absence of the continuous X period 1 the 4th gather element.
It should be noted that the element that third set, the 4th set all have the condition that meets just executes step 405, otherwise
Execute step 406.
Step 405: data access equipment is according to the element for meeting condition in third set, the 4th set, to corresponding data
Unit carries out swapping in and out operation, is specifically executed by cache manager 106.
Step 406: terminating process.
As shown in figure 5, being a kind of specific execution flow chart of data processing method provided in an embodiment of the present invention.
Step 501: data access equipment receives physics plan.The physics plan indicates data cell to be treated.
Step 502: data access equipment extracts data cell to be treated.
Step 503: data access equipment updates processing statistics.For example, the number of processes of data cell and every time place
The time of reason.
Step 504: data access equipment determines whether alignment processing unit meets the condition of swapping in and out.Specifically judge item
Part is described in detail in step 301~304, repeats no more.
Step 505: data access equipment executes caching swapping in and out operation.
Step 506: data access equipment executes processing operation to corresponding data unit.
Step 507: data access equipment updates the attribute information value of corresponding data unit according to this treatment process.
The embodiment of the present invention according to the attribute information value of each data cell at least one of, whether determine the data cell
, by CPU processing but not in the data cell of cpu cache, to ensure that accuracy in N number of period；If so, data access equipment will
The data cell is stored to cpu cache, so that the cold and hot biggish data cell of data target value is preferentially processed, is improved slow
Hit rate and hot spot data access efficiency are deposited, disposed of in its entirety performance is improved.
As shown in fig. 6, being the structural schematic diagram of data processing equipment provided in an embodiment of the present invention.
The embodiment of the present invention provides a kind of data processing equipment, which includes:
Obtain module 601, at least one data cell handled within the period 1 for obtaining central processor CPU；
Processing module 602, for determining the cold and hot data target of each data cell at least one described data cell
Value；The attribute information value of the data cell determines according to the cold and hot data target value of each data cell；The attribute letter
Breath includes following one or more: the predicted value of the auto regressive moving average arma modeling of data cell, the data access are set
Caching needed for the visiting frequency of standby central processing unit data cell, the storage access price of data cell and data cell is empty
Between size；The data cell storage of the first Rule of judgment will be met at least one described data cell to the caching of the CPU
In；First Rule of judgment is that data cell is respectively positioned on described first in continuous N number of period before the period 1
Set, and be not stored in the caching of the CPU, the N is the integer greater than 0.
Optionally, the processing module 602, is specifically used for:
For each data cell at least one described data cell, the every of the attribute information value of the data cell is determined
One and the corresponding weighted value of this；And by each single item of the attribute information value of the data cell and the corresponding power of this
Weight values substitute into the operation result of preset formula, the cold and hot data target value as the data cell.
Optionally, the processing module 602, is specifically used for:
If the attribute information value of the data cell includes the storage access price of data cell, existed according to the data cell
In the period 1, accessed number and duration accessed every time determine the storage access price of the data cell.
Optionally, the processing module 602, is specifically used for:
If the attribute information value of the data cell includes the visiting frequency of the data cell, according to the data cell in institute
The number being accessed in the period 1 and the duration of the period 1 are stated, determines the visit of the data cell of the data cell
Ask frequency.
Optionally, the processing module 602 is also used to that the second Rule of judgment will be met at least one described data cell
Data cell release；Second Rule of judgment was not located in the first set for data cell, and at described first week
It is not located in the first set in continuous N number of period before phase.
It should be understood by those skilled in the art that, the embodiment of the present invention can provide as method, system or computer program
Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the present invention
Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the present invention, which can be used in one or more,
The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces
The form of product.
The present invention be referring to according to the method for the embodiment of the present invention, the process of device (system) and computer program product
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
Although preferred embodiments of the present invention have been described, it is created once a person skilled in the art knows basic
Property concept, then additional changes and modifications may be made to these embodiments.So it includes excellent that the following claims are intended to be interpreted as
It selects embodiment and falls into all change and modification of the scope of the invention.
Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art
Mind and range.In this way, if these modifications and changes of the present invention belongs to the range of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to include these modifications and variations.
Claims (10)
1. a kind of data processing method characterized by comprising
The central processor CPU that data access equipment obtains the data access equipment handled within the period 1 at least one
Data cell；
The data access equipment determines the cold and hot data target value of each data cell at least one described data cell；Often
The attribute information value of the data cell determines according to the cold and hot data target value of a data cell；The attribute information includes
It is one or more below: the predicted value of auto regressive moving average arma modeling of data cell, the visiting frequency of data cell, number
Spatial cache size needed for storage access price and data cell according to unit；
The data access equipment makees the maximum T data cell of data target value cold and hot at least one described data cell
For first set；
The data access equipment arrives the data cell storage for meeting the first Rule of judgment at least one described data cell
In the caching of the CPU；First Rule of judgment is that data cell is equal in continuous N number of period before the period 1
It positioned at the first set, and is not stored in the caching of the CPU, the N is the integer greater than 0.
2. the method as described in claim 1, which is characterized in that the data access equipment determines at least one described data sheet
The cold and hot data target value of each data cell in member, comprising:
For each data cell at least one described data cell, the data access equipment determines the category of the data cell
The each single item and the corresponding weighted value of this of the property value of information；
The data access equipment substitutes into each single item of the attribute information value of the data cell and the corresponding weighted value of this
The operation result of preset formula, the cold and hot data target value as the data cell.
3. method according to claim 2, which is characterized in that the data access equipment determines the attribute letter of the data cell
The each single item of breath value, comprising:
If the attribute information value of the data cell includes the storage access price of data cell, the data access equipment according to
For the data cell within the period 1, accessed number and duration accessed every time determine depositing for the data cell
Store up access price.
4. method according to claim 2, which is characterized in that the data access equipment determines the attribute letter of the data cell
The each single item of breath value, comprising:
If the attribute information value of the data cell includes the visiting frequency of the data cell, the data access equipment is according to this
The number and the duration of the period 1 that data cell is accessed within the period 1, determine the described of the data cell
The visiting frequency of data cell.
5. the method as described in claim 1-4 is any, which is characterized in that further include:
The data access equipment discharges the data cell for meeting the second Rule of judgment at least one described data cell；Institute
It states the second Rule of judgment not being located in the first set for data cell, and the continuous N before the period 1
It is not located in the first set in a period.
6. a kind of data processing equipment characterized by comprising
Obtain module, at least one data cell handled within the period 1 for obtaining central processor CPU；
Processing module, for determining the cold and hot data target value of each data cell at least one described data cell；Each
The attribute information value of the data cell determines according to the cold and hot data target value of data cell；The attribute information include with
The next item down is multinomial: the center of the predicted value of the auto regressive moving average arma modeling of data cell, the data access equipment
Spatial cache size needed for the visiting frequency of processor data unit, the storage access price of data cell and data cell；
The data cell for meeting the first Rule of judgment at least one described data cell is stored into the caching of the CPU；It is described
First Rule of judgment is respectively positioned on the first set in continuous N number of period before the period 1 for data cell, and
It is not stored in the caching of the CPU, the N is the integer greater than 0.
7. device as claimed in claim 6, which is characterized in that the processing module is specifically used for:
For each data cell at least one described data cell, each single item of the attribute information value of the data cell is determined
And the corresponding weighted value of this；And by each single item of the attribute information value of the data cell and the corresponding weight of this
Value, substitutes into the operation result of preset formula, the cold and hot data target value as the data cell.
8. device as claimed in claim 7, which is characterized in that the processing module is specifically used for:
If the attribute information value of the data cell includes the storage access price of data cell, according to the data cell described
In period 1, accessed number and duration accessed every time determine the storage access price of the data cell.
9. device as claimed in claim 7, which is characterized in that the processing module is specifically used for:
If the attribute information value of the data cell includes the visiting frequency of the data cell, according to the data cell described
The number and the duration of the period 1 being accessed in one period determine the access frequency of the data cell of the data cell
Degree.
10. the device as described in claim 6-9 is any, which is characterized in that
The processing module is also used to release the data cell for meeting the second Rule of judgment at least one described data cell
It puts；Second Rule of judgment be data cell be not located in the first set, and before the period 1 described in
It is not located in the first set in continuous N number of period.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811621057.8A CN109739646A (en) | 2018-12-28 | 2018-12-28 | A kind of data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811621057.8A CN109739646A (en) | 2018-12-28 | 2018-12-28 | A kind of data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109739646A true CN109739646A (en) | 2019-05-10 |
Family
ID=66361716
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811621057.8A Pending CN109739646A (en) | 2018-12-28 | 2018-12-28 | A kind of data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109739646A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110399316A (en) * | 2019-06-28 | 2019-11-01 | 苏州浪潮智能科技有限公司 | A kind of method and apparatus for the optimization of operation system performance |
| CN112379842A (en) * | 2020-11-18 | 2021-02-19 | 深圳安捷丽新技术有限公司 | Method and device for predicting cold and hot properties of data |
| CN112699142A (en) * | 2020-12-29 | 2021-04-23 | 平安普惠企业管理有限公司 | Cold and hot data processing method and device, electronic equipment and storage medium |
| CN113064930A (en) * | 2020-12-29 | 2021-07-02 | 中国移动通信集团贵州有限公司 | Cold and hot data identification method and device of data warehouse and electronic equipment |
| CN113360553A (en) * | 2020-03-03 | 2021-09-07 | 中国移动通信集团贵州有限公司 | Data cold and hot degree evaluation method and server |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101369451A (en) * | 2007-08-14 | 2009-02-18 | 三星电子株式会社 | Solid state memory (ssm), computer system including an ssm, and method of operating an ssm |
| CN103688259A (en) * | 2011-05-19 | 2014-03-26 | 甲骨文国际公司 | Techniques for automatic data placement with compression and columnar storage |
| CN107844269A (en) * | 2017-10-17 | 2018-03-27 | 华中科技大学 | A kind of layering mixing storage system and method based on uniformity Hash |
| CN107870729A (en) * | 2016-09-23 | 2018-04-03 | 伊姆西Ip控股有限责任公司 | It is a kind of to be used for data cached method, apparatus and system |
| CN108874311A (en) * | 2018-05-29 | 2018-11-23 | 北京盛和大地数据科技有限公司 | Merge the data migration method and device in storage system |
-
2018
- 2018-12-28 CN CN201811621057.8A patent/CN109739646A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101369451A (en) * | 2007-08-14 | 2009-02-18 | 三星电子株式会社 | Solid state memory (ssm), computer system including an ssm, and method of operating an ssm |
| CN103688259A (en) * | 2011-05-19 | 2014-03-26 | 甲骨文国际公司 | Techniques for automatic data placement with compression and columnar storage |
| CN107870729A (en) * | 2016-09-23 | 2018-04-03 | 伊姆西Ip控股有限责任公司 | It is a kind of to be used for data cached method, apparatus and system |
| CN107844269A (en) * | 2017-10-17 | 2018-03-27 | 华中科技大学 | A kind of layering mixing storage system and method based on uniformity Hash |
| CN108874311A (en) * | 2018-05-29 | 2018-11-23 | 北京盛和大地数据科技有限公司 | Merge the data migration method and device in storage system |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110399316A (en) * | 2019-06-28 | 2019-11-01 | 苏州浪潮智能科技有限公司 | A kind of method and apparatus for the optimization of operation system performance |
| CN110399316B (en) * | 2019-06-28 | 2021-10-01 | 苏州浪潮智能科技有限公司 | Method and device for optimizing performance of service system |
| CN113360553A (en) * | 2020-03-03 | 2021-09-07 | 中国移动通信集团贵州有限公司 | Data cold and hot degree evaluation method and server |
| CN112379842A (en) * | 2020-11-18 | 2021-02-19 | 深圳安捷丽新技术有限公司 | Method and device for predicting cold and hot properties of data |
| CN112699142A (en) * | 2020-12-29 | 2021-04-23 | 平安普惠企业管理有限公司 | Cold and hot data processing method and device, electronic equipment and storage medium |
| CN113064930A (en) * | 2020-12-29 | 2021-07-02 | 中国移动通信集团贵州有限公司 | Cold and hot data identification method and device of data warehouse and electronic equipment |
| CN113064930B (en) * | 2020-12-29 | 2023-04-28 | 中国移动通信集团贵州有限公司 | Cold and hot data identification method and device of data warehouse and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109739646A (en) | A kind of data processing method and device | |
| Kraska et al. | Sagedb: A learned database system | |
| CN103345514B (en) | Streaming data processing method under big data environment | |
| US7716214B2 (en) | Automated and dynamic management of query views for database workloads | |
| CN107451393B (en) | Nurse Scheduling method and apparatus based on random variable neighborhood search algorithm | |
| CN104361113B (en) | A kind of OLAP query optimization method under internal memory flash memory mixing memory module | |
| CN110457397A (en) | A kind of method and device that data are synchronous | |
| JP2004518226A (en) | Database system and query optimizer | |
| JP6020448B2 (en) | Database performance prediction apparatus and database prediction method | |
| JP2017194778A (en) | Tuning device and method for relational database | |
| CN103164418A (en) | Method and device for processing serial numbers | |
| Hassan et al. | Optimizing the performance of data warehouse by query cache mechanism | |
| CN107533511A (en) | The prediction of behaviour is cached using the real time high-speed of imaginary cache | |
| Mukherjee | Synthesis of non-replicated dynamic fragment allocation algorithm in distributed database systems | |
| CN109783441A (en) | Mass data inquiry method based on Bloom Filter | |
| Wang et al. | Skyline preference query based on massive and incomplete dataset | |
| CN108509723A (en) | LRU Cache based on artificial neural network prefetch mechanism performance income evaluation method | |
| Zhang et al. | Buffer pool aware query scheduling via deep reinforcement learning | |
| CN106681830B (en) | A kind of task buffer space monitoring method and apparatus | |
| CN103365923A (en) | Method and device for assessing partition schemes of database | |
| CN112597076B (en) | Spark-oriented cache replacement method and system based on data perception | |
| Wang et al. | An Improved Memory Cache Management Study Based on Spark. | |
| Huang et al. | A grid and density based fast spatial clustering algorithm | |
| CN118210603A (en) | Cloud resource scheduling method based on enhanced growth optimizer | |
| CN117215785A (en) | Micro-service resource scheduling method for quotation playback system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190510 |
|
| RJ01 | Rejection of invention patent application after publication |