CN110995273B - Data compression method, device, equipment and medium for power database - Google Patents
Data compression method, device, equipment and medium for power database Download PDFInfo
- Publication number
- CN110995273B CN110995273B CN201910998288.9A CN201910998288A CN110995273B CN 110995273 B CN110995273 B CN 110995273B CN 201910998288 A CN201910998288 A CN 201910998288A CN 110995273 B CN110995273 B CN 110995273B
- Authority
- CN
- China
- Prior art keywords
- value
- data
- compression
- compressed
- algorithm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013144 data compression Methods 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000007906 compression Methods 0.000 claims abstract description 85
- 230000006835 compression Effects 0.000 claims abstract description 85
- 238000004364 calculation method Methods 0.000 claims abstract description 41
- 241000700605 Viruses Species 0.000 claims description 8
- 238000013500 data storage Methods 0.000 claims description 8
- 238000012216 screening Methods 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012731 temporal analysis Methods 0.000 description 1
- 238000000700 time series analysis Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- H03M7/3059—Digital compression and data reduction techniques where the original information is represented by a subset or similar information, e.g. lossy compression
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The invention provides a data compression method, a device, equipment and a medium for a power database. The method comprises the following steps: acquiring a time series data set, the time series data set comprising: a timestamp and a data value associated with the timestamp; and establishing a high-performance high-compression-ratio compression algorithm, and respectively compressing the timestamp and the value according to the compression algorithm. When the timestamp is compressed, recording a first time point, only recording an offset value at the later time, and compressing and storing the offset value; when the values are compressed, an exclusive-or algorithm is established, two data values are subjected to exclusive-or calculation according to the exclusive-or algorithm to obtain a calculation result, and the values are compressed and stored according to the calculation result. According to the time sequence data compression method and device, the time sequence data set is divided into the time stamps and the data values related to the time stamps, the time stamps are compressed through the compression algorithm, and the data values are compressed through the XOR algorithm, so that the precision of the time sequence data is not influenced, and meanwhile, the compression can be carried out more effectively.
Description
Technical Field
The invention relates to the technical field of wind power plant monitoring, in particular to a data compression method, device, equipment and medium for a power database.
Background
Data compression is a technical method for reducing the data volume to reduce the storage space and improve the transmission, storage and processing efficiency of the data on the premise of not losing useful information, or reorganizing the data according to a certain algorithm and reducing the redundancy and storage space of the data.
However, for the storage system of the existing monitoring system, data compression is generally realized by dead zone compression and spiral gate compression, and such data compression algorithms are usually at the cost of losing data accuracy, but the compression ratio of time series data is often limited by a common lossless compression mode, so that providing a compression method which has no influence on the accuracy of time series data and can more effectively compress the time series data becomes an urgent problem to be solved.
The above is only for the purpose of assisting understanding of the technical aspects of the present invention, and does not represent an admission that the above is prior art.
Disclosure of Invention
In view of this, the present invention provides a data compression method, apparatus, device and medium for a power database, and aims to solve the problem that the prior art cannot provide a compression method that can compress time series data more effectively without affecting the precision of the time series data.
The technical scheme of the invention is realized as follows:
in one aspect, the present invention provides a data compression method for a power database, where the data compression method for the power database includes the following steps:
s1, acquiring a time sequence data set, wherein the time sequence data set comprises: a timestamp and a data value associated with the timestamp;
s2, establishing a compression algorithm, setting a range of a compression threshold value, compressing the timestamp according to the compression algorithm, obtaining a compressed value, comparing the compression threshold value with the compressed value, and storing the compressed value according to a comparison result;
and S3, establishing an exclusive-OR algorithm, carrying out exclusive-OR calculation on every two data values according to the exclusive-OR algorithm to obtain a calculation result, and compressing and storing the data values according to the calculation result.
On the basis of the above technical solution, preferably, in step S1, a time series data set is obtained, where the time series data set includes: a time stamp and a data value associated with the time stamp, further comprising the steps of obtaining a time series data set, the time series data set comprising: and sequencing the timestamps according to the sequence of the timestamps and the data values associated with the timestamps, taking the first timestamp as the starting time, and scanning the virus of the time sequence data set to screen out the time sequence data set with the virus.
On the basis of the above technical solution, preferably, the step S2 further includes the following steps, and the compression algorithm is:
D＝(t n -t n-1 )-(t n-1 -t n-2 )；
where D is the compressed value, t n Represents a time stamp, n ≧ 0, and t when n =0 -1 Is the start time, if D is 0, 1 bit '0' is stored; if D belongs to [ -63,64]Storing a '10' followed by a 7-bit D value; if D belongs to [ -255,256 [)]Storing a '110' heel 9-bit D value; if D belongs to [ -2047,2048]Memory'1110' followed by a 12 bit D value; otherwise, the '1111' followed by the 32-bit D value is stored.
On the basis of the above technical solution, preferably, in step S2, a compression algorithm is established, a range of a compression threshold is set, the timestamp is compressed according to the compression algorithm, a compressed value is obtained, the compression threshold is compared with the compression value, and the compression value is stored according to a comparison result, and the method further includes the steps of establishing the compression algorithm, setting the range of the compression threshold, setting a data storage manner of the range, compressing the timestamp according to the compression algorithm, obtaining the compressed value, comparing the compression threshold with the compression value, determining the range of the compression value according to the comparison result, and storing the compression value according to the range and a corresponding data storage manner.
On the basis of the above technical solution, preferably, in step S3, an exclusive or algorithm is established, two by two data values are subjected to exclusive or calculation according to the exclusive or algorithm, a calculation result is obtained, and the data values are compressed and stored according to the calculation result, and the method further includes the steps of establishing an exclusive or algorithm, setting a storage mode corresponding to the calculation result, performing two by two exclusive or calculation on the data values according to the exclusive or algorithm, and when the exclusive or result is 0, the system stores 1 bit of '0'; when the XOR result is not 0, the system stores a 1 bit of '1'.
On the basis of the technical scheme, preferably, the data values are subjected to XOR calculation pairwise according to an XOR algorithm, and when the XOR result is 0, the system stores 1 bit of '0'; when the XOR result is not 0, the system stores 1 bit '1', and further comprises the following steps of carrying out XOR calculation on every two data values according to an XOR algorithm, judging the compressed data values when the XOR result is 0 and the system stores 1 bit '0', and only storing the XOR values if the number of leading zeros and trailing zeros in the compressed data values is the same as the number of leading zeros and trailing zeros in the previous compressed data values; when the XOR result is not 0, the system stores 1 bit of '1', stores the leading zero length with 5 bits, stores the length value of the XOR value with 6 bits, and finally stores the XOR value.
On the basis of the above technical solution, preferably, the data values are compressed and stored according to the calculation result, and the method further includes the steps of examining the data values, marking the values appearing in the data values for many times, screening out the values not marked, and classifying, compressing and storing the values not marked according to the corresponding data values.
Still further preferably, the data compression device of the power database includes:
an acquisition module configured to acquire a time series data set, the time series data set including: a timestamp and a data value associated with the timestamp;
the time stamp compression storage module is used for establishing a compression algorithm, setting the range of a compression threshold value, compressing the time stamp according to the compression algorithm, acquiring a compressed value, comparing the compression threshold value with the compressed value, and storing the compressed value according to a comparison result;
and the data value compression storage module is used for establishing an exclusive-or algorithm, carrying out exclusive-or calculation on every two data values according to the exclusive-or algorithm to obtain a calculation result, and compressing and storing the data values according to the calculation result.
In a second aspect, the method for data compression of the power database further comprises an apparatus, the apparatus comprising: a memory, a processor and a data compression method program of a power database stored on the memory and executable on the processor, the data compression method program of the power database being configured to implement the steps of the data compression method of the power database as described above.
In a third aspect, the data compression method of the power database further includes a medium, which is a computer medium, and the computer medium stores thereon a data compression method program of the power database, and the data compression method program of the power database, when executed by a processor, implements the steps of the data compression method of the power database as described above.
Compared with the prior art, the data compression method of the power database has the following beneficial effects:
(1) The time sequence data set is divided into the time stamps and the data values associated with the time stamps, the time stamps and the data values are compressed respectively, and after the compression is completed, the compressed time stamps and the corresponding compressed data values are packed and stored;
(2) The time sequence data is processed and compressed by using a compression algorithm and an exclusive-or algorithm, the compression algorithm can set a corresponding compression range and a corresponding storage mode, and after the data is compressed, the corresponding storage mode can be selected according to a compressed data value.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic diagram of an apparatus in a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart illustrating a first embodiment of a data compression method for a power database according to the present invention;
fig. 3 is a functional block diagram illustrating a data compression method for a power database according to a first embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
As shown in fig. 1, the apparatus may include: a processor 1001, such as a Central Processing Unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a WIreless interface (e.g., a WIreless-FIdelity (WI-FI) interface). The Memory 1005 may be a Random Access Memory (RAM) Memory, or may be a Non-Volatile Memory (NVM), such as a disk Memory. The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the configuration shown in fig. 1 does not constitute a limitation of the device, and that in actual implementations the device may include more or less components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a medium, may include therein data compression method programs of an operating system, a network communication module, a user interface module, and a power database.
In the device shown in fig. 1, the network interface 1004 is mainly used to establish a communication connection between the device and a server that stores all data required in the data compression method system of the power database; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 of the data compression method apparatus for a power database of the present invention may be provided in a data compression method apparatus for a power database that calls a data compression method program of a power database stored in the memory 1005 through the processor 1001 and executes a data compression method of a power database provided by the present invention.
Referring to fig. 2, fig. 2 is a schematic flow chart of a data compression method of a power database according to a first embodiment of the present invention.
In this embodiment, the data compression method for the power database includes the following steps:
s10: acquiring a time series data set, the time series data set comprising: a timestamp and a data value associated with the timestamp.
It should be understood that the time series data refers to time series data, the time series data is a data column recorded in time sequence by the same unified index, each data in the same data column must have the same caliber, and is required to have comparability, the time series data may be a time period number or a time point number, the purpose of time series analysis is to construct a time series model by finding out the statistical characteristics and the development rule of the time series in a sample, and perform an off-sample prediction, in this embodiment, the time series data set includes: a timestamp and a data value associated with the timestamp.
It should be understood that a timestamp refers to a complete, verifiable piece of data that indicates that a piece of data existed before a particular time, and is typically a sequence of characters that uniquely indicates the time at a particular time.
It should be understood that after the time series data set is acquired, the timestamps are sorted according to the sequence of the timestamps, the timestamp arranged at the first position is taken as a starting timestamp, and then the subsequent timestamps are named in sequence, wherein the naming mode is as follows: a first timestamp. Meanwhile, the system scans the virus for the time sequence data set so as to prevent the virus from damaging the data and influencing the compression efficiency.
S20: establishing a compression algorithm, setting a range of a compression threshold value, compressing the timestamp according to the compression algorithm, acquiring a compressed value, comparing the compression threshold value with the compressed value, and storing the compressed value according to a comparison result.
It should be understood that the compression algorithm is:
D＝(t n -t n-1 )-(t n-1 -t n-2 )；
where D is the compressed value, t n Represents a timestamp, n is greater than or equal to 0, when n isWhen =0, t -1 Is the start time, if D is 0, 1 bit '0' is stored; if D belongs to [ -63,64]Storing a '10' followed by a 7-bit D value; if D belongs to [ -255,256 [)]Store the '110' followed by a 9-bit D value; if D belongs to [ -2047,2048]Storing a '1110' heel 12-bit D value; otherwise, the '1111' followed by the 32-bit D value is stored.
It should be understood that a definition is added to the compression algorithm, and the compression algorithm is labeled func: D = (t) n -t n-1 )-(t n-1 -t n-2 ) The func is a definition added to the compression algorithm, has no meaning, and only plays a role in marking the compression algorithm.
It should be appreciated that establishing a compression algorithm and setting a range of compression thresholds, compressing the timestamp may compare the compressed value to the range of compression thresholds and store the compressed value according to the range of compression thresholds.
S30: and establishing an exclusive-or algorithm, performing exclusive-or calculation on every two data values according to the exclusive-or algorithm to obtain a calculation result, and compressing and storing the data values according to the calculation result.
It should be understood that the exclusive or applies to the logical operation, which is the algorithm: if the values of a and b are different, the XOR result is 1; if the values of a and b are the same, the XOR result is 0. In this implementation, when the xor result of two data values is 0, the system stores 1 bit of '0', and meanwhile, if the number of leading zeros and trailing zeros in the compressed data value is the same as the number of leading zeros and trailing zeros in the previous compressed data value, only the xor value is stored; when the xor result of the two data values is not 0, the system stores 1 bit of '1', stores the leading zero length with 5 bits, stores the length value of the xor value with 6 bits, and finally stores the xor value.
The above description is only for illustration and does not limit the technical solution of the present application in any way.
As is apparent from the above description, the present embodiment obtains a time-series data set including: a timestamp and a data value associated with the timestamp; and establishing a high-performance high-compression-ratio compression algorithm, and respectively compressing the timestamp and the value according to the compression algorithm. When the timestamp is compressed, recording a first time point, only recording an offset value at the later time, and compressing and storing the offset value; when the values are compressed, an exclusive-or algorithm is established, two data values are subjected to exclusive-or calculation according to the exclusive-or algorithm to obtain a calculation result, and the values are compressed and stored according to the calculation result. The time sequence data set is divided into the time stamps and the data values associated with the time stamps, the time stamps are compressed through the compression algorithm, the data values are compressed through the XOR algorithm, the precision of the time sequence data is not affected, and meanwhile the time sequence data can be compressed more effectively.
In addition, the embodiment of the invention also provides a data compression device of the power database. As shown in fig. 3, the data compression apparatus of the power database includes: the device comprises an acquisition module 10, a timestamp compression storage module 20 and a data value compression storage module 30.
An obtaining module 10, configured to obtain a time series data set, where the time series data set includes: a timestamp and a data value associated with the timestamp;
the timestamp compression storage module 20 is configured to establish a compression algorithm, set a range of a compression threshold, compress the timestamp according to the compression algorithm, obtain a compressed value, compare the compression threshold with the compressed value, and store the compressed value according to a comparison result;
and the data value compression and storage module 30 is used for establishing an exclusive-or algorithm, performing exclusive-or calculation on every two data values according to the exclusive-or algorithm to obtain a calculation result, and compressing and storing the data values according to the calculation result.
In addition, it should be noted that the above-described embodiments of the apparatus are merely illustrative, and do not limit the scope of the present invention, and in practical applications, a person skilled in the art may select some or all of the modules to implement the purpose of the embodiments according to actual needs, and the present invention is not limited herein.
In addition, the technical details that are not described in detail in this embodiment may refer to the data compression method of the power database provided in any embodiment of the present invention, and are not described herein again.
Furthermore, an embodiment of the present invention further provides a medium, where the medium is a computer medium, and the computer medium stores a data compression method program of a power database, where the data compression method program of the power database, when executed by a processor, implements the following operations:
s1, acquiring a time sequence data set, wherein the time sequence data set comprises: a timestamp and a data value associated with the timestamp;
s2, establishing a compression algorithm, setting a range of a compression threshold value, compressing the timestamp according to the compression algorithm, obtaining a compressed value, comparing the compression threshold value with the compressed value, and storing the compressed value according to a comparison result;
and S3, establishing an XOR algorithm, performing XOR calculation on the data values pairwise according to the XOR algorithm to obtain a calculation result, and compressing and storing the data values according to the calculation result.
Further, the data compression method program of the power database when executed by the processor further realizes the following operations:
acquiring a time series data set, the time series data set comprising: and sequencing the timestamps according to the sequence of the timestamps and the data values associated with the timestamps, taking the first timestamp as the starting time, and scanning the virus of the time sequence data set to screen out the time sequence data set with the virus.
Further, the data compression method program of the power database further realizes the following operations when executed by the processor:
the compression algorithm is as follows:
D＝(t n -t n-1 )-(t n-1 -t n-2 )；
where D is the compressed value, t n Represents a time stamp, n ≧ 0, and t when n =0 -1 Is a starting time, e.g.If D is 0, then store 1 bit '0'; if D belongs to [ -63,64]Storing a '10' followed by a 7-bit D value; if D belongs to [ -255,256 [)]Store the '110' followed by a 9-bit D value; if D belongs to [ -2047,2048]Storing a '1110' heel 12-bit D value; otherwise, the '1111' followed by the 32-bit D value is stored.
Further, the data compression method program of the power database further realizes the following operations when executed by the processor:
establishing a compression algorithm, setting a range of a compression threshold value, setting a data storage mode of the range, compressing the timestamp according to the compression algorithm, acquiring a compressed value, comparing the compression threshold value with the compressed value, judging the range of the compressed value according to a comparison result, and storing the compressed value according to the range and a corresponding data storage mode.
Further, the data compression method program of the power database when executed by the processor further realizes the following operations:
establishing an exclusive-or algorithm, setting a storage mode corresponding to a calculation result, performing exclusive-or calculation on data values pairwise according to the exclusive-or algorithm, and when the exclusive-or result is 0, storing 1 bit of '0' by the system; when the XOR result is not 0, the system stores a 1 bit of '1'.
Further, the data compression method program of the power database when executed by the processor further realizes the following operations:
performing XOR calculation on every two data values according to an XOR algorithm, judging the compressed data values when the XOR result is 0 and the system stores 1 bit of '0', and only storing the XOR values if the number of leading zeros and trailing zeros in the compressed data values is the same as that of leading zeros and trailing zeros in the previous compressed data values; when the XOR result is not 0, the system stores 1 bit of '1', stores the leading zero length with 5 bits, stores the length value of the XOR value with 6 bits, and finally stores the XOR value.
Further, the data compression method program of the power database further realizes the following operations when executed by the processor:
and checking the data values, marking numerical values appearing in the data values for many times, screening unmarked numerical values, classifying, compressing and storing the unmarked numerical values according to the corresponding data values.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and should not be taken as limiting the scope of the present invention, which is intended to cover any modifications, equivalents, improvements, etc. within the spirit and scope of the present invention.
Claims (6)
1. A data compression method of a power database is characterized in that: comprises the following steps;
s1, acquiring a time sequence data set, wherein the time sequence data set comprises: a timestamp and a data value associated with the timestamp;
s2, establishing a compression algorithm, setting a range of a compression threshold value, setting a data storage mode of the range, compressing the timestamp according to the compression algorithm, obtaining a compressed value, comparing the compression threshold value with the compressed value, judging the range of the compressed value according to a comparison result, and storing the compressed value according to the range and a corresponding data storage mode; wherein, the compression algorithm is as follows:
D＝(t n -t n-1 )-(t n-1 -t n-2 )；
where D is the compressed value, t n Represents a time stamp, n ≧ 0, and t when n =0 -1 Is the start time, if D is 0, 1 bit '0' is stored; if D belongs to [ -63,64]Storing a '10' followed by a 7-bit D value; if D belongs to [ -255,256]Store the '110' followed by a 9-bit D value; if D belongs to [ -2047,2048]Storing a '1110' heel 12-bit D value; otherwise, store the '1111' followed by a 32-bit D value;
s3, establishing an XOR algorithm, setting a storage mode corresponding to a calculation result, performing XOR calculation on every two data values according to the XOR algorithm to obtain the calculation result, and compressing and storing the data values according to the calculation result; wherein,
when the XOR result is 0, the system stores 1 bit '0', judges the compressed data value, and only stores the XOR value if the number of leading zeros and trailing zeros in the compressed data value is the same as the number of leading zeros and trailing zeros in the previous compressed data value;
when the XOR result is not 0, the system stores 1 bit '1'; the leading zero length is stored with 5 bits, the length value of the xor value is stored with 6 bits, and finally the xor value is stored.
2. The data compression method of the power database according to claim 1, characterized in that: in step S1, a time series data set is obtained, the time series data set including: a timestamp and a data value associated with the timestamp, further comprising the steps of obtaining a time series data set, the time series data set comprising: and sequencing the timestamps according to the sequence of the timestamps and the data values associated with the timestamps, taking the first timestamp as the starting time, and scanning the virus of the time sequence data set to screen out the time sequence data set with the virus.
3. The method of data compression of a power database according to claim 1, wherein: and compressing and storing the data values according to the calculation result, and also comprises the following steps of checking the data values, marking numerical values appearing for many times in the data values, screening numerical values which are not marked, and classifying, compressing and storing the numerical values which are not marked according to the corresponding data values.
4. A data compression device of a power database is characterized by comprising:
an acquisition module configured to acquire a time series data set, the time series data set including: a timestamp and a data value associated with the timestamp;
the time stamp compression storage module is used for establishing a compression algorithm, setting the range of a compression threshold value, setting a data storage mode of the range, compressing the time stamp according to the compression algorithm, acquiring a compressed value after compression, comparing the compression threshold value with the compressed value, judging the range of the compressed value according to a comparison result, and storing the compressed value according to the range and a corresponding data storage mode; wherein, the compression algorithm is as follows:
D＝(t n -t n-1 )-(t n-1 -t n-2 )；
where D is the compressed value, t n Represents a time stamp, n ≧ 0, and t when n =0 -1 Is the start time, if D is 0, 1 bit '0' is stored; if D belongs to [ -63,64]Storing a '10' followed by a 7-bit D value; if D belongs to [ -255,256 [)]Storing a '110' heel 9-bit D value; if D belongs to [ -2047,2048]Storing a '1110' heel 12-bit D value; otherwise, store the '1111' followed by a 32-bit D value;
the data value compression storage module is used for establishing an exclusive-or algorithm, setting a storage mode corresponding to a calculation result, performing exclusive-or calculation on every two data values according to the exclusive-or algorithm to obtain the calculation result, and compressing and storing the data values according to the calculation result; wherein,
when the XOR result is 0, the system stores 1 bit '0', judges the compressed data value, and only stores the XOR value if the number of leading zeros and trailing zeros in the compressed data value is the same as the number of leading zeros and trailing zeros in the previous compressed data value;
when the XOR result is not 0, the system stores 1 bit '1'; the leading zero length is stored with 5 bits, the length value of the xor value is stored with 6 bits, and finally the xor value is stored.
5. An apparatus, characterized in that the apparatus comprises: a memory, a processor and a data compression method program of a power database stored on the memory and executable on the processor, the data compression method program of the power database being configured to implement the steps of the data compression method of the power database as claimed in any one of claims 1 to 3.
6. A medium, characterized in that the medium is a computer medium having stored thereon a data compression method program of a power database, which when executed by a processor implements the steps of the data compression method of the power database according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910998288.9A CN110995273B (en) | 2019-10-21 | 2019-10-21 | Data compression method, device, equipment and medium for power database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910998288.9A CN110995273B (en) | 2019-10-21 | 2019-10-21 | Data compression method, device, equipment and medium for power database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110995273A CN110995273A (en) | 2020-04-10 |
| CN110995273B true CN110995273B (en) | 2023-04-07 |
Family
ID=70082191
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910998288.9A Active CN110995273B (en) | 2019-10-21 | 2019-10-21 | Data compression method, device, equipment and medium for power database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110995273B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113078908B (en) * | 2021-03-10 | 2022-03-25 | 杭州又拍云科技有限公司 | Simple encoding and decoding method suitable for time sequence database |
| CN114327264B (en) * | 2021-12-22 | 2023-05-12 | 北京力控元通科技有限公司 | Time sequence data compression method, device and equipment |
| CN114374393A (en) * | 2021-12-31 | 2022-04-19 | 天津大学 | Redundancy removal compression method for industrial Internet of things time sequence data |
| CN114665885B (en) * | 2022-03-29 | 2022-11-04 | 北京诺司时空科技有限公司 | Self-adaptive data compression method for time sequence database |
| CN117118456B (en) * | 2023-10-25 | 2024-01-26 | 山东德源电力科技股份有限公司 | Magnetic control switch control data processing method based on depth fusion |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103577456A (en) * | 2012-07-31 | 2014-02-12 | 国际商业机器公司 | Method and device for processing time series data |
| CN108153483A (en) * | 2016-12-06 | 2018-06-12 | 南京南瑞继保电气有限公司 | A kind of time series data compression method based on attribute grouping |
| CN110266316A (en) * | 2019-05-08 | 2019-09-20 | 阿里巴巴集团控股有限公司 | A kind of data compression, decompressing method, device and equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3853765B2 (en) * | 2002-11-08 | 2006-12-06 | Ｎｅｃインフロンティア株式会社 | Packet compression method, packet restoration method, packet compression method, and packet restoration method |
-
2019
- 2019-10-21 CN CN201910998288.9A patent/CN110995273B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103577456A (en) * | 2012-07-31 | 2014-02-12 | 国际商业机器公司 | Method and device for processing time series data |
| CN108153483A (en) * | 2016-12-06 | 2018-06-12 | 南京南瑞继保电气有限公司 | A kind of time series data compression method based on attribute grouping |
| CN110266316A (en) * | 2019-05-08 | 2019-09-20 | 阿里巴巴集团控股有限公司 | A kind of data compression, decompressing method, device and equipment |
Non-Patent Citations (1)
| Title |
|---|
| 郭威 ; 方宽 ; 黄慧欣 ; 林俊 ; .DM数据库中大规模数据智能自适应压缩算法.科学技术与工程.2018,(03),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110995273A (en) | 2020-04-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110995273B (en) | Data compression method, device, equipment and medium for power database | |
| CN110569214B (en) | Index construction method and device for log file and electronic equipment | |
| US20060173878A1 (en) | Efficient processing of time series data | |
| WO2021101451A1 (en) | Method and apparatus for storing data, and computer device and storage medium thereof | |
| Ali et al. | A review of digital forensics methods for JPEG file carving | |
| CN111475324A (en) | Log information analysis method and device, computer equipment and storage medium | |
| CN112115965A (en) | SVM-based passive operating system identification method, storage medium and equipment | |
| CN112328631A (en) | Production fault analysis method and device, electronic equipment and storage medium | |
| EP4042307A1 (en) | Method, system, electronic device, and storage medium for storing and collecting temperature data | |
| CN115687674A (en) | Big data demand analysis method and system serving smart cloud service platform | |
| US20240329835A1 (en) | Method and apparatus for training data compression model, and storage medium | |
| CN113297208A (en) | Data processing method and device | |
| CN109240916B (en) | Information output control method, information output control device and computer readable storage medium | |
| CN112101543A (en) | Neural network model determination method and device, electronic equipment and readable storage medium | |
| CN107169057B (en) | Method and device for detecting repeated pictures | |
| CN110032432B (en) | Example compression method and device and example decompression method and device | |
| CN114466082B (en) | Data compression and data decompression method and system and artificial intelligent AI chip | |
| CN107797898B (en) | Data acquisition method and device thereof | |
| CN112445687A (en) | Blocking detection method of computing equipment and related device | |
| CN104637496A (en) | Computer system and audio comparison method | |
| CN110119337B (en) | Data analysis method and device and server | |
| CN116189706A (en) | Data transmission method, device, electronic equipment and computer readable storage medium | |
| CN112597498A (en) | Webshell detection method, system and device and readable storage medium | |
| CN117251384B (en) | Interface automation test case generation method and system | |
| CN117851159A (en) | Time sequence data rapid compression method and system based on rounding transformation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |