CN111581220A - Storage and retrieval method, device, equipment and storage medium for time series data - Google Patents
Storage and retrieval method, device, equipment and storage medium for time series data Download PDFInfo
- Publication number
- CN111581220A CN111581220A CN202010465714.5A CN202010465714A CN111581220A CN 111581220 A CN111581220 A CN 111581220A CN 202010465714 A CN202010465714 A CN 202010465714A CN 111581220 A CN111581220 A CN 111581220A
- Authority
- CN
- China
- Prior art keywords
- attribute
- key
- query
- time
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 230000004044 response Effects 0.000 claims abstract description 7
- 238000004590 computer program Methods 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 7
- 238000010276 construction Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000003203 everyday effect Effects 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 239000003550 marker Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Economics (AREA)
- General Business, Economics & Management (AREA)
- Technology Law (AREA)
- Strategic Management (AREA)
- Marketing (AREA)
- Development Economics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a method, a device, equipment and a storage medium for storing and retrieving time series data, wherein the method comprises the following steps: constructing a keyword based on each attribute of the object, the constructed keyword comprising at least three parts: the name of the attribute, the identification of the object, a timestamp; storing the constructed key words and the attribute values of the attributes as key value pairs; and constructing one or more query keywords according to the keyword structure used for storage in response to the relevant information in the query request, and extracting corresponding values from the key-value database based on the constructed query keywords. The technical scheme of the embodiment of the invention not only can utilize the characteristics of large key value storage data volume and high query speed, but also can support flexible condition query and improve the retrieval efficiency of time series data.
Description
Technical Field
The present invention relates to the field of data processing, and more particularly, to a method, apparatus, device, and storage medium for storing and retrieving time series data.
Background
In the financial field of insurance, securities, etc., a large amount of real-time market data and financial data are generated every day, and these data are collected in time sequence, reflect the time-varying state or degree of a certain thing, phenomenon, etc., and thus may be also referred to as time series data. Data providers such as WIND currently employ relational databases to store such data. However, the conventional relational database is directed to processing of tabular data, and cannot perform efficient storage and data query on massive time-series data. The Key-Value storage system has large storage data volume and high query speed, but has a single query mode, only supports extracting a corresponding Value by a given Key, and does not support conditional query. This means that when an specific Key value is not known, all Key values need to be traversed to find the corresponding value, which is very inefficient for query retrieval of time-series data that often needs to be statistically analyzed over a given time range or value range.
Disclosure of Invention
Therefore, an object of embodiments of the present invention is to provide a method, an apparatus, a device, and a storage medium for storing and retrieving time-series data, which can utilize key-value (key-value) to store a large amount of data and have a fast query speed, and support flexible conditional query to improve the retrieval efficiency of time-series data.
The above purpose is realized by the following technical scheme:
according to a first aspect of embodiments of the present invention, there is provided a storage method for time-series data, including: in response to receiving information related to the object, extracting each attribute of the object and a corresponding attribute value thereof; constructing a keyword based on each attribute of the object, the constructed keyword comprising at least three parts: the name of the attribute, the identification of the object and a time stamp are separated by a preset separator, and the time stamp comprises the time generated by the attribute value corresponding to the attribute; and storing the constructed key word and the attribute value of the attribute as a key value pair in a key value database.
In some embodiments of the present invention, the timestamp may further include a time when the attribute value is received, which is separated by a time when the preset separator is generated from the attribute value corresponding to the attribute.
In some embodiments of the present invention, storing keys in an ordered tree structure is employed in the key-value store.
In some embodiments of the invention, the ordered tree may be a prefix tree or a suffix tree.
According to a second aspect of the embodiments of the present invention, there is provided a retrieval method for time-series data, including: receiving a query request, and constructing one or more query keywords according to related information in the query request, wherein the query keywords at least comprise three parts: the method comprises the following steps that the name of an attribute to be queried, the identification of an object to be queried and a timestamp are divided by preset separators, and the timestamp comprises the time generated by the attribute value of the attribute to be queried; and extracting corresponding values from a key-value database constructed according to the storage method of the first aspect of the embodiment of the present invention based on the constructed query keyword.
In some embodiments of the present invention, constructing one or more query keywords according to the relevant information in the query request may include constructing a query keyword for each combination of the name of the attribute, the identification of the object to which the attribute belongs, and the specified time included in the query request.
In some embodiments of the present invention, the constructing one or more query keywords according to the related information in the query request may include, for each combination of the name of the attribute contained in the query request, the identification of the object to which the attribute belongs, and a specified time period, respectively constructing two query keywords based on the start time and the end time of the specified time period.
According to a third aspect of the embodiments of the present invention, there is provided a storage device for time-series data, including attribute extraction means and data saving means. The attribute extracting device is used for responding to the received information related to the object and extracting each attribute of the object and the corresponding attribute value. The data holding means is for constructing a keyword based on each attribute of the object, the constructed keyword including at least three parts: the name of the attribute, the identification of the object and a time stamp are separated by a preset separator, and the time stamp comprises the time generated by the attribute value corresponding to the attribute; and storing the constructed key word and the attribute value of the attribute as a key value pair in a key value database.
According to a fourth aspect of the embodiments of the present invention, there is provided a retrieval system for time-series data, including keyword construction means and query means. The keyword construction device is used for receiving a query request and constructing one or more query keywords according to related information in the query request, wherein the query keywords at least comprise three parts: the method comprises the following steps that the name of an attribute to be inquired, the identification of an object to be inquired and a timestamp are divided by a preset separator, and the timestamp comprises the time generated by the attribute value of the attribute to be inquired. The query device is configured to extract a corresponding value from the key-value database constructed according to the storage method described in the first aspect of the embodiment of the present invention according to the query keyword constructed by the keyword construction device.
According to a fifth aspect of embodiments of the present invention, there is provided a computer readable storage medium having stored thereon a computer program which, when executed, implements a method as described in the first or second aspect of the embodiments above.
According to a sixth aspect of embodiments of the present invention, there is provided an electronic device comprising a processor and a memory, wherein the memory is configured to store executable instructions; the processor is configured to implement the method of the first or second aspect of the embodiments described above via execution of the executable instructions.
The technical scheme provided by the embodiment of the invention has the following beneficial effects:
each attribute of an object is used as a basic unit to store time sequence data, and a keyword and an attribute value are constructed by using an attribute name, an object identifier and a time stamp and stored as a key value pair, so that the characteristics of large data storage quantity and high query speed of the key value can be utilized, flexible condition query is supported, and the retrieval efficiency of the time sequence data is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
In the drawings:
fig. 1 shows a flow chart diagram of a storage method for time-series data according to an embodiment of the present invention.
Fig. 2 shows a schematic flow diagram of a retrieval method for time series data according to one embodiment of the invention.
Fig. 3 shows a functional block diagram of a storage device for time-series data according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail by embodiments with reference to the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention may be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations or operations have not been shown or described in detail to avoid obscuring aspects of the invention.
The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.
The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
As mentioned in the background section, in the financial field of insurance, securities, etc., a large amount of real-time market data and financial data are generated every day, and the WIND database, analyst database, market database, etc. all use relational databases to store these data. The underlying data sets from different data providers are in different formats and there is some overlap of data, both administrative and query inconvenient. The embodiment of the invention provides a storage and retrieval method for time series data, which aggregates bottom layer data of various big data suppliers to the same data platform and can provide convenient and fast data query service for upper layer application.
Fig. 1 is a flowchart illustrating a storage method for time-series data according to an embodiment of the present invention. The method mainly comprises the following steps: step S101, in response to receiving information related to an object, extracting each attribute of the object and a corresponding attribute value; step S102, constructing a keyword based on each attribute of the object, wherein the constructed keyword at least comprises three parts: the name of the attribute, the identification of the object and a time stamp are separated by a preset separator, and the time stamp comprises the time generated by the attribute value corresponding to the attribute; in step S103, the constructed keyword and the attribute value of the attribute are saved as a key value pair in the key value database. The execution subject of the method can be a server or any one or more computing devices and combinations thereof for providing processing, storage, and/or retrieval services of time series data.
More specifically, in step S101, in response to receiving information related to an object, each attribute of the object and its corresponding attribute value are extracted. The object here may be any observation or monitoring object that can generate time series data. The data related to the objects are usually sensitive to time, and a corresponding time stamp is required to be set to record the time when the data is generated when the data is recorded, collected or stored; for example stock market quote transaction data such as stocks, securities, etc., large amounts of sensor data from monitoring devices, or log data, etc. For convenience of description, the following description will be given by taking "stock" as an example of the object. Attributes of a stock may include opening price, net worth, net income, rating, industry of belongings, and the like. In this embodiment, each attribute of each object is used as a basic unit to store and retrieve data, and therefore, when information related to the object is received, each attribute of each object and its corresponding attribute value need to be extracted for subsequent processing.
In step S102, a keyword is constructed for each attribute of each object, and the constructed keyword includes at least three parts: the name of the attribute, the identification of the object, a timestamp. The identification of the object may be any identifier preset to distinguish specific objects, such as an object name, an object code, and the like. Taking stocks as an example, stock codes can be used as object identifications. For example, for the object related information of 16.7 yuan of the opening price of the stock with the stock code 021 of 5, 10 and 2018, the Key may be constructed for the attribute of "opening price" of the stock as follows: open price/021/20180510, wherein "/" is a preset separator for separating each part of the keyword, for example, also "+" can be used as the separator, i.e., "open price 021 20180510", it should be understood that this is only an example and not a limitation. The Value corresponding to the constructed Key is set to the attribute Value 16.7 corresponding to the attribute. In this embodiment, the timestamp includes the time (also referred to as the generation time) when the attribute value corresponding to the attribute is generated, i.e., the opening price of the stock is indicated to be 16.7 yuan on 5/10/2018.
In yet another embodiment, the timestamp may also include the time at which the attribute value was received (which may also be referred to as the warehousing time), i.e., the time indicating that the relevant information or data was collected or observed or stored. This is to take into account that when data is extracted from the underlying supplier for analysis, it is often not extracted in real time, but periodically or on demand. This binning time is also separated from the generation time by the above-mentioned preset separator. Of course, the generation time and the binning time may also be understood or considered as two timestamps (i.e., a first timestamp and a second timestamp, or a data generation timestamp and an observation timestamp). For example, the relevant financial data for a stock is typically published quarterly, with some time difference between the time of creation and the time of collection or observation. For example, a stock with a stock code of 021 earns in the 3 rd quarter of 2019, the data objectively results from 30/9/2019, but the collection time of the data usually lags about 1 month, so the constructed Key word Key usually needs to set two timestamps, such as "earnings/021/20190930/20191030". In the embodiment, the observation time stamp is set by considering the characteristics of the time series data, especially the stock-related time series data, and the value when the data is generated may be corrected at a later stage, so that the latest data value from a specified time needs to be extracted when the data is queried. For example, continuing the above example, assuming that the stock with the stock code 021 is corrected in the business income of the 3 rd quarter in 2019 at 11/5 th in 2019, the latest attribute value corrected in the 5 th day in 11/9 should be obtained by querying the attribute of the business income of the 3 rd quarter after 11/5 th in 2019 (for example, 6 th in 11/9). That is, for time series data, different attribute values may be obtained by querying the same attribute at different time points. Therefore, in the embodiment of the invention, besides the data generation time stamp, the observation time stamp is also set so as to improve the query accuracy of the time series data in the subsequent retrieval process.
With continued reference to fig. 1, at S103, the constructed key for each attribute of each object and the attribute value of the attribute are stored as a key-value pair, which may be saved, for example, to a key-value database for use in subsequent queries. In the preferred embodiment, the Key is stored in order in an ordered tree structure, so that the query is convenient and quick, and the memory access times in the retrieval process are reduced. Wherein the ordered tree may be in the form of a prefix tree or a suffix tree, without limitation.
By adopting the storage method of the embodiment, the key words and the attribute values are constructed by the attribute names, the object identifications and the time stamps and are used as key value pairs for storage, so that the time series data can be efficiently stored by utilizing the characteristics of large data storage quantity and high query speed of the key value pairs, the association relation among all the data is reserved, and the follow-up flexible query and retrieval of the time series data are facilitated.
Fig. 2 shows a schematic flow diagram of a retrieval method for time series data according to one embodiment of the invention. As shown in fig. 2, a query request is first received at S201, and one or more query keywords are constructed according to relevant information in the query request, where the query keyword includes at least three parts: the method comprises the following steps that the name of an attribute to be inquired, the identification of an object to be inquired and a timestamp are divided by a preset separator, and the timestamp comprises the time generated by the attribute value of the attribute to be inquired. Still taking "stock" as an example, if the object, attribute and time of a specific query are explicitly specified in the query request, corresponding query keywords may be respectively constructed for the combination of each attribute name, the identification of the object to which the attribute belongs and the specified time contained in the query request. For example, if the query request issued by the user is to query the opening price of the stock with the stock code 021 in 5/10/2018, the query keyword Key may be directly constructed as "opening price/021/20180510" below. Next, in S202, the constructed query key is used to extract a corresponding value from a key-value database constructed according to the storage method described in conjunction with fig. 1.
In one embodiment, where a data generation timestamp and an observation timestamp are set, the query request may be the latest value of a certain attribute of the object since the specified query time at a specific time. For example, taking "stock" as an example, the latest values of several attributes (indicators) of several stocks (tickers) at several specific times (dates) are queried at a specified query time (timestamp). For each ticker, date, indicator combination queried, the following two query keys, namely start key and end key, can be constructed:
initial bond: indicator < sep > pointer < sep > date < sep >0
End key: indicator < sep > ticker < sep > date < sep > timestamp, where < sep > is a preset delimiter as described above. As mentioned above, each Key (also referred to simply as a Key or Key) is stored in order in a Key-value store constructed in connection with the storage method described in fig. 1. Taking an end key as a query key word to query the key value database, and if a value corresponding to the end key exists, extracting the corresponding value; if there is no value corresponding to the end key, the value corresponding to the first valid key in the key-value database that is smaller than the end key is extracted, which is the latest value of the attribute (indicator) of the stock (ticker) at time (date) since the specified query time timestamp. Of course, it is also possible to continue to extract the values corresponding to other valid keys before the first valid key until the first valid key larger than or equal to the start key is reached, so as to obtain several values of the attribute (indicator) of the stock (ticker) at the time (date) since the specified query time timestamp. It can be seen that what is extracted in the above-described retrieval and query process is the value corresponding to the valid key in the key-value store that is located between the start key and the end key. For example, if the user needs to simulate the business income published in the 3 rd quarter (i.e., 9/30 th in 2018) of a stock with a stock code of 021 (i.e., published in 28 th in 2018) at the query time of 2018, 10/30 th in 2018, construct a start key "business income/021/20180930/0" and an end key "business income/021/20180930/20181030", query from the key-value database with the end key as a query key, and then do not have a value corresponding to the end key (i.e., a value not binned in 30 th in 10 th in the database), and then continue to locate the first valid key in the key-value database which is smaller than the end key, i.e., "business income/021/20180930/20181028"), and the value corresponding to the valid key is the data to be extracted.
In yet another embodiment, where data generation timestamps and observation timestamps are set, the query request may be a value of a query over a period of time for some attribute of the object since a specified query time. That is, when the user is making a query, the generation time of the specific data is not exactly known, but only the time period expressed in the start time and the end time is given. Still taking "stock" as an example, the values of several attributes (indicators) of several stocks (tickers) in the interval from the start time (start _ date) to the end time (end _ date) are queried at a specified query time (timestamp). For each ticker queried, the indicator combined query procedure is as follows:
step (1) constructs the first pair of start and end keys as follows:
initiation bond I: indicator < sep > ticker < sep > start _ date < sep >0
End key I: indicator < sep > marker < sep > end _ date < sep > MAX _ TIMESTAMP, wherein MAX _ TIMESTAMP is the maximum time stamp set according to requirements;
step (2) taking the first valid key which is the same as or larger than the initial key I from the key value database;
step (3) takes out its "data generation time" portion (denoted as date1) from the extracted valid key, and constructs a second pair of start and end keys based on this date 1:
initiation bond II: indicator < sep > pointer < sep > date1< sep >0
End bond II: indicator < sep > ticker < sep > date1< sep > timestamp
Wherein timestamp is the query time specified in the query request;
step (4) takes the first effective key which is the same as or smaller than the end key II from the key value database, namely the latest value of the attribute (indicator) of the stock (marker) at the time (date1) since the specified query time timestamp;
step (5) constructs the query key with its "data generation time" section date1 taken from the extracted valid key: indicator < sep > pointer < sep > date < sep > MAX _ TIMESTAMP, if the query key word is equal to or greater than the end key I, the query process is stopped; otherwise, the constructed query keyword is used as an initial key I, and the Step (2) is returned to continue to execute the query process.
For example, a user needs to simulate the revenue of the first half year of a stock 2018 with a stock code 021 queried at a specified query time 2018, 10, 30 (assuming that data for the first and second quarters are published at 2018, 4, 30 and 7, 30, respectively), a start key I "revenue/021/20180101/0" and an end key I "revenue/021/20180630/99991231" are constructed at Step (1); at Step (2), the first valid key "revenue/021/20180331/20180430" equal to or greater than the start key I is retrieved from the key-value store; the "20180331" portion of the active key is extracted at Step (3) and the start key II "revenue/021/20180331/0" and end key II "revenue/021/20180331/20181030" are constructed; the value corresponding to the first valid key which is the same as or smaller than the ending key II is taken from the key value database at Step (4), namely the latest income in the first quarter; then, constructing a query keyword 'business income/021/20180331/99991231' by using the '20180331' part of the effective key at Step (5), taking the constructed query keyword as a new round of start key I, returning to Step (2) to continue execution, and taking the first effective key 'business income/021/20180630/20180730' which is equal to or larger than the start key I from the key value database; then extracting '20180630' at Setp (3) to construct a new round of start key II 'revenue/021/20180630/0' and end key II 'revenue/021/20180630/20181030'; the value corresponding to the first valid key which is the same as or smaller than the ending key II is taken from the key value database at Step (4), namely the latest income in the second quarter; a new round of query key "revenue/021/20180630/99991231" is then constructed at Step (5) with the "20180630" portion of the active key, the query key being equal to or greater than the end key I, and the process ends.
As can be seen from the above embodiments, various conditional queries can be conveniently performed based on keywords constructed with attribute names, object identifications, and timestamps. As mentioned in the background section, the conventional Key-Value storage system has a single query mode, only supports extracting the corresponding Value by giving the Key, and does not support conditional query. Unlike the conventional retrieval method based on a key value storage system, the retrieval method of the embodiment performs retrieval based on the key value database constructed according to the storage method of the embodiment of the present application, and can conveniently and flexibly construct a conditional query by using a keyword constructed by an attribute name, an object identifier and a timestamp, which is particularly advantageous for improving the query retrieval efficiency of time series data which often needs to be statistically analyzed in a given time range or value range.
Fig. 3 is a functional block diagram of a storage device 300 for time-series data according to an embodiment of the present invention. As shown in fig. 3, the apparatus 300 includes attribute extracting means 301 and data saving means 302. Wherein the attribute extracting means 301 is configured to extract each attribute of the object and its corresponding attribute value in response to receiving the information related to the object. The data retention device 302 constructs a key based on each attribute of the object as described above in connection with FIG. 1; and storing the constructed key and the attribute value of the attribute as a key value pair. Although the block diagrams depict components in a functionally separate manner, such depiction is for illustrative purposes only. The components shown in the figures may be arbitrarily combined or separated into separate software, firmware, and/or hardware components. Moreover, regardless of how such components are combined or divided, they may execute on the same host or multiple hosts, where multiple hosts may be connected by one or more networks.
In another embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program or executable instructions are stored, and when the computer program or the executable instructions are executed, the technical solution as described in the foregoing embodiments is implemented, and the implementation principle thereof is similar, and is not described herein again. In embodiments of the present invention, the computer readable storage medium may be any tangible medium that can store data and that can be read by a computing device. Examples of computer readable storage media include hard disk drives, Network Attached Storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-R, CD-RWs, magnetic tapes, and other optical or non-optical data storage devices. The computer readable storage medium may also include computer readable media distributed over a network coupled computer system so that computer programs or instructions may be stored and executed in a distributed fashion.
In another embodiment of the present invention, an electronic device is further provided, which includes a processor and a memory, where the memory is used for storing executable instructions that can be executed by the processor, and the processor is configured to execute the executable instructions stored in the memory, and when the executable instructions are executed, the technical solution described in any one of the foregoing embodiments is implemented, and the implementation principles thereof are similar, and are not described herein again.
Reference in the specification to "various embodiments," "some embodiments," "one embodiment," or "an embodiment," etc., means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment," or the like, in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Thus, a particular feature, structure, or characteristic illustrated or described in connection with one embodiment may be combined, in whole or in part, with a feature, structure, or characteristic of one or more other embodiments without limitation, as long as the combination is not logical or operational.
The terms "comprises," "comprising," and "having," and similar referents in this specification, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may alternatively include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. The word "a" or "an" does not exclude a plurality. Additionally, the various elements of the drawings of the present application are merely schematic illustrations and are not drawn to scale.
Although the present invention has been described by the above embodiments, the present invention is not limited to the embodiments described herein, and various changes and modifications may be made without departing from the scope of the present invention.
Claims (10)
1. A storage method for time series data, comprising:
in response to receiving information related to the object, extracting each attribute of the object and a corresponding attribute value thereof;
constructing a keyword based on each attribute of the object, the constructed keyword comprising at least three parts: the name of the attribute, the identification of the object and a time stamp are separated by a preset separator, and the time stamp comprises the time generated by the attribute value corresponding to the attribute;
the constructed key and the attribute value of the attribute are stored as a key value pair.
2. The method of claim 1, wherein the timestamp further comprises a time of receipt of the attribute value separated by a time at which the predetermined separator and the attribute value corresponding to the attribute were generated.
3. The method of claim 1 or 2, wherein keys in key-value pairs are stored in an ordered tree structure.
4. The method of claim 1, wherein the ordered tree is a prefix tree or a suffix tree.
5. A retrieval method for time series data, comprising:
receiving a query request, and constructing one or more query keywords according to related information in the query request, wherein the query keywords at least comprise three parts: the method comprises the following steps that the name of an attribute to be queried, the identification of an object to be queried and a timestamp are divided by preset separators, and the timestamp comprises the time generated by the attribute value of the attribute to be queried;
extracting corresponding values from a key-value database built according to the storage method of any one of claims 1 to 4 based on the constructed query key.
6. The method of claim 5, wherein constructing one or more query keys based on the relevant information in the query request comprises constructing a query key for each combination of the name of the attribute, the identity of the object to which the attribute belongs, and the specified time included in the query request.
7. The method of claim 5, wherein constructing one or more query keys based on the relevant information in the query request comprises constructing two query keys based on the start time and the end time of a specified time period for each combination of the name of the attribute contained in the query request, the identification of the object to which the attribute belongs, and the specified time period.
8. A storage device for time series data, comprising:
attribute extraction means for extracting each attribute of an object and its corresponding attribute value in response to receiving information related to the object;
data holding means for constructing a keyword based on each attribute of the object, the constructed keyword including at least three parts: the name of the attribute, the identification of the object and a time stamp are separated by a preset separator, and the time stamp comprises the time generated by the attribute value corresponding to the attribute; and storing the constructed key and the attribute value of the attribute as a key value pair.
9. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which program, when executed, carries out the method of any one of claims 1-7.
10. An electronic device comprising a processor and a memory, wherein the memory is configured to store executable instructions; the processor is configured to implement the method of any of claims 1-7 via execution of the executable instructions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010465714.5A CN111581220A (en) | 2020-05-28 | 2020-05-28 | Storage and retrieval method, device, equipment and storage medium for time series data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010465714.5A CN111581220A (en) | 2020-05-28 | 2020-05-28 | Storage and retrieval method, device, equipment and storage medium for time series data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111581220A true CN111581220A (en) | 2020-08-25 |
Family
ID=72127085
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010465714.5A Pending CN111581220A (en) | 2020-05-28 | 2020-05-28 | Storage and retrieval method, device, equipment and storage medium for time series data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111581220A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112307051A (en) * | 2020-10-10 | 2021-02-02 | 蔚来汽车科技(安徽)有限公司 | Data query method, device, medium and data processing system |
| CN114647618A (en) * | 2020-12-18 | 2022-06-21 | 南京中兴新软件有限责任公司 | Signaling data query method, signaling data index database construction method and server |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126383A (en) * | 2016-06-01 | 2016-11-16 | 杭州华三通信技术有限公司 | A kind of log processing method and device |
| CN107066511A (en) * | 2017-01-20 | 2017-08-18 | 龚杰 | A kind of Distributed Time sequence service system of gis and method |
| CN108256088A (en) * | 2018-01-23 | 2018-07-06 | 清华大学 | A kind of storage method and system of the time series data based on key value database |
| CN109684374A (en) * | 2018-11-28 | 2019-04-26 | 海南电网有限责任公司信息通信分公司 | A kind of extracting method and device of the key-value pair of time series data |
| CN110046183A (en) * | 2019-04-16 | 2019-07-23 | 北京易沃特科技有限公司 | A kind of time series data polymerization search method, equipment and medium |
| CN110888849A (en) * | 2019-11-06 | 2020-03-17 | 国网上海市电力公司 | Online log analysis method and system and electronic terminal equipment thereof |
| US20200089798A1 (en) * | 2018-09-17 | 2020-03-19 | Paul R Ganichot | High volume-velocity time series data ingestion, analysis and reporting method and system |
| CN111090794A (en) * | 2019-11-07 | 2020-05-01 | 远景智能国际私人投资有限公司 | Meteorological data query method, device and storage medium |
| CN111125089A (en) * | 2019-11-05 | 2020-05-08 | 远景智能国际私人投资有限公司 | Time sequence data storage method, device, server and storage medium |
-
2020
- 2020-05-28 CN CN202010465714.5A patent/CN111581220A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126383A (en) * | 2016-06-01 | 2016-11-16 | 杭州华三通信技术有限公司 | A kind of log processing method and device |
| CN107066511A (en) * | 2017-01-20 | 2017-08-18 | 龚杰 | A kind of Distributed Time sequence service system of gis and method |
| CN108256088A (en) * | 2018-01-23 | 2018-07-06 | 清华大学 | A kind of storage method and system of the time series data based on key value database |
| US20200089798A1 (en) * | 2018-09-17 | 2020-03-19 | Paul R Ganichot | High volume-velocity time series data ingestion, analysis and reporting method and system |
| CN109684374A (en) * | 2018-11-28 | 2019-04-26 | 海南电网有限责任公司信息通信分公司 | A kind of extracting method and device of the key-value pair of time series data |
| CN110046183A (en) * | 2019-04-16 | 2019-07-23 | 北京易沃特科技有限公司 | A kind of time series data polymerization search method, equipment and medium |
| CN111125089A (en) * | 2019-11-05 | 2020-05-08 | 远景智能国际私人投资有限公司 | Time sequence data storage method, device, server and storage medium |
| CN110888849A (en) * | 2019-11-06 | 2020-03-17 | 国网上海市电力公司 | Online log analysis method and system and electronic terminal equipment thereof |
| CN111090794A (en) * | 2019-11-07 | 2020-05-01 | 远景智能国际私人投资有限公司 | Meteorological data query method, device and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112307051A (en) * | 2020-10-10 | 2021-02-02 | 蔚来汽车科技(安徽)有限公司 | Data query method, device, medium and data processing system |
| WO2022073298A1 (en) * | 2020-10-10 | 2022-04-14 | 蔚来汽车科技(安徽)有限公司 | Data query method and device, medium, and data processing system |
| CN114647618A (en) * | 2020-12-18 | 2022-06-21 | 南京中兴新软件有限责任公司 | Signaling data query method, signaling data index database construction method and server |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9262767B2 (en) | Systems and methods for generating statistics from search engine query logs | |
| CN103020281B (en) | A kind of data storage and retrieval method based on spatial data numerical index | |
| CN104424258B (en) | Multidimensional data query method, query server, column storage server and system | |
| RU2691595C2 (en) | Constructed data stream for improved event processing | |
| CN108647357B (en) | Data query method and device | |
| CN111008265A (en) | Enterprise information searching method and device | |
| CN111881011A (en) | Log management method, platform, server and storage medium | |
| US20230126706A1 (en) | Information traceability method and system based on blockchain | |
| CN113849702B (en) | Method and device for determining target data, electronic equipment and storage medium | |
| US11210312B2 (en) | Storing data items and identifying stored data items | |
| CN108319608A (en) | The method, apparatus and system of access log storage inquiry | |
| US20060093109A1 (en) | Methods, systems, and computer-readable mediums for indexing and rapidly searching data records | |
| CN113032420A (en) | Data query method and device and server | |
| CN107783974B (en) | Data processing system and method | |
| CN111581220A (en) | Storage and retrieval method, device, equipment and storage medium for time series data | |
| JP2008515061A (en) | A method for searching data elements on the web using conceptual and contextual metadata search engines | |
| CN101963993A (en) | Method for fast searching database sheet table record | |
| CN112307318A (en) | Content publishing method, system and device | |
| US20120109875A1 (en) | Organization of data mart using clustered key | |
| CN111680072A (en) | Social information data-based partitioning system and method | |
| CN117056477A (en) | Case data retrieval method, device, equipment and readable storage medium | |
| US8572041B2 (en) | Representing records | |
| CN107238764B (en) | The electric energy quality monitoring method and device of electric system | |
| CN113282694B (en) | Geographic entity database user customization method, system and storage medium | |
| CN112131215B (en) | Bottom-up database information acquisition method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200825 |
|
| RJ01 | Rejection of invention patent application after publication |