CN111125171A - Monitoring data access method, device, equipment and readable storage medium - Google Patents
Monitoring data access method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN111125171A CN111125171A CN201911332558.9A CN201911332558A CN111125171A CN 111125171 A CN111125171 A CN 111125171A CN 201911332558 A CN201911332558 A CN 201911332558A CN 111125171 A CN111125171 A CN 111125171A
- Authority
- CN
- China
- Prior art keywords
- data
- monitoring data
- file
- monitoring
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a monitoring data access method, a device, equipment and a readable storage medium, wherein the method comprises the following steps: the target cluster periodically collects monitoring data of each node; storing the monitoring data into a data file corresponding to the date to which the current timestamp belongs, and acquiring the initial position of the monitoring data of each node in the data file; writing the initial position of the monitoring data into a retrieval file corresponding to the data file, and updating the storage rule of the monitoring data of each node in the retrieval file; and when a data query request is received, reading specified monitoring data corresponding to the data query request from the data file by using the monitoring data storage rule and the monitoring data initial position. In the method, for each storage of the monitoring data, only one data file is needed to be written with the monitoring data, and the retrieval information is recorded in the retrieval file; when the monitoring data is inquired or read each time, only the corresponding data file and the corresponding retrieval file need to be accessed, and the time consumption of data inquiry can be reduced.
Description
Technical Field
The present invention relates to the field of monitoring technologies, and in particular, to a method, an apparatus, a device, and a readable storage medium for accessing monitoring data.
Background
When nodes (each controller is a node of the cluster, and a unique configuration node exists at the same time) of a cluster (the double controls form an IO group, and the multiple double controls form the cluster through an FC network) run actually, each node needs to monitor component objects such as voltage and temperature of the component of the node, and the like, and is used for closed-loop control due to the fact that field conditions are complex and changeable; the cluster needs to periodically collect the monitoring data of each node, store the monitoring data as curve data and realize the monitoring data unit supporting the rapid retrieval of the specific time of the designated node.
In practical application, the number of each node under each cluster is different, but the maximum number is generally limited; due to the difference of node hardware, the monitored component objects of each node are different, so that node curve data units stored in the cluster are inconsistent; the cluster stores and collects the monitoring data of each node according to the configuration parameter period and stores the monitoring data, and the storage time interval is flexibly configured. When the cluster starts to collect the monitoring data of each node, the received nodes are unordered, and the data of each received node is stored in time to prevent loss. And the rapid retrieval of the monitoring data units at a specific moment of a specified node is supported.
At present, monitoring data of clusters are stored in the same directory, each monitoring node corresponds to one folder, and monitoring curve data of one component object is recorded in each file in each folder, that is, if there are a nodes and the number of the component objects corresponding to each node is at least B, there are a × B files in the directory, and the number of the files is huge. When storing the monitoring data, the monitoring data corresponding to different monitoring items of each node need to be stored in different files respectively; when searching for all the node monitoring data within a certain time, as the data needs to be stored under the file of the monitoring item corresponding to each node, each file is traversed and accessed, and the monitoring data corresponding to the time is searched out, the time spent on inquiring and reading is long, and a large amount of computing resources are occupied.
In summary, how to effectively solve the problems of monitoring data access and the like is a technical problem that needs to be solved urgently by those skilled in the art at present.
Disclosure of Invention
The invention aims to provide a monitoring data access method, a monitoring data access device, monitoring data access equipment and a readable storage medium, which are used for changing a monitoring data storage structure and facilitating quick reading and storage of monitoring data.
In order to solve the technical problems, the invention provides the following technical scheme:
a method for monitoring data access management, comprising:
the target cluster periodically collects monitoring data of each node;
storing the monitoring data into a data file corresponding to the date to which the current timestamp belongs, and acquiring the initial position of the monitoring data of each node in the data file;
writing the initial position of the monitoring data into a retrieval file corresponding to the data file, and updating the storage rule of the monitoring data of each node in the retrieval file;
and when a data query request is received, reading specified monitoring data corresponding to the data query request from the data file by using the monitoring data storage rule and the monitoring data initial position.
Preferably, one of said data files and one of said search files constitute a file pair, one of said file pairs corresponding to one of said dates, further comprising:
and performing clipping management on the file pair from a storage time dimension and/or a storage space dimension.
Preferably, the reading of the data query request from the data file by using the monitoring data storage rule and the monitoring data start position includes:
and when the data query request is used for querying all the node monitoring data of the specified date, reading the data file corresponding to the specified date.
Preferably, writing the monitoring data start position into a retrieval file corresponding to the data file includes:
and writing the current timestamp and the monitoring data starting position into the retrieval file.
Preferably, the method further comprises the following steps:
when the time correction is carried out, the data across days in the time error data are cut by taking the days as units; and updating the monitoring data storage rule for the non-cross-day data in the time-staggered data.
Preferably, the reading of the data query request from the data file by using the monitoring data storage rule and the monitoring data start position includes:
analyzing the data query request to obtain a target date, a target timestamp, a target node and a target monitoring item;
determining a corresponding target data file and a corresponding target retrieval file by using the target date;
determining the storage positions of the designated monitoring data corresponding to the target timestamp, the target node and the target monitoring item in the data file by combining the monitoring data storage rule and the monitoring data initial position;
and reading the specified monitoring data stored in the storage position in the data file.
Preferably, determining the storage locations of the designated monitoring data corresponding to the target timestamp, the target node, and the target monitoring item in the data file by combining the monitoring data storage rule and the monitoring data starting location includes:
retrieving a target monitoring data starting position corresponding to the target timestamp by using a bisection method;
screening out a target monitoring data storage rule corresponding to the target node;
and determining the storage position based on the initial position of the target monitoring data and the storage rule of the target monitoring data.
A surveillance data access device, comprising:
the monitoring data receiving module is used for collecting the monitoring data of each node periodically by the target cluster;
the monitoring data storage module is used for storing the monitoring data into a data file corresponding to the date to which the current timestamp belongs and acquiring the initial position of the monitoring data of each node in the data file;
the retrieval information storage module is used for writing the initial position of the monitoring data into a retrieval file corresponding to the data file and updating the storage rule of the monitoring data of each node in the retrieval file;
and the monitoring data reading module is used for reading the specified monitoring data corresponding to the data query request from the data file by utilizing the monitoring data storage rule and the monitoring data initial position when the data query request is received.
A monitoring data access device comprising:
a memory for storing a computer program;
and the processor is used for realizing the steps of the monitoring data access method when executing the computer program.
A readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the above-described method for monitoring data access.
By applying the method provided by the embodiment of the invention, the target cluster periodically collects the monitoring data of each node; storing the monitoring data into a data file corresponding to the date to which the current timestamp belongs, and acquiring the initial position of the monitoring data of each node in the data file; writing the initial position of the monitoring data into a retrieval file corresponding to the data file, and updating the storage rule of the monitoring data of each node in the retrieval file; and when a data query request is received, reading specified monitoring data corresponding to the data query request from the data file by using the monitoring data storage rule and the monitoring data initial position.
In the method, the monitoring data of each node in the target cluster are stored into the same data file according to the date, the initial position of the monitoring data of each node is recorded in a retrieval file, and the storage rule of the monitoring data of each node is recorded. Therefore, when a data query request is received, the storage position of the specified monitoring data corresponding to the data query request stored in the data file can be determined only by reading the monitoring data storage rule and the monitoring data initial position in the retrieval file, and then the specified monitoring data is directly read from the data file. Therefore, in the method, for each storage of the monitoring data, the monitoring data is only written into one data file, and the retrieval information is recorded in the retrieval file; when the monitoring data is inquired or read each time, only the corresponding data file and the corresponding retrieval file need to be accessed, a large number of files do not need to be inquired in a traversing mode, and the time consumption of data inquiry can be reduced.
Accordingly, embodiments of the present invention further provide a monitoring data access device, a device and a readable storage medium corresponding to the above monitoring data access method, which have the above technical effects and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart illustrating an embodiment of a method for monitoring data access;
FIG. 2 is a schematic diagram of a logical structure of a data file according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a logic structure of a search document according to an embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating a monitoring data reading according to an embodiment of the present invention;
FIG. 5 is a diagram illustrating a correspondence between a data file and a search file according to an embodiment of the present invention;
FIG. 6 is a schematic diagram illustrating a process for determining a storage location of monitored data according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a monitor data access device according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of a monitor data access device according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of a monitoring data access device according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The first embodiment is as follows:
referring to fig. 1, fig. 1 is a flowchart illustrating a method for monitoring data access according to an embodiment of the present invention, the method including the following steps:
s101, collecting monitoring data of each node in a target cluster period.
The target cluster may be any cluster which includes a plurality of nodes and needs to monitor component objects of the nodes.
The monitoring data of the nodes can be set into corresponding component objects according to specific monitoring requirements. The component objects may generally be temperature, voltage, etc.
Since the monitoring of each node in the target cluster is periodic, the target cluster periodically collects the monitoring data of each node. That is, the monitoring data of each node may be collected after the periodic collection task is started. Since the monitoring process is periodically continued, the monitoring data of each component object of each node may constitute monitoring curve data. That is, the monitoring data herein may include monitoring curve data of the component object of each node.
S102, storing the monitoring data into a data file corresponding to the date to which the current timestamp belongs, and acquiring the initial position of the monitoring data of each node in the data file.
Wherein one day corresponds to 1 day, i.e. 24 hours. In particular, for convenience of management, corresponding data files may be set for different dates. Specifically, one date may be set to correspond to one data file (i.e., one data file for one day), one date may be set to correspond to multiple data files (e.g., one data file for half a day, or a divided date with finer granularity), and multiple dates may also be set to correspond to the same data file (e.g., one data file for two days, or one data file for one week). In practical application, it may be preferable to set one data file for each day, so as to perform processing such as quick viewing, deleting or migrating on the monitoring data based on the day as the search keyword.
After receiving the monitoring data, the monitoring data can be stored in the data file corresponding to the current timestamp based on the current timestamp, and the initial position of the monitoring data of each node in the data file is obtained.
When the monitoring data is stored, the monitoring data is stored according to the logic structure of the data file. In particular, the logical structure of the data file can be seen in fig. 2. For N time stamps, namely all the nodes are collected for N times in a period, corresponding monitoring data can be stored from the time stamp 1 to the time stamp N in sequence, and for N nodes, the monitoring data can be stored from the curve data 1 to the curve data N in sequence. For the convenience of reading management, the data occupation size of each node in the same group of time stamp points is different, but the size of a specific node can be fixed.
After the storage of the monitoring data is finished, the initial position of the monitoring data of each node in the data file needs to be recorded. Specifically, the storage start position of each timestamp data of each node is recorded.
S103, writing the initial position of the monitoring data into a retrieval file corresponding to the data file, and updating the storage rule of the monitoring data of each node in the retrieval file.
The data file and the retrieval file have a corresponding relation.
And recording the initial position of the monitoring data stored every time and the monitoring data storage rule of each node recorded in the data file in the retrieval file. That is, the monitoring data recorded in the data file can be analyzed by reading the search file, and the monitoring data in the data file is searched.
Preferably, in order to facilitate reading of the monitoring data corresponding to the specified time, the current timestamp may also be written into the retrieval file when the starting position of the monitoring data is written. That is, the current timestamp and the start position of the monitored data are written into the retrieval file.
The logical structure of the search file can be seen in fig. 3. Wherein, TOP information is the storage rule of monitoring data; pos value is the initial position of monitoring data; MAX is the total number of nodes. I.e. retrieving the file may comprise:
the current storage timestamp + reservation + sequentially stores TOP information of each node; wherein, each node TOP information comprises the storage unit size of the node in a data file (named date. dat, same dat file), the node type, the position of the node to be ready for updating the next time stamp Pos value, and the reservation, the occupied size of each node TOP information is fixed, and the unused node can be set to 0. The curve point timestamp can occupy 4 BYTEs (BYTE), the Pos value of each node in the dat file can occupy 4 BYTEs, the Pos values of all nodes in the same group of timestamps are sequentially arranged and stored, unused node data is set to be 0, and the size occupied by the same group of timestamps is fixed to be (1+ MAX) × 4 BYTEs.
And S104, when the data query request is received, reading the specified monitoring data corresponding to the data query request from the data file by using the monitoring data storage rule and the monitoring data initial position.
When a data query request is received, specified monitoring data corresponding to the data query request can be read from the data file by reading the monitoring data storage rule and the monitoring data initial position in the retrieval file.
Reading the data query request from the data file to specify the monitoring data may specifically include:
step one, analyzing a data query request to obtain a target date, a target timestamp, a target node and a target monitoring item;
determining a corresponding target data file and a corresponding target retrieval file by using a target date;
thirdly, determining the storage positions of the designated monitoring data corresponding to the target timestamp, the target node and the target monitoring item in the data file by combining the monitoring data storage rule and the monitoring data initial position;
and step four, reading the specified monitoring data stored in the storage position in the data file.
For convenience of description, the above steps will be described below with reference to fig. 4 and 5. Fig. 4 is a schematic diagram illustrating reading of monitoring data according to an embodiment of the present invention, and fig. 5 is a schematic diagram illustrating a correspondence relationship between a data file and a search file according to an embodiment of the present invention; in fig. 4, search is a target search file, and dat is a target data file; in fig. 5, the data D to be read is the designated monitoring data; pos _ P specifies the storage location of the monitored data in the data file.
Preferably, in order to shorten the time for determining the storage position, the search file can be searched by adopting a dichotomy method. Namely, the third step may specifically include:
step 1, retrieving a target monitoring data initial position corresponding to a target timestamp by using a dichotomy;
step 2, screening out target monitoring data storage rules corresponding to the target nodes;
and 3, determining a storage position based on the initial position of the target monitoring data and the storage rule of the target monitoring data.
For convenience of explanation, the above-described dichotomy determination of storage locations will be described below with reference to fig. 6. Fig. 6 is a schematic diagram illustrating a process of determining a storage location of monitoring data according to an embodiment of the present invention.
Reading a start time stamp (Tm _ start), an end time stamp (Tm _ end), a start position (Pos _ start) and an end position (Pos _ end) from a target retrieval file; in order to avoid the confusion of the timestamp, whether the ending timestamp is larger than the starting timestamp can be judged firstly; if yes, finishing searching input failure information; if not, the search continues. And then searching whether the storage position (Tm _ search) is between Tm _ start and Tm _ end, if not, finishing searching input failure information, if so, calculating a binary position (Pos _ search), and repeatedly judging the binary position until the storage position is found.
Preferably, when one data file and one search file constitute one file pair, one file pair corresponds to one date, and when the data query requests that data be monitored for all nodes querying for the specified date, the data file corresponding to the specified date may be read. Therefore, the monitoring data corresponding to the data query request can be read more quickly.
In the present application, the search file and the data file are paired, and the search file and the data file are in one-to-one correspondence. Preferably, when a data file and a search file form a file pair, a file pair corresponds to a date. In order to avoid occupation of storage resources due to overlarge monitoring data, the file pair can be cut and managed from a storage time dimension and/or a storage space dimension. That is, when the monitoring data is deleted or cut, both the data file and the search file corresponding thereto are deleted or cut. Specifically, the old file pair may be deleted/clipped from the storage time dimension, for example, only monitoring data of the last M days is retained (M may be set according to specific requirements, for example, M is 60); file pairs may also be deleted/pruned from the storage space dimension, for example, when the size of the target cluster monitoring directory (including data files and search files) exceeds a preset threshold (which may be set according to specific requirements, such as 10G), a specified number of old file pairs may be deleted.
By applying the method provided by the embodiment of the invention, the target cluster periodically collects the monitoring data of each node; storing the monitoring data into a data file corresponding to the date to which the current timestamp belongs, and acquiring the initial position of the monitoring data of each node in the data file; writing the initial position of the monitoring data into a retrieval file corresponding to the data file, and updating the storage rule of the monitoring data of each node in the retrieval file; and when a data query request is received, reading specified monitoring data corresponding to the data query request from the data file by using the monitoring data storage rule and the monitoring data initial position.
In the method, the monitoring data of each node in the target cluster are stored into the same data file according to the date, the initial position of the monitoring data of each node is recorded in a retrieval file, and the storage rule of the monitoring data of each node is recorded. Therefore, when a data query request is received, the storage position of the specified monitoring data corresponding to the data query request stored in the data file can be determined only by reading the monitoring data storage rule and the monitoring data initial position in the retrieval file, and then the specified monitoring data is directly read from the data file. Therefore, in the method, for each storage of the monitoring data, the monitoring data is only written into one data file, and the retrieval information is recorded in the retrieval file; when the monitoring data is inquired or read each time, only the corresponding data file and the corresponding retrieval file need to be accessed, a large number of files do not need to be inquired in a traversing mode, and the time consumption of data inquiry can be reduced.
It should be noted that, based on the above embodiments, the embodiments of the present invention also provide corresponding improvements. In the preferred/improved embodiment, the same steps as those in the above embodiment or corresponding steps may be referred to each other, and corresponding advantageous effects may also be referred to each other, which are not described in detail in the preferred/improved embodiment herein.
Preferably, based on the monitoring data access management method shown in the first embodiment, the backward timing process can be further simplified. That is, when the time correction is performed, the data across days in the time error data is cut in a day unit; and for non-day-crossing data in the time-staggered data, updating the storage rule of the monitoring data. For example, when the date of the target cluster is ahead or behind 1 day or several days, the monitoring file and the retrieval file are only required to be cut by taking the day as a unit, and the timestamp in the monitoring data storage rule can be adjusted corresponding to the non-cross-day data. For how to determine the wrong time data, reference may be made to a wrong time data processing flow in the existing backward timing processing, which is not described in detail herein.
Example two:
corresponding to the above method embodiments, the present invention further provides a monitoring data access apparatus, and the monitoring data access apparatus described below and the monitoring data access method described above may be referred to correspondingly.
Referring to fig. 7, the apparatus includes the following modules:
the monitoring data receiving module 101 is used for collecting monitoring data of each node periodically by the target cluster;
the monitoring data storage module 102 is configured to store the monitoring data into a data file corresponding to a date to which the current timestamp belongs, and acquire a starting position of the monitoring data of each node in the data file;
the retrieval information storage module 103 is configured to write the initial position of the monitoring data into a retrieval file corresponding to the data file, and update a storage rule of the monitoring data of each node in the retrieval file;
and the monitoring data reading module 104 is configured to, when a data query request is received, read specified monitoring data corresponding to the data query request from the data file by using the monitoring data storage rule and the monitoring data start position.
By applying the device provided by the embodiment of the invention, the target cluster periodically collects the monitoring data of each node; storing the monitoring data into a data file corresponding to the date to which the current timestamp belongs, and acquiring the initial position of the monitoring data of each node in the data file; writing the initial position of the monitoring data into a retrieval file corresponding to the data file, and updating the storage rule of the monitoring data of each node in the retrieval file; and when a data query request is received, reading specified monitoring data corresponding to the data query request from the data file by using the monitoring data storage rule and the monitoring data initial position.
In the device, the monitoring data of each node in the target cluster are stored into the same data file according to the date, the initial position of the monitoring data of each node is recorded in the retrieval file, and the storage rule of the monitoring data of each node is recorded. Therefore, when a data query request is received, the storage position of the specified monitoring data corresponding to the data query request stored in the data file can be determined only by reading the monitoring data storage rule and the monitoring data initial position in the retrieval file, and then the specified monitoring data is directly read from the data file. Therefore, in the device, for each storage of the monitoring data, the monitoring data is only needed to be written into one data file, and the retrieval information is recorded in the retrieval file; when the monitoring data is inquired or read each time, only the corresponding data file and the corresponding retrieval file need to be accessed, a large number of files do not need to be inquired in a traversing mode, and the time consumption of data inquiry can be reduced.
In one embodiment of the present invention, a data file and a search file form a file pair, and a file pair corresponds to a date, further comprising:
and the clipping management module is used for carrying out clipping management on the file pair from the storage time dimension and/or the storage space dimension.
In an embodiment of the present invention, the monitoring data reading module 104 is specifically configured to, when the data query request queries all the node monitoring data on the specified date, read the data file corresponding to the specified date.
In an embodiment of the present invention, the monitoring data storage module 102 is specifically configured to write the current timestamp and the starting position of the monitoring data into the search file.
In one embodiment of the present invention, the method further comprises:
the backward timing processing module is used for cutting off the cross-day data in the time error data by taking a day as a unit during backward timing processing; and for non-day-crossing data in the time-staggered data, updating the storage rule of the monitoring data.
In an embodiment of the present invention, the monitoring data reading module 104 includes:
the analysis unit is used for analyzing the data query request to obtain a target date, a target timestamp, a target node and a target monitoring item;
the file determining unit is used for determining a corresponding target data file and a corresponding target retrieval file by using a target date;
the storage position determining unit is used for determining the storage positions of the specified monitoring data corresponding to the target timestamp, the target node and the target monitoring item in the data file by combining the monitoring data storage rule and the monitoring data initial position;
and the data reading unit is used for reading the specified monitoring data stored in the storage position in the data file.
In a specific embodiment of the present invention, the storage location determining unit is specifically configured to retrieve a starting location of the target monitoring data corresponding to the target timestamp by using a bisection method; screening out a target monitoring data storage rule corresponding to the target node; and determining the storage position based on the initial position of the target monitoring data and the storage rule of the target monitoring data.
Example three:
corresponding to the above method embodiment, an embodiment of the present invention further provides a monitoring data access device, and a monitoring data access device described below and a monitoring data access method described above may be referred to in correspondence.
Referring to fig. 8, the monitoring data access apparatus includes:
a memory D1 for storing computer programs;
the processor D2 is configured to implement the steps of the monitoring data access method of the above-mentioned method embodiment when executing the computer program.
Specifically, referring to fig. 9, a specific structural diagram of a monitoring data access device provided in this embodiment is shown, where the monitoring data access device may generate a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 322 (e.g., one or more processors) and a memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) storing an application 342 or data 344. Memory 332 and storage media 330 may be, among other things, transient storage or persistent storage. The program stored on the storage medium 330 may include one or more modules (not shown), each of which may include a series of instructions operating on a data processing device. Still further, the central processor 322 may be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the monitoring data access device 301.
The monitoring data access apparatus 301 may also include one or more power sources 326, one or more wired or wireless network interfaces 350, one or more input-output interfaces 358, and/or one or more operating systems 341. Such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
The steps in the above described monitoring data access method may be implemented by the structure of a monitoring data access device.
Example four:
corresponding to the above method embodiment, the embodiment of the present invention further provides a readable storage medium, and a readable storage medium described below and a monitoring data access method described above may be referred to in correspondence.
A readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method for monitoring data access of the above-mentioned method embodiments.
The readable storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various other readable storage media capable of storing program codes.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
Claims (10)
1. A method for monitoring data access, comprising:
the target cluster periodically collects monitoring data of each node;
storing the monitoring data into a data file corresponding to the date to which the current timestamp belongs, and acquiring the initial position of the monitoring data of each node in the data file;
writing the initial position of the monitoring data into a retrieval file corresponding to the data file, and updating the storage rule of the monitoring data of each node in the retrieval file;
and when a data query request is received, reading specified monitoring data corresponding to the data query request from the data file by using the monitoring data storage rule and the monitoring data initial position.
2. The method for monitoring data access of claim 1, wherein one of said data files and one of said search files form a file pair, one of said file pairs corresponding to one of said dates, further comprising:
and performing clipping management on the file pair from a storage time dimension and/or a storage space dimension.
3. The method for accessing monitored data according to claim 2, wherein reading the data query request specifying monitored data from the data file using the monitored data storage rule and the monitored data start position comprises:
and when the data query request is used for querying all the node monitoring data of the specified date, reading the data file corresponding to the specified date.
4. The method according to claim 1, wherein writing the start position of the monitored data into the corresponding search file of the data file comprises:
and writing the current timestamp and the monitoring data starting position into the retrieval file.
5. The method for monitoring data access according to any one of claims 1 to 4, further comprising:
when the time correction is carried out, the data across days in the time error data are cut by taking the days as units; and updating the monitoring data storage rule for the non-cross-day data in the time-staggered data.
6. The method for accessing monitored data according to claim 4, wherein reading the data query request specifying monitored data from the data file using the monitored data storage rule and the monitored data start position comprises:
analyzing the data query request to obtain a target date, a target timestamp, a target node and a target monitoring item;
determining a corresponding target data file and a corresponding target retrieval file by using the target date;
determining the storage positions of the designated monitoring data corresponding to the target timestamp, the target node and the target monitoring item in the data file by combining the monitoring data storage rule and the monitoring data initial position;
and reading the specified monitoring data stored in the storage position in the data file.
7. The method for accessing monitoring data according to claim 6, wherein determining, in combination with the monitoring data storage rule and the monitoring data start position, a storage position of the designated monitoring data corresponding to the target timestamp, the target node, and the target monitoring item in the data file comprises:
retrieving a target monitoring data starting position corresponding to the target timestamp by using a bisection method;
screening out a target monitoring data storage rule corresponding to the target node;
and determining the storage position based on the initial position of the target monitoring data and the storage rule of the target monitoring data.
8. A surveillance data access device, comprising:
the monitoring data receiving module is used for collecting the monitoring data of each node periodically by the target cluster;
the monitoring data storage module is used for storing the monitoring data into a data file corresponding to the date to which the current timestamp belongs and acquiring the initial position of the monitoring data of each node in the data file;
the retrieval information storage module is used for writing the initial position of the monitoring data into a retrieval file corresponding to the data file and updating the storage rule of the monitoring data of each node in the retrieval file;
and the monitoring data reading module is used for reading the specified monitoring data corresponding to the data query request from the data file by utilizing the monitoring data storage rule and the monitoring data initial position when the data query request is received.
9. A monitoring data access device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the method of monitoring data access according to any one of claims 1 to 7 when executing said computer program.
10. A readable storage medium, having stored thereon a computer program which, when executed by a processor, carries out the steps of the method of monitoring data access according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911332558.9A CN111125171A (en) | 2019-12-22 | 2019-12-22 | Monitoring data access method, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911332558.9A CN111125171A (en) | 2019-12-22 | 2019-12-22 | Monitoring data access method, device, equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111125171A true CN111125171A (en) | 2020-05-08 |
Family
ID=70501377
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911332558.9A Withdrawn CN111125171A (en) | 2019-12-22 | 2019-12-22 | Monitoring data access method, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111125171A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000619A (en) * | 2020-08-21 | 2020-11-27 | 杭州安恒信息技术股份有限公司 | Time sequence data storage method, device, equipment and readable storage medium |
| CN114356853A (en) * | 2020-10-12 | 2022-04-15 | 南宁富桂精密工业有限公司 | Data access method and system thereof |
| CN116610256A (en) * | 2023-04-27 | 2023-08-18 | 上海勘测设计研究院有限公司 | Automatic port water condition data reading and writing method, system, medium and equipment |
| WO2023173874A1 (en) * | 2022-03-14 | 2023-09-21 | 苏州浪潮智能科技有限公司 | Device fault diagnosis system and device fault diagnosis methods |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902797A (en) * | 2012-10-11 | 2013-01-30 | 福建亿同世纪软件有限公司 | Method for storing and retrieving real-time monitored data of large-volume equipment |
| CN103500173A (en) * | 2013-09-03 | 2014-01-08 | 北京泰乐德信息技术有限公司 | Method for inquiring rail transit monitoring data |
| US9043327B1 (en) * | 2013-06-13 | 2015-05-26 | Amazon Technologies, Inc. | Performing flexible pivot querying of monitoring data using a multi-tenant monitoring system |
-
2019
- 2019-12-22 CN CN201911332558.9A patent/CN111125171A/en not_active Withdrawn
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902797A (en) * | 2012-10-11 | 2013-01-30 | 福建亿同世纪软件有限公司 | Method for storing and retrieving real-time monitored data of large-volume equipment |
| US9043327B1 (en) * | 2013-06-13 | 2015-05-26 | Amazon Technologies, Inc. | Performing flexible pivot querying of monitoring data using a multi-tenant monitoring system |
| CN103500173A (en) * | 2013-09-03 | 2014-01-08 | 北京泰乐德信息技术有限公司 | Method for inquiring rail transit monitoring data |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000619A (en) * | 2020-08-21 | 2020-11-27 | 杭州安恒信息技术股份有限公司 | Time sequence data storage method, device, equipment and readable storage medium |
| CN114356853A (en) * | 2020-10-12 | 2022-04-15 | 南宁富桂精密工业有限公司 | Data access method and system thereof |
| WO2023173874A1 (en) * | 2022-03-14 | 2023-09-21 | 苏州浪潮智能科技有限公司 | Device fault diagnosis system and device fault diagnosis methods |
| CN116610256A (en) * | 2023-04-27 | 2023-08-18 | 上海勘测设计研究院有限公司 | Automatic port water condition data reading and writing method, system, medium and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111125171A (en) | Monitoring data access method, device, equipment and readable storage medium | |
| US10459898B2 (en) | Configurable-capacity time-series tables | |
| US7117294B1 (en) | Method and system for archiving and compacting data in a data storage array | |
| KR102031588B1 (en) | Method and system for implementing index when saving file | |
| US20110161406A1 (en) | Storage management system, storage hierarchy management method, and management server | |
| CN103605585A (en) | Intelligent backup method based on data discovery | |
| CN110727406B (en) | Data storage scheduling method and device | |
| EP3788505B1 (en) | Storing data items and identifying stored data items | |
| EP3789883A1 (en) | Storage fragment managing method and terminal | |
| CN111930716A (en) | Database capacity expansion method, device and system | |
| CN113760847A (en) | Log data processing method, device, equipment and storage medium | |
| US10642530B2 (en) | Global occupancy aggregator for global garbage collection scheduling | |
| US10891266B2 (en) | File handling in a hierarchical storage system | |
| CN110580253B (en) | Time sequence data set loading method and device, storage medium and electronic equipment | |
| CN114297196B (en) | Metadata storage method and device, electronic equipment and storage medium | |
| JP5956064B2 (en) | Computer system, data management method, and computer | |
| CN118152357A (en) | Data storage method and device of data lake, storage medium and electronic equipment | |
| US10877881B2 (en) | In-place garbage collection of a sharded, replicated distributed state machine based on mergeable operations | |
| CN112765170B (en) | Embedded time sequence data management method and device | |
| CN116775646A (en) | Database data management method, device, computer equipment and storage medium | |
| CN111147226A (en) | Data storage method, device and storage medium | |
| CN111782588A (en) | File reading method, device, equipment and medium | |
| CN108376104B (en) | Node scheduling method and device and computer readable storage medium | |
| US8296336B2 (en) | Techniques for efficient dataloads into partitioned tables using swap tables | |
| CN110968587A (en) | Data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200508 |