CN118394802A - Power monitoring data storage management method, system, equipment and medium - Google Patents
Power monitoring data storage management method, system, equipment and medium Download PDFInfo
- Publication number
- CN118394802A CN118394802A CN202410842153.4A CN202410842153A CN118394802A CN 118394802 A CN118394802 A CN 118394802A CN 202410842153 A CN202410842153 A CN 202410842153A CN 118394802 A CN118394802 A CN 118394802A
- Authority
- CN
- China
- Prior art keywords
- data
- index
- storage node
- data block
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 56
- 238000012544 monitoring process Methods 0.000 title claims abstract description 38
- 238000013500 data storage Methods 0.000 title claims abstract description 31
- 238000012545 processing Methods 0.000 claims abstract description 26
- 238000000034 method Methods 0.000 claims abstract description 23
- 238000012795 verification Methods 0.000 claims abstract description 18
- 238000012790 confirmation Methods 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 2
- 238000004891 communication Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- Software Systems (AREA)
- Water Supply & Treatment (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Strategic Management (AREA)
- Primary Health Care (AREA)
- Marketing (AREA)
- Human Resources & Organizations (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to the technical field of power data processing, and particularly provides a power monitoring data storage management method, a system, equipment and a medium, wherein the method comprises the following steps: receiving a data writing request sent by power equipment; carrying out identity verification of a transmitting end on the data writing request, and receiving data transmitted by the power equipment after verification; processing the received data to divide the data blocks and calculating the data block ID; judging whether the index table contains the data block ID; the index table is a corresponding relation table of the data block ID and the physical address and records the writing times of the physical address; if not, acquiring the state of each storage node, and determining a target storage node according to the state of each storage node; writing the data block corresponding to the data block ID into a target storage node; creating a corresponding index for the data block written into the target storage node; the index is defined as a target index; the index table is updated based on the target index. And the data storage efficiency is improved.
Description
Technical Field
The invention relates to the technical field of power data processing, in particular to a power monitoring data storage management method, a system, equipment and a medium.
Background
In the power industry, with the rapid development of smart grids and the advancement of digital transformation, cross-department, cross-system sharing and information integration of power monitoring data are becoming increasingly important. The power monitoring data not only cover a plurality of aspects such as power grid running state, equipment health condition, energy consumption condition and the like, but also is a key basis for decision making, running management and fault processing of the power company. However, data sharing and information integration face many challenges due to severe data islanding between different departments, systems.
Through data storage management, reasonable classification, indexing and storage can be carried out on the data, and the retrieval and access efficiency of the data is improved. Therefore, when the power company needs to inquire or analyze the data, the required data can be quickly positioned, and the working efficiency is improved.
The existing power monitoring data storage management method is generally based on a relational database, adopts a fixed data structure and a storage mode, and cannot effectively cope with the characteristics of power monitoring data. On one hand, the electric power monitoring data has the characteristics of high real-time performance and quick change, and the data needs to be written in and updated frequently; on the other hand, the power monitoring data often has a large amount of repeated and redundant information, and effective de-duplication and compression processing is required. Therefore, the development of the efficient, flexible and extensible power monitoring data storage management method has important significance for improving the performance and efficiency of the power monitoring system.
Disclosure of Invention
Aiming at the problems existing in the storage management of the power monitoring data, the invention provides a storage management method, a system, equipment and a medium of the power monitoring data.
In a first aspect, the present invention provides a method for storing and managing power monitoring data, where the method is based on a system, and the system includes a management node and a plurality of storage nodes that are disposed in the same cluster; the management node is also connected with a plurality of power devices; the method comprises the following steps:
the management node receives a data writing request sent by the power equipment;
Carrying out identity verification of a transmitting end on the data writing request, and receiving data transmitted by the power equipment after verification;
processing the received data to divide the data blocks and calculating the data block ID;
judging whether the index table contains the data block ID; the index table is a corresponding relation table of the data block ID and the physical address and records the writing times of the physical address;
If yes, updating the corresponding writing times in the index table and caching the received data;
if not, acquiring the state of each storage node, and determining a target storage node according to the state of each storage node;
writing the data block corresponding to the data block ID into a target storage node;
creating a corresponding index for the data block written into the target storage node; the index is defined as a target index;
The index table is updated based on the target index.
As a further limitation of the present invention, the step of processing the received data to divide the data block and calculating the data block ID includes:
Dividing the received data into a plurality of data blocks with the same length according to the set length;
The data block ID of each data block is set by calculating the hash value of the corresponding data block.
As a further limitation of the technical solution of the present invention, the step of obtaining the state of each storage node and determining the destination storage node according to the state of each storage node includes:
acquiring the fault times of each storage node recorded by the latest K-1 times;
Setting a weight coefficient corresponding to each record;
calculating the average failure times of any storage node according to the set weight;
Calculating the average fault times of n nodes according to the average fault times of single storage nodes;
Sorting the storage nodes according to the calculated average failure times from high reliability to low reliability;
And determining a target storage node according to the reliability of the storage node and the free space of the storage node.
As a further limitation of the technical solution of the present invention, the step of updating the index table based on the target index includes:
inquiring an index table according to the target index;
Judging whether the index table comprises the index identical to the target index according to the query result;
if yes, marking the target index as a deleting state;
if not, creating an index corresponding to the updated data to an index table based on the target index.
As a further limitation of the present invention, the step of marking the target index as the deleted state includes, before or after:
And adding one to the write times of the physical address corresponding to the index identical to the target index in the index table.
As a further limitation of the technical solution of the present invention, the step of writing the data block corresponding to the data block ID into the storage node includes:
acquiring a data block corresponding to the data block ID;
calculating the number of occupied data lines according to the length of the data block, acquiring the position of the first idle data line from the index table, and writing the data into the data line;
If the current data length exceeds the bearing capacity of one data line, continuing to acquire the idle data line until the writing of all the data blocks is completed; and establishes an associative link between the rows of data.
As a further limitation of the technical solution of the present invention, the method further comprises:
After receiving the access request, based on the content of the access request, scanning the data line to identify the position of the first data line of the data record, thereby realizing the extraction of the access data;
and returning the extracted data to the front end for display.
In a second aspect, the present invention further provides a power monitoring data storage management system, where the system includes a management node and a plurality of storage nodes that are disposed in the same cluster; the management node is also connected with a plurality of power devices; the management node comprises a receiving module, a processing module, a data block ID judging module, an update buffer module, a storage node confirming module, a data writing module and an index creating module;
The receiving module is used for receiving a data writing request sent by the power equipment; carrying out identity verification of a transmitting end on the data writing request, and receiving data transmitted by the power equipment after verification;
The processing module is used for processing the received data to divide the data blocks and calculating the ID of the data blocks;
The data block ID judging module is used for judging whether the index table contains the data block ID or not; the index table is a corresponding relation table of the data block ID and the physical address and records the writing times of the physical address;
the updating and caching module is used for updating the corresponding writing times in the index table and caching the received data; updating an index table based on the target index;
The storage node confirmation module is used for acquiring the state of each storage node and determining a target storage node according to the state of each storage node;
The data writing module is used for writing the data block corresponding to the data block ID into the target storage node;
the index creating module is used for creating a corresponding index for the data block written into the target storage node; the index is defined as a target index.
As a further limitation of the technical scheme of the invention, the processing module comprises a dividing unit and a calculation setting unit;
the dividing unit is used for dividing the received data into a plurality of data blocks with the same length according to the set length;
a calculation setting unit for setting the data block ID of each data block by calculating the hash value of the corresponding data block.
As a further limitation of the technical scheme of the invention, the storage node confirmation module is specifically configured to obtain the number of failures of each storage node recorded in the latest K-1 times; setting a weight coefficient corresponding to each record; calculating the average failure times of any storage node according to the set weight; calculating the average fault times of n nodes according to the average fault times of single storage nodes; sorting the storage nodes according to the calculated average failure times from high reliability to low reliability; and determining a target storage node according to the reliability of the storage node and the free space of the storage node.
As a further limitation of the technical scheme of the present invention, the index table updating module is specifically configured to query the index table according to the target index; judging whether the index table comprises the index identical to the target index according to the query result; if yes, marking the target index as a deleting state; if not, creating an index corresponding to the updated data to an index table based on the target index.
As a further limitation of the technical scheme of the invention, the index table updating module is also used for adding one to the writing times of the physical address corresponding to the index which is the same as the target index in the index table.
As a further limitation of the technical scheme of the present invention, the data writing module is specifically configured to obtain a data block corresponding to the data block ID; calculating the number of occupied data lines according to the length of the data block, acquiring the position of the first idle data line from the index table, and writing the data into the data line; if the current data length exceeds the bearing capacity of one data line, continuing to acquire the idle data line until the writing of all the data blocks is completed; and establishes an associative link between the rows of data.
As a further limitation of the technical scheme of the invention, the request receiving module is also used for receiving the access request;
A request processing module; the method is also used for scanning the data line to identify the position of the first data line of the data record based on the content of the access request, so as to realize the extraction of the access data; and returning the extracted data to the front end for display.
In a third aspect, the present invention further provides an electronic device, where the electronic device includes: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores computer program instructions executable by the at least one processor to enable the at least one processor to perform the power monitoring data storage management method according to the first aspect.
In a fourth aspect, the present disclosure further provides a non-transitory computer readable storage medium, where the non-transitory computer readable storage medium stores computer instructions, where the computer instructions cause the computer to perform the power monitoring data storage management method according to the first aspect.
From the above technical scheme, the invention has the following advantages: by analyzing the data writing request, dividing the data into data blocks with fixed lengths and calculating the hash value as the data block ID, the quick positioning and searching of the data are realized. Meanwhile, the index table is used for recording the corresponding relation between the data block ID and the physical address, so that the whole table scanning in the database is avoided, and the data storage efficiency is improved. By checking whether the index table contains the data block ID, it can be judged whether the data to be written already exists. If the data block exists, the data block is not saved, and redundant storage of the data is reduced. Meanwhile, the index table also records the writing times of the physical address, and can perform data optimization and cleaning operation according to the writing times, so that the data redundancy is further reduced. For the data blocks written into the database, the position of the target data block can be quickly positioned by creating the corresponding index and updating the index table, so that the data query speed is improved. Meanwhile, the index creation and updating operation and the data writing operation are mutually independent, and concurrency of data writing and inquiring is guaranteed. The power monitoring data storage management method adopts a modularized design idea, and each module is relatively independent and can be expanded and customized according to actual requirements. For example, different data partitioning strategies and hash algorithms can be designed according to the characteristics of the power monitoring data so as to adapt to different application scenes.
In addition, the invention has reliable design principle, simple structure and very wide application prospect.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
FIG. 1 is a schematic flow chart of a method of one embodiment of the invention.
Fig. 2 is a schematic flow chart of a method of another embodiment of the invention.
FIG. 3 is a schematic block diagram of a system of one embodiment of the present invention.
Detailed Description
In order to make the technical solution of the present invention better understood by those skilled in the art, the technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
As shown in fig. 1, an embodiment of the present invention provides a power monitoring data storage management method, where the method is based on a pipe system, and the system includes a management node and a plurality of storage nodes that are disposed in the same cluster; the management node is also connected with a plurality of power devices; the method comprises the following steps:
step 1: the management node receives a data writing request sent by the power equipment;
Step 2: carrying out identity verification of a transmitting end on the data writing request, and receiving data transmitted by the power equipment after verification;
step 3: processing the received data to divide the data blocks and calculating the data block ID;
Step 4: judging whether the index table comprises a data block ID or not; the index table is a corresponding relation table of the data block ID and the physical address and records the writing times of the physical address;
If yes, executing the step 5; if not, executing the step 6;
it should be further noted that, the data written into the database in real time may be update data, or may be newly added data, for example, the database stores data of the user X, and the update data may be data for updating all or part of the data of the user X; the newly added data is considered to be the data added by a new user. That is, the update data is data that updates data existing in the database.
Step 5: updating the corresponding writing times in the index table and caching the received data;
in the embodiment of the invention, the management node is cached at regular time to obtain data deletion.
Step 6: acquiring the state of each storage node and determining a target storage node according to the state of each storage node;
step 7: writing the data block corresponding to the data block ID into a target storage node;
Step 8: creating a corresponding index for the data block written into the target storage node; the index is defined as a target index.
It should be noted that, creating a corresponding index for a data block written into the database specifically includes: the database comprises data blocks written based on the data writing request; creating a corresponding index for the data block written into the database; the index is defined as a target index; the index table is updated based on the target index.
Step 9: the index table is updated based on the target index.
As shown in fig. 2, in some embodiments, the step of processing the received data to divide the data blocks and calculate the data block IDs includes:
step 31: dividing the received data into a plurality of data blocks with the same length according to the set length;
Step 32: the data block ID of each data block is set by calculating the hash value of the corresponding data block.
In some embodiments, the step of obtaining the state of each storage node and determining the destination storage node based on the state of each storage node comprises:
Step 51: acquiring the fault times of each storage node recorded by the latest K-1 times;
Step 52: setting a weight coefficient corresponding to each record;
step 53: calculating the average failure times of any storage node according to the set weight;
step 54: calculating the average fault times of n nodes according to the average fault times of single storage nodes;
step 55: sorting the storage nodes according to the calculated average failure times from high reliability to low reliability;
Step 56: and determining a target storage node according to the reliability of the storage node and the free space of the storage node. The purpose of calculating the average number of failures of the n nodes is to select a storage node that satisfies a condition and that has an average number of failures less than the average number of failures of the n nodes when selecting the storage node.
When the storage node is selected, the storage node with high selection reliability, namely low average failure times and large free storage space is used as the target storage node.
In some embodiments, the step of updating the index table based on the target index comprises:
step 911: inquiring an index table according to the target index;
Step 912: judging whether the index table comprises the index identical to the target index according to the query result;
if yes, go to step 913; if not, go to step 914;
step 913: marking the target index as a deleted state;
step 914: and creating an index corresponding to the updated data to an index table based on the target index.
The same index is marked with an undeleted state to prevent errors in data backfilling and to prevent the writing of the index of old data (updated data) in the case of the index of new data.
In the embodiment of the present invention, before or after the step of marking the target index as the deleted state, the method includes:
And adding one to the write times of the physical address corresponding to the index identical to the target index in the index table.
The application designs that the same data is not repeatedly written, but the repeated data is required to be recorded, and the same data is recorded by setting the writing times.
The power equipment in the application can be a data acquisition terminal, or an edge terminal, wherein the edge terminal needs to transmit received data to a remote server or a cloud server in real time.
In some embodiments, the step of writing the data block corresponding to the data block ID to the storage node comprises:
acquiring a data block corresponding to the data block ID;
calculating the number of occupied data lines according to the length of the data block, acquiring the position of the first idle data line from the index table, and writing the data into the data line;
If the current data length exceeds the bearing capacity of one data line, continuing to acquire the idle data line until the writing of all the data blocks is completed; and establishes an associative link between the rows of data.
When data is written, writing is carried out row by row, the data blocks are agreed to have multiple data rows, the data rows are established with association links, at the moment, the association links can be carried out by using the data block IDs, and each data row of the multiple data rows of the same data block is provided with the data block IDs to establish row association.
In an embodiment of the present invention, the data change of the database includes operations of deleting data and writing data (including updating data). The delete operation does not actually perform a flush of disk storage space, but a new data overwrite write. This greatly improves the operational efficiency of the data operation.
In some embodiments, the method further comprises:
After receiving the access request, based on the content of the access request, scanning the data line to identify the position of the first data line of the data record, thereby realizing the extraction of the access data;
and returning the extracted data to the front end for display.
As shown in fig. 3, the embodiment of the invention further provides a power monitoring data storage management system, which comprises a management node and a plurality of storage nodes, wherein the management nodes and the storage nodes are arranged in the same cluster; the management node is also connected with a plurality of power devices; the management node comprises a receiving module, a processing module, a data block ID judging module, an update buffer module, a storage node confirming module, a data writing module and an index creating module;
The receiving module is used for receiving a data writing request sent by the power equipment; carrying out identity verification of a transmitting end on the data writing request, and receiving data transmitted by the power equipment after verification;
The processing module is used for processing the received data to divide the data blocks and calculating the ID of the data blocks;
The data block ID judging module is used for judging whether the index table contains the data block ID or not; the index table is a corresponding relation table of the data block ID and the physical address and records the writing times of the physical address;
the updating and caching module is used for updating the corresponding writing times in the index table and caching the received data; updating an index table based on the target index;
The storage node confirmation module is used for acquiring the state of each storage node and determining a target storage node according to the state of each storage node;
The data writing module is used for writing the data block corresponding to the data block ID into the target storage node;
the index creating module is used for creating a corresponding index for the data block written into the target storage node; the index is defined as a target index.
In some embodiments, the processing module includes a dividing unit and a calculation setting unit;
the dividing unit is used for dividing the received data into a plurality of data blocks with the same length according to the set length;
a calculation setting unit for setting the data block ID of each data block by calculating the hash value of the corresponding data block.
In some embodiments, the storage node confirmation module is specifically configured to obtain the number of failures of each storage node recorded in the latest K-1 times; setting a weight coefficient corresponding to each record; calculating the average failure times of any storage node according to the set weight; calculating the average fault times of n nodes according to the average fault times of single storage nodes; sorting the storage nodes according to the calculated average failure times from high reliability to low reliability; and determining a target storage node according to the reliability of the storage node and the free space of the storage node. And setting a weight coefficient according to the historical use condition of the storage node, wherein the number of faults in the historical use process is small and the weight coefficient is small.
In some embodiments, the index table updating module is specifically configured to query the index table according to the target index; judging whether the index table comprises the index identical to the target index according to the query result; if yes, marking the target index as a deleting state; if not, creating an index corresponding to the updated data to an index table based on the target index; and the method is also used for adding one to the writing times of the physical address corresponding to the index which is the same as the target index in the index table.
In some embodiments, the data writing module is specifically configured to obtain a data block corresponding to the data block ID; calculating the number of occupied data lines according to the length of the data block, acquiring the position of the first idle data line from the index table, and writing the data into the data line; if the current data length exceeds the bearing capacity of one data line, continuing to acquire the idle data line until the writing of all the data blocks is completed; and establishes an associative link between the rows of data.
In some embodiments, the request receiving module is further configured to receive an access request;
A request processing module; the method is also used for scanning the data line to identify the position of the first data line of the data record based on the content of the access request, so as to realize the extraction of the access data; and returning the extracted data to the front end for display.
The embodiment of the invention also provides electronic equipment, which comprises: the device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are in communication with each other through the communication bus. The communication bus may be used for information transfer between the electronic device and the sensor. The processor may call logic instructions in memory to perform the following method: step 1: the management node receives a data writing request sent by the power equipment; step 2: carrying out identity verification of a transmitting end on the data writing request, and receiving data transmitted by the power equipment after verification; step 3: processing the received data to divide the data blocks and calculating the data block ID; step 4: judging whether the index table comprises a data block ID or not; the index table is a corresponding relation table of the data block ID and the physical address and records the writing times of the physical address; if yes, executing the step 5; if not, executing the step 6; step 5: updating the corresponding writing times in the index table and caching the received data; step 6: acquiring the state of each storage node and determining a target storage node according to the state of each storage node; step 7: writing the data block corresponding to the data block ID into a target storage node; step 8: creating a corresponding index for the data block written into the target storage node; the index is defined as a target index; step 9: the index table is updated based on the target index.
Further, the logic instructions in the memory described above may be implemented in the form of software functional units and stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Embodiments of the present invention provide a non-transitory computer readable storage medium storing computer instructions that cause a computer to perform the methods provided by the method embodiments described above, for example, including: step 1: the management node receives a data writing request sent by the power equipment; step 2: carrying out identity verification of a transmitting end on the data writing request, and receiving data transmitted by the power equipment after verification; step 3: processing the received data to divide the data blocks and calculating the data block ID; step 4: judging whether the index table comprises a data block ID or not; the index table is a corresponding relation table of the data block ID and the physical address and records the writing times of the physical address; if yes, executing the step 5; if not, executing the step 6; step 5: updating the corresponding writing times in the index table and caching the received data; step 6: acquiring the state of each storage node and determining a target storage node according to the state of each storage node; step 7: writing the data block corresponding to the data block ID into a target storage node; step 8: creating a corresponding index for the data block written into the target storage node; the index is defined as a target index; step 9: the index table is updated based on the target index.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present invention.
In the embodiment of the power monitoring data storage management system provided by the embodiment of the present invention, the system and the power monitoring data storage management method of each embodiment belong to the same inventive concept, and reference may be made to the embodiment of the power monitoring data storage management method for details that are not described in detail in the embodiment of the power monitoring data storage management system.
The power monitoring data storage management system is the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein, and may be embodied in electronic hardware, computer software, or a combination of the two, the components and steps of each example having been generally described in terms of functionality in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
Those skilled in the art will appreciate that aspects of the power monitoring data storage management method may be implemented as a system, method, or program product. Accordingly, various aspects of the disclosure may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.
Although the present invention has been described in detail by way of preferred embodiments with reference to the accompanying drawings, the present invention is not limited thereto. Various equivalent modifications and substitutions may be made in the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and it is intended that all such modifications and substitutions be within the scope of the present invention/be within the scope of the present invention as defined by the appended claims.
Claims (10)
1. The power monitoring data storage management method is characterized by being based on a pipe system, wherein the system comprises a management node and a plurality of storage nodes, wherein the management node and the storage nodes are arranged in the same cluster; the management node is also connected with a plurality of power devices; the method comprises the following steps:
the management node receives a data writing request sent by the power equipment;
Carrying out identity verification of a transmitting end on the data writing request, and receiving data transmitted by the power equipment after verification;
processing the received data to divide the data blocks and calculating the data block ID;
judging whether the index table contains the data block ID; the index table is a corresponding relation table of the data block ID and the physical address and records the writing times of the physical address;
If yes, updating the corresponding writing times in the index table and caching the received data;
if not, acquiring the state of each storage node, and determining a target storage node according to the state of each storage node;
writing the data block corresponding to the data block ID into a target storage node;
creating a corresponding index for the data block written into the target storage node; the index is defined as a target index;
The index table is updated based on the target index.
2. The power monitoring data storage management method according to claim 1, wherein the step of processing the received data to divide the data blocks and calculating the data block IDs comprises:
Dividing the received data into a plurality of data blocks with the same length according to the set length;
The data block ID of each data block is set by calculating the hash value of the corresponding data block.
3. The power monitoring data storage management method according to claim 2, wherein the step of acquiring the state of each storage node and determining the destination storage node based on the state of each storage node comprises:
acquiring the fault times of each storage node recorded by the latest K-1 times;
Setting a weight coefficient corresponding to each record;
calculating the average failure times of any storage node according to the set weight;
Calculating the average fault times of n nodes according to the average fault times of single storage nodes;
Sorting the storage nodes according to the calculated average failure times from high reliability to low reliability;
And determining a target storage node according to the reliability of the storage node and the free space of the storage node.
4. The power monitoring data storage management method according to claim 3, wherein the step of updating the index table based on the target index comprises:
inquiring an index table according to the target index;
Judging whether the index table comprises the index identical to the target index according to the query result;
if yes, marking the target index as a deleting state;
if not, creating an index corresponding to the updated data to an index table based on the target index.
5. The power monitoring data storage management method according to claim 4, wherein the step of marking the target index as a deleted state includes, before or after:
And adding one to the write times of the physical address corresponding to the index identical to the target index in the index table.
6. The power monitoring data storage management method according to claim 5, wherein the step of writing the data block corresponding to the data block ID into the storage node includes:
acquiring a data block corresponding to the data block ID;
calculating the number of occupied data lines according to the length of the data block, acquiring the position of the first idle data line from the index table, and writing the data into the data line;
If the current data length exceeds the bearing capacity of one data line, continuing to acquire the idle data line until the writing of all the data blocks is completed; and establishes an associative link between the rows of data.
7. The power monitoring data storage management method of claim 6, further comprising:
After receiving the access request, based on the content of the access request, scanning the data line to identify the position of the first data line of the data record, thereby realizing the extraction of the access data;
and returning the extracted data to the front end for display.
8. The power monitoring data storage management system is characterized by comprising management nodes and a plurality of storage nodes, wherein the management nodes and the storage nodes are arranged in the same cluster; the management node is also connected with a plurality of power devices; the management node comprises a receiving module, a processing module, a data block ID judging module, an update buffer module, a storage node confirming module, a data writing module and an index creating module;
The receiving module is used for receiving a data writing request sent by the power equipment; carrying out identity verification of a transmitting end on the data writing request, and receiving data transmitted by the power equipment after verification;
The processing module is used for processing the received data to divide the data blocks and calculating the ID of the data blocks;
The data block ID judging module is used for judging whether the index table contains the data block ID or not; the index table is a corresponding relation table of the data block ID and the physical address and records the writing times of the physical address;
the updating and caching module is used for updating the corresponding writing times in the index table and caching the received data; updating an index table based on the target index;
The storage node confirmation module is used for acquiring the state of each storage node and determining a target storage node according to the state of each storage node;
The data writing module is used for writing the data block corresponding to the data block ID into the target storage node;
the index creating module is used for creating a corresponding index for the data block written into the target storage node; the index is defined as a target index.
9. An electronic device, the electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores computer program instructions executable by at least one processor to enable the at least one processor to perform the power monitoring data storage management method of any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform the power monitoring data storage management method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410842153.4A CN118394802A (en) | 2024-06-27 | 2024-06-27 | Power monitoring data storage management method, system, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410842153.4A CN118394802A (en) | 2024-06-27 | 2024-06-27 | Power monitoring data storage management method, system, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118394802A true CN118394802A (en) | 2024-07-26 |
Family
ID=91993072
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410842153.4A Pending CN118394802A (en) | 2024-06-27 | 2024-06-27 | Power monitoring data storage management method, system, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118394802A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105242881A (en) * | 2015-10-12 | 2016-01-13 | 创新科软件技术（深圳）有限公司 | Distributed storage system and data read-write method for same |
| CN106230982A (en) * | 2016-09-08 | 2016-12-14 | 哈尔滨工程大学 | A kind of dynamic self-adapting secure cloud storage method considering node reliability |
| CN106503084A (en) * | 2016-10-10 | 2017-03-15 | 中国科学院软件研究所 | A kind of storage and management method of the unstructured data of facing cloud database |
| CN108008918A (en) * | 2017-11-30 | 2018-05-08 | 联想(北京)有限公司 | Data processing method, memory node and distributed memory system |
| CN109241023A (en) * | 2018-09-21 | 2019-01-18 | 郑州云海信息技术有限公司 | Distributed memory system date storage method, device, system and storage medium |
| CN111522791A (en) * | 2020-04-30 | 2020-08-11 | 电子科技大学 | Distributed file repeating data deleting system and method |
| CN115981573A (en) * | 2023-02-16 | 2023-04-18 | 荣耀终端有限公司 | Data management method, electronic device and computer readable and writable storage medium |
-
2024
- 2024-06-27 CN CN202410842153.4A patent/CN118394802A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105242881A (en) * | 2015-10-12 | 2016-01-13 | 创新科软件技术（深圳）有限公司 | Distributed storage system and data read-write method for same |
| CN106230982A (en) * | 2016-09-08 | 2016-12-14 | 哈尔滨工程大学 | A kind of dynamic self-adapting secure cloud storage method considering node reliability |
| CN106503084A (en) * | 2016-10-10 | 2017-03-15 | 中国科学院软件研究所 | A kind of storage and management method of the unstructured data of facing cloud database |
| CN108008918A (en) * | 2017-11-30 | 2018-05-08 | 联想(北京)有限公司 | Data processing method, memory node and distributed memory system |
| CN109241023A (en) * | 2018-09-21 | 2019-01-18 | 郑州云海信息技术有限公司 | Distributed memory system date storage method, device, system and storage medium |
| CN111522791A (en) * | 2020-04-30 | 2020-08-11 | 电子科技大学 | Distributed file repeating data deleting system and method |
| CN115981573A (en) * | 2023-02-16 | 2023-04-18 | 荣耀终端有限公司 | Data management method, electronic device and computer readable and writable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103902623B (en) | Method and system for the accessing file in storage system | |
| US11232073B2 (en) | Method and apparatus for file compaction in key-value store system | |
| CN110196847A (en) | Data processing method and device, storage medium and electronic device | |
| CN107943718A (en) | A kind of method and apparatus for clearing up cache file | |
| CN104584524A (en) | Aggregating data in a mediation system | |
| CN110928690B (en) | Method for synchronizing and verifying block chain data in 5G network environment | |
| CN112463542B (en) | Log abnormality cause diagnosis method, device, computer equipment and storage medium | |
| CN113867627A (en) | Method and system for optimizing performance of storage system | |
| CN110502510B (en) | Real-time analysis and duplicate removal method and system for WIFI terminal equipment trajectory data | |
| CN113326343B (en) | Road network data storage method and system based on multi-level grids and file indexes | |
| CN110727733A (en) | Main and standby server system and data synchronization method | |
| CN113420036A (en) | Consistency checking method for internal storage relation database of power grid monitoring system | |
| CN110888846B (en) | Data memory management method and system | |
| CN108959614A (en) | A kind of snapshot management method, system, device, equipment and readable storage medium storing program for executing | |
| CN118394802A (en) | Power monitoring data storage management method, system, equipment and medium | |
| CN107465706B (en) | Distributed data object storage device based on wireless communication network | |
| CN111966635A (en) | Method and device for improving file detection speed of distributed storage file system | |
| CN113626382B (en) | Index method, system and related device for aggregation object in distributed file system | |
| CN116820323A (en) | Data storage method, device, electronic equipment and computer readable storage medium | |
| CN112596682A (en) | Storage device and storage method for block chain | |
| CN117472918B (en) | Data processing method, system, electronic device and storage medium | |
| CN118093557B (en) | Block chain node state data processing method, electronic device and storage medium | |
| CN117992257B (en) | Parallel data acquisition and processing method for distributed database | |
| CN117971905B (en) | Caching and indexing method for real-time statistics of historical data of industrial production process | |
| CN117056134B (en) | Method for quickly backing up database data in energy consumption monitoring system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |