CN107992519A - The multi-source heterogeneous data verification system and method for a kind of smart grid-oriented big data - Google Patents
The multi-source heterogeneous data verification system and method for a kind of smart grid-oriented big data Download PDFInfo
- Publication number
- CN107992519A CN107992519A CN201711053946.4A CN201711053946A CN107992519A CN 107992519 A CN107992519 A CN 107992519A CN 201711053946 A CN201711053946 A CN 201711053946A CN 107992519 A CN107992519 A CN 107992519A
- Authority
- CN
- China
- Prior art keywords
- verification
- database
- rule
- task
- content
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000013524 data verification Methods 0.000 title claims abstract description 32
- 238000012795 verification Methods 0.000 claims abstract description 291
- 238000012360 testing method Methods 0.000 claims description 14
- 238000004458 analytical method Methods 0.000 claims description 7
- 208000011580 syndromic disease Diseases 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 206010054949 Metaplasia Diseases 0.000 claims 1
- 230000015689 metaplastic ossification Effects 0.000 claims 1
- 238000012942 design verification Methods 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 49
- 238000013461 design Methods 0.000 description 9
- 230000003993 interaction Effects 0.000 description 8
- 238000012545 processing Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 230000010354 integration Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000008859 change Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 241000894007 species Species 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000003326 Quality management system Methods 0.000 description 1
- 235000004240 Triticum spelta Nutrition 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000009412 basement excavation Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000000547 structure data Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Water Supply & Treatment (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to the multi-source heterogeneous data verification system and method for a kind of smart grid-oriented big data, and content and the verification pre-established rule generation verification task and configuration file corresponding with the verification task are verified according to obtaining；When needing to perform verification task, the corresponding configuration file of the verification task is performed；The verification rule pre-established includes：Verification rule is set based on verifying function.The present invention can support single table in multi-class data storehouse to verify, the association of single database multilist verifies, the association verification of multiple database multilist, foundation verifying function and method design verification rule library structure, realize flexibility and the scalability of data check function.
Description
Technical field
The present invention relates to the calibration technology in power dispatching automation field, and in particular to a kind of smart grid-oriented big data
Multi-source heterogeneous data verification system and method.
Background technology
With developing rapidly for intelligent grid, the computerized information communication technology is in power industry extensive utilization, support power grid
Each operation system of safe and stable operation through accumulating for many years, the information resources that are richly stored with and knowhow.To meet data
Synthesis use demand, big data Platform integration and manage these be dispersed in different institutions, different majors, different place of safety it is more
Source isomery electric network data, forms unified, organized shared resource, and the data service environment concentrated, be conducive to from
Middle analysis, excavate valuable information and rule, and effective technical support is provided for operation of power networks decision-making.
The structure of big data platform realizes integration storage, management and the application of data, to ensure the data of high quality
And data service, it is necessary to effective data quality management system is established during Data Integration.Data check is as data
The necessary basis and key link of quality management, for location data problem accurately and in time, ETL is repaired easy to related personnel
The wrong data of task or operation system.However, calibration tool is directed to traditional relational at present more, verifies rule and lack
Design is customized, and method of calibration is difficult to extend.
The content of the invention
To overcome verification rule uncertain, method of calibration is difficult to extend, and there are no centralized database and multiple database
The problem of verification is distinguished, the object of the present invention is to provide a kind of multi-source heterogeneous data verification system of smart grid-oriented big data
And method, with reference to intelligent grid business and big data feature, the design to verifying being customized of rule base, it would be preferable to support single
Database verifies and multiple database verification.
The purpose of the present invention is what is realized using following technical proposals：
The present invention provides a kind of multi-source heterogeneous data verification method of smart grid-oriented big data, its improvements exists
In：
According to obtaining verification content and the verification pre-established rule generates verification task and corresponding with the verification task
Configuration file；
When needing to perform verification task, the corresponding configuration file of the verification task is performed；
The verification rule pre-established includes：Verification rule is set based on verifying function.
Further：It is described to be included based on verifying function setting verification rule：
Big rule-like is verified based on verifying function setting；
It is the main table of rule by the verification major class rule settings；
According to the verification content setting syndrome rule-like, the structure according to the subclass rule settings rule from table,
The verification content is corresponded to and is stored to described regular from table；
The big rule-like of verification includes：Centralized database verification, multiple database verification connect verification with database.
Further：The verification content according to acquisition and the verification rule generation verification task pre-established and with institute
The corresponding configuration file of verification task is stated, including：
The corresponding content generation verification from table of the regular main table stored in the database table structure and rule is read to appoint
Business, and the regular main table stored in the database table structure and the therefrom corresponding content of rule are carried out being converted into XML lattice
The configuration file of formula.
Further：The centralized database verification includes：Centralized database verification content is carried out according to verification task
Verification, the centralized database verification content include database linkage information and centralized database information；
The multiple database verification includes：Multiple database verification content is verified according to verification task, the majority
Include database linkage information and multiple database information according to storehouse verification content.
Further：The database linkage information includes type of database, IP, port, username and password；
The centralized database information includes the verification of individual character section, single table multi-field association verification associates school with multilist multi-field
Test；The multiple database information includes the association verification of multilist multi-field；
The database linkage information, centralized database information, the verification content of multiple database information include to be verified
Table name, field name and associate field relation.
Further：It is described to perform the corresponding configuration file of the verification task, including：
The configuration file of the corresponding XML format of the verification task is parsed according to verification task；
Configuration file after parsing is performed, and generates check results.
Further：The configuration file of the parsing XML format, including：
The configuration file of XML format is parsed into specific Java Bean objects；
The school of database linkage information, centralized database information and multiple database information is obtained from Java Bean objects
Test content.
Further：The generation check results, including：
According to database linkage information, centralized database information, the verification content of multiple database information, database company is carried out
Connect and call basic function storehouse；
Spelling centralized database or multiple database SQL statement are verified, and obtain centralized database check results, more data
The check results of storehouse check results or database linkage information.
Further：The execution verification task includes：It is immediately performed the verification task and is set for the verification task
Surely the time is performed, the verification task is performed according to the timing of the execution time of setting.
The present invention provides a kind of multi-source heterogeneous data verification system of smart grid-oriented big data, its improvements exists
In：
Set generation module, for according to obtain verification content and pre-establish verification rule generate verification task and with
The corresponding configuration file of the verification task；
Execution module, for when needing to perform verification task, performing the corresponding configuration file of the verification task；
The verification rule pre-established includes：Verification rule is set based on verifying function.
Further：The setting generation module, including：
Submodule is set, for setting verification rule based on verifying function；
Generate submodule, according to obtain verification content and pre-establish verification rule generate verification task and with the school
Test the corresponding configuration file of task.
Further：The setting submodule, including：
First setup unit, for verifying big rule-like based on verification rule settings；
Second setup unit, for being the main table of rule by the verification major class rule settings；
3rd setup unit, for according to the verification content setting syndrome rule-like, being set according to the subclass rule
Set pattern then from the structure of table, the verification content is corresponded to and is stored to described regular from table；
The big rule-like of verification includes：Centralized database verification, multiple database verification and database correctly connect verification.
Further：The generation submodule, including：
Reading unit, for reading the structure of the regular main table structure stored in the database table structure and rule from table
In corresponding content generation verification task；
Conversion unit, for by the regular main table stored in the database table structure and the therefrom corresponding content of rule into
Row is converted into the configuration file of XML format.
Further：The execution module, including：
Resolution unit, for parsing the configuration file of the corresponding XML format of the verification task according to verification task；
Generation unit, configuration file after being parsed for execution, and generate check results.
Further：The resolution unit, including：
Analysis of object subelement, for XML file to be parsed into specific Java Bean objects；
Subelement is obtained, for obtaining database linkage information, centralized database information, more from Java Bean objects
The verification content of database information.
Further：The database linkage information includes type of database, IP, port, username and password；
The centralized database information includes the verification of individual character section, single table multi-field association verification associates school with multilist multi-field
Test；
The multiple database information includes the association verification of multilist multi-field；
The database linkage information, centralized database information, the verification content of multiple database information include to be verified
Table name, field name and associate field relation.
Further：The check results include centralized database check results, multiple database check results and database
The check results of link information.
Compared with the immediate prior art, technical solution provided by the invention has an advantageous effect in that：
1st, the present invention by the following technical programs：According to the verification rule generation verification for obtaining verification content with pre-establishing
The corresponding configuration file of task；When needing to perform verification task, the corresponding configuration file of the verification task is performed；It is described pre-
The verification rule first formulated includes：Verification rule is set based on verifying function, calibration tool is solved and is directed to conventional relationship more
The problem of database, verifies rule and lacks customization design, and method of calibration is difficult to extend.
2nd, with reference to intelligent grid business and big data feature, the design to verifying being customized of rule base, it would be preferable to support
Centralized database verifies and multiple database verification, including：Two major class verifying functions are designed and Implemented in checking treatment module, are wrapped
Include general verifying function and electrical network business verifying function, it would be preferable to support including the multiclass such as traditional relational, Hive, Hbase
The verification of single table, single database multilist association verification, the association verification of multiple database multilist of database.
3rd, according to verifying function and method design verification rule library structure, and rule parsing generation module and man-machine friendship are utilized
The customization configuration of Rule content is mutually carried out, realizes and is directed to the flexible, reliable of power grid big data feature and data integration demand
Data check function, lay the foundation with management for data quality accessment, in order to accurate analysis and the excavation of valuable information,
Effective support is provided for power network safety operation.
Brief description of the drawings
Fig. 1 is multi-source heterogeneous data verification system structure chart provided by the invention；
Fig. 2 is that multi-source heterogeneous data verification method flow provided by the invention is schemed in detail；
Fig. 3 is multi-source heterogeneous data verification method general flow chart provided by the invention.
Embodiment
The embodiment of the present invention is described in further detail below in conjunction with the accompanying drawings.
The following description and drawings fully show specific embodiments of the present invention, to enable those skilled in the art to
Put into practice them.Other embodiments can include structure, logic, it is electric, process and other change.Embodiment
Only represent possible change.Unless explicitly requested, otherwise single component and function are optional, and the order operated can be with
Change.The part of some embodiments and feature can be included in or replace part and the feature of other embodiments.This hair
The scope of bright embodiment includes the gamut of claims, and claims is all obtainable equivalent
Thing.Herein, these embodiments of the invention can individually or generally be represented that this is only with term " invention "
For convenience, and if in fact disclosing the invention more than one, the scope for being not meant to automatically limit the application is to appoint
What single invention or inventive concept.
Embodiment one,
Power grid big data Platform integration is scattered in the multi-source heterogeneous data of different business systems, to meet each application to high-quality
The requirement of data and data service is measured, the present invention provides a kind of multi-source heterogeneous data check system of smart grid-oriented big data
System and method, effective support is provided for power network safety operation.
Data verification system provided by the invention is to be based on power grid big data platform, passes through the multi-source heterogeneous data of comprehensive analysis
Integration and integrated demand, and electrical network business data characteristics are combined, design and Implement two major class verifying functions, including general verification work(
Single table verification, the association of single database multilist can be supported to verify, the association verification of multiple database multilist with business verifying function.Business
Personnel can be set by man-machine interface, change checking parameter, and checked, exported check results.
Multi-source heterogeneous data verification system structure chart provided by the invention as shown in Figure 1, including：
Set generation module, for according to obtain verification content and pre-establish verification rule generate verification task and with
The corresponding configuration file of the verification task；
Execution module, for when needing to perform verification task, performing the corresponding configuration file of the verification task；
The verification rule pre-established includes：Verification rule is set based on verifying function.
Further：The setting generation module, including：
Submodule is set, for setting verification rule based on verifying function；
Generate submodule, according to obtain verification content and pre-establish verification rule generate verification task and with the school
Test the corresponding configuration file of task.
Further：The setting submodule, including：
First setup unit, for verifying big rule-like based on verification rule settings；
Second setup unit, for being the main table of rule by the verification major class rule settings；
3rd setup unit, for according to the verification content setting syndrome rule-like, being set according to the subclass rule
Set pattern then from the structure of table, the verification content is corresponded to and is stored to described regular from table；
The big rule-like of verification includes：Centralized database verification, multiple database verification and database correctly connect verification.
Further：The generation submodule, including：
Reading unit, for reading the structure of the regular main table structure stored in the database table structure and rule from table
In corresponding content generation verification task；
Conversion unit, for by the regular main table stored in the database table structure and the therefrom corresponding content of rule into
Row is converted into the verification task configuration file of XML format.
Further：The execution module, including：
Resolution unit, for parsing the configuration file of the corresponding XML format of the verification task according to verification task；
Generation unit, configuration file after being parsed for execution, and generate check results.
Further：The resolution unit, including：
Analysis of object subelement, for the configuration file of XML format to be parsed into specific Java Bean objects；
Subelement is obtained, for obtaining database linkage information, centralized database information, more from Java Bean objects
The verification content of database information.
Further：The database linkage information includes type of database, IP, port, username and password；
The centralized database information includes the verification of individual character section, single table multi-field association verification associates school with multilist multi-field
Test；
The multiple database information includes the association verification of multilist multi-field；
The database linkage information, centralized database information, the verification content of multiple database information include to be verified
Table name, field name and associate field relation.
Further：The check results include centralized database check results, multiple database check results and database
The check results of link information.
Multi-source heterogeneous data verification system structure chart provided by the invention is as shown in Figure 1, in practical applications, can develop
For：Pass through five human-computer interaction, verification rule base, rule parsing generation module, checking treatment module, result-generation module parts
Realize the multi-source heterogeneous data check towards power grid big data.
Human-computer interaction is used for business personnel's configuration, modification verification rule, selection check classification, setting data quality accessment power
Weight, checks and exports check results.The method of calibration that verification rule base is provided according to the system, is designed different by difference in functionality
Table structure, stores verification rule input by user in relevant database.
Rule parsing generation module is used for realization from man-machine interface and receives verification rule, is stored by verification classification to different
Table structure, and the verification Rule content stored in database table can be converted, XML format file is generated, is transferred to school
Test processing module, including memory module, conversion module and delivery module；
The memory module is used to receive verification rule from human-computer interaction module, and is stored by verification classification to different tables
Structure；
The conversion module is used to be converted the verification Rule content stored in database table, generation XML format text
Part；
The delivery module is used to the XML format file of generation being sent to checking treatment module.
Checking treatment module is the core of check system, including all methods of calibration and function that the system possesses, mainly
Cover two major classes：General verifying function, the business verifying function for power grid big data feature, can support conventional relationship data
Storehouse, data warehouse (Hive) and distributed data base (Hbase), can realize table verification and multi-class data inside centralized database
Association verification between the difference table of storehouse, and according to business demand, it is easy to accomplish the exploitation and extension of new verifying function.
Result-generation module is used to obtain from checking treatment module and store check results, is generated according to human-computer interaction demand
Verification report.
The design considerations of multi-source heterogeneous data check rule base is the function that checking treatment module is covered, and designs main table and uses
In the function major class that records the system and can verify, including centralized database verification, multiple database verification, database correctly connect
Verification.The subclass of all kinds of verifying functions determines the species and quantity from table, and the specific method of verifying function is determined from table
Table structure.
The fill substance source of verification each table of rule base is that the verification that business personnel is inputted by man-machine interface configures, towards
The verification content of intelligent grid big data includes general rule and business rule.Wherein the non-NULL of general rule such as critical field with
Uniqueness verification, data type verification, value range verification, associate field non-NULL and consistency desired result etc.；Business rule is as closed
Whether key field fill substance meets normal structure, power network topology rule verifies, whether the fusion of more sources operation systems meets model
Whether unique mark, business datum value are beyond abnormal ranges etc..
Data check function realizes that the module is given birth to verification rule base, rule parsing with method in checking treatment module
Into module, result-generation module, there are interactive relation.Task to be verified is obtained by rule parsing generation module, through method of calibration
Check results are obtained after processing, are exported to result-generation module.Meanwhile the method that the verifying function is provided is used to verify rule
The global design in storehouse and each table structure.
Data check module is connected correct by basic function storehouse, centralized database verification, multiple database verification and database
Property verification four parts composition.Basic function storehouse includes the basic methods needed for verifying function, as type judges, for empty judgement, several
It is worth scope judgement etc., is called for database verifying function.Centralized database verification, multiple database verification are according to verification task
Content, calls basic function storehouse, realizes the verification of individual character section, single table multi-field association verification, the association verification of multilist multi-field respectively.
Embodiment two,
The present invention also provides a kind of method of calibration based on multi-source heterogeneous data verification system, its flow chart such as Fig. 2 and 3 institutes
Show including：
According to obtaining verification content and the verification pre-established rule generates verification task and corresponding with the verification task
Configuration file；
When needing to perform verification task, the corresponding configuration file of the verification task is performed；
The verification rule pre-established includes：Verification rule is set based on verifying function.
Further：It is described to be included based on verifying function setting verification rule：
Big rule-like is verified based on verifying function setting；
It is the main table of rule by the verification major class rule settings；
According to the verification content setting syndrome rule-like, the structure according to the subclass rule settings rule from table,
The verification content is corresponded to and is stored to described regular from table；
The big rule-like of verification includes：Centralized database verification, multiple database verification connect verification with database.
Further：The verification content according to acquisition and the verification rule generation verification task pre-established and with institute
The corresponding configuration file of verification task is stated, including：
The corresponding content generation verification from table of the regular main table stored in the database table structure and rule is read to appoint
Business, and the regular main table stored in the database table structure and the therefrom corresponding content of rule are carried out being converted into XML lattice
The configuration file of formula.
Further：The centralized database verification includes：Centralized database verification content is carried out according to verification task
Verification, the centralized database verification content include database linkage information and centralized database information；
The multiple database verification includes：Multiple database verification content is verified according to verification task, the majority
Include database linkage information and multiple database information according to storehouse verification content.
Further：The database linkage information includes type of database, IP, port, username and password；
The centralized database information includes the verification of individual character section, single table multi-field association verification associates school with multilist multi-field
Test；The multiple database information includes the association verification of multilist multi-field；
The database linkage information, centralized database information, the verification content of multiple database information include to be verified
Table name, field name and associate field relation.
Further：It is described to perform the corresponding configuration file of the verification task, including：
The configuration file of the corresponding XML format of the verification task is parsed according to verification task；
Configuration file after parsing is performed, and generates check results.
Further：The configuration file of the parsing XML format, including：
The configuration file of XML format is parsed into specific Java Bean objects；
The school of database linkage information, centralized database information and multiple database information is obtained from Java Bean objects
Test content.
Further：The generation check results, including：
According to database linkage information, centralized database information, the verification content of multiple database information, database company is carried out
Connect and call basic function storehouse；
Spelling centralized database or multiple database SQL statement are verified, and obtain centralized database check results, more data
The check results of storehouse check results or database linkage information.
Further：The execution verification task includes：It is immediately performed the verification task and is set for the verification task
Surely the time is performed, the verification task is performed according to the timing of the execution time of setting.
Specifically：
S101, the verifying function included according to checking treatment module and method, carry out verification rule base global design and
Each table structure；
S102, rule parsing generation module are obtained by human-computer interaction and parse verification rule, fill in verification rule base
The content of corresponding table；
The verification species that S103, rule parsing generation module are specified according to human-computer interaction, the school that will be stored in database table
Test Rule content to be converted, generate the verification task configuration file of XML format；
S104, checking treatment module parsing verification task configuration file, read operation system to be verified, perform verification and appoint
Business, and count generation check results；
S105, result-generation module obtain and store check results, and verification report is generated according to human-computer interaction demand.
Wherein：S104, the verification step of checking treatment module are as follows：
1) the corresponding configuration text of verification task that processing module reads the XML format generated by rule parsing generation module is tested
Part；
2) checking treatment module parsing verification task configuration file, specific method are：Based on Apache Commons instruments
Bag, using I/O tool sets therein, is parsed into a specific Java Bean object by XML file, stores and filled in template
Verification rule type, the information such as content；
3) database connection correctness verification is carried out, specific method is：For disparate databases, its corresponding java is called
Jdbc, obtains link information, if successful connection returns to true, otherwise returns to false；
4) Java Bean objects are called, the rule type mark of wherein affiliated verification task is obtained, is marked according to the rule
Know, judge to need the method for calibration specifically called；
5) verified if centralized database, then carry out the parsing and classification of verification task, called basic function storehouse, perform list
One database verifies task, and specific method is：Information needed, including database connection letter are verified from Java Bean object acquisitions
Cease (type of database, IP, port, user name, password etc.), verification type information (single form fields verification, single table multi-field school
Test, multilist association verification etc.) and specifically verify content information (table name to be verified, field name, associate field relation etc.),
According to above- mentioned information, database connection is carried out, by calling basic function storehouse, spelling SQL statement is verified, final to obtain list
One database check results；
6) verified if multiple database, then carry out the parsing and classification of verification task, called basic function storehouse, perform majority
According to the corresponding configuration file of storehouse verification task, specific method is：Information needed is verified from Java Bean object acquisitions, including it is more
A database linkage information (type of database, IP, port, user name, password etc.), (multilist associates school to verification type information
Test) and specifically verify content information (table name to be verified, field name, associate field relation etc.), according to above- mentioned information, point
Multiple databases are not connected, by calling basic function storehouse, are spelt SQL statement in disparate databases, are obtained and store to be compared
Information, then program realize comparison handle, finally obtain multiple database check results；
7) according to different check type, count and export corresponding check results.
Data verification system provided by the invention and method are to be based on power grid big data platform, different by comprehensive analysis multi-source
Structure Data Integration and integrated demand, and electrical network business data characteristics are combined, two major classes are designed and Implemented in checking treatment module
Verifying function, including general verifying function and electrical network business verifying function, according to verifying function and method design verification rule base
Structure, and configured using the customization of rule parsing generation module and human-computer interaction progress Rule content, realize data check
The flexibility of function and scalability.
It should be understood by those skilled in the art that, embodiments herein can be provided as method, system or computer program
Product.Therefore, the application can use the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware
Apply the form of example.Moreover, the application can use the computer for wherein including computer usable program code in one or more
The computer program production that usable storage medium is implemented on (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
The form of product.
The application is with reference to the flow according to the method for the embodiment of the present application, equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that it can be realized by computer program instructions every first-class in flowchart and/or the block diagram
The combination of flow and/or square frame in journey and/or square frame and flowchart and/or the block diagram.These computer programs can be provided
The processors of all-purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce
A raw machine so that the instruction performed by computer or the processor of other programmable data processing devices, which produces, to be used in fact
The device for the function of being specified in present one flow of flow chart or one square frame of multiple flows and/or block diagram or multiple square frames.
These computer program instructions, which may also be stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory, which produces, to be included referring to
Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one square frame of block diagram or
The function of being specified in multiple square frames.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that counted
Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, thus in computer or
The instruction performed on other programmable devices is provided and is used for realization in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in a square frame or multiple square frames.
The above embodiments are merely illustrative of the technical scheme of the present invention and are not intended to be limiting thereof, although with reference to above-described embodiment pair
The present invention is described in detail, those of ordinary skill in the art still can to the present invention embodiment into
Row modification either equivalent substitution these without departing from any modification of spirit and scope of the invention or equivalent substitution, applying
Within pending claims of the invention.
Claims (17)
- A kind of 1. multi-source heterogeneous data verification method of smart grid-oriented big data, it is characterised in that：According to obtaining verification content and the verification pre-established rule generates verification task and corresponding with the verification task matches somebody with somebody Put file；When needing to perform verification task, the corresponding configuration file of the verification task is performed；The verification rule pre-established includes：Verification rule is set based on verifying function.
- 2. multi-source heterogeneous data verification method as claimed in claim 1, it is characterised in that：It is described that school is set based on verifying function Testing rule includes：Big rule-like is verified based on verifying function setting；It is the main table of rule by the verification major class rule settings；According to the verification content setting syndrome rule-like, the structure according to the subclass rule settings rule from table, by institute Verification content correspondence is stated to store to described regular from table；The big rule-like of verification includes：Centralized database verification, multiple database verification connect verification with database.
- 3. multi-source heterogeneous data verification method as claimed in claim 2, it is characterised in that：The verification content according to acquisition Verification task and configuration file corresponding with the verification task are generated with the verification rule pre-established, including：The regular main table stored in the database table structure and the rule corresponding content generation verification task from table are read, and The regular main table stored in the database table structure and the therefrom corresponding content of rule are carried out being converted into matching somebody with somebody for XML format Put file.
- 4. multi-source heterogeneous data verification method as claimed in claim 2, it is characterised in that：The centralized database verification bag Include：Centralized database verification content is verified according to verification task, the centralized database verification content includes database Link information and centralized database information；The multiple database verification includes：Multiple database verification content is verified according to verification task, the multiple database Verification content includes database linkage information and multiple database information.
- 5. multi-source heterogeneous data verification method as claimed in claim 4, it is characterised in that：The database linkage information includes Type of database, IP, port, username and password；The centralized database information includes the verification of individual character section, single table multi-field association verification associates verification with multilist multi-field； The multiple database information includes the association verification of multilist multi-field；The database linkage information, centralized database information, the verification content of multiple database information include table name to be verified, Field name and associate field relation.
- 6. multi-source heterogeneous data verification method as claimed in claim 3, it is characterised in that：It is described to perform the verification task pair The configuration file answered, including：The configuration file of the corresponding XML format of the verification task is parsed according to verification task；Configuration file after parsing is performed, and generates check results.
- 7. multi-source heterogeneous data verification method as claimed in claim 6, it is characterised in that：The configuration of the parsing XML format File, including：XML file is parsed into specific Java Bean objects；In the verification that database linkage information, centralized database information and multiple database information are obtained from Java Bean objects Hold.
- 8. multi-source heterogeneous data verification method as claimed in claim 7, it is characterised in that：The generation check results, including：According to database linkage information, centralized database information, the verification content of multiple database information, database connection is carried out simultaneously Call basic function storehouse；Spelling centralized database or multiple database SQL statement are verified, and obtain centralized database check results, multiple database school Test result or the check results of database linkage information.
- 9. multi-source heterogeneous data verification method as claimed in claim 1, it is characterised in that：The execution verification task includes： It is immediately performed the verification task and performs the time for the verification task setting, institute is performed according to the timing of the execution time of setting State verification task.
- A kind of 10. multi-source heterogeneous data verification system of smart grid-oriented big data, it is characterised in that：Set generation module, for according to obtain verification content and pre-establish verification rule generate verification task and with it is described The corresponding configuration file of verification task；Execution module, for when needing to perform verification task, performing the corresponding configuration file of the verification task；The verification rule pre-established includes：Verification rule is set based on verifying function.
- 11. multi-source heterogeneous data verification system as claimed in claim 10, it is characterised in that：The setting generation module, bag Include：Submodule is set, for setting verification rule based on verifying function；Generate submodule, according to obtain verification content and pre-establish verification rule generation verification task and with it is described verification times It is engaged in corresponding configuration file.
- 12. multi-source heterogeneous data verification system as claimed in claim 11, it is characterised in that：The setting submodule, including：First setup unit, for verifying big rule-like based on verification rule settings；Second setup unit, for being the main table of rule by the verification major class rule settings；3rd setup unit, for according to the verification content setting syndrome rule-like, being advised according to the subclass rule settings Then from the structure of table, the verification content is corresponded to and is stored to described regular from table；The big rule-like of verification includes：Centralized database verification, multiple database verification and database correctly connect verification.
- 13. multi-source heterogeneous data verification system as claimed in claim 11, it is characterised in that：The generation submodule, including：Reading unit, for reading the regular main table structure stored in the database table structure and the rule phase from the structure of table The content generation verification task answered；Conversion unit, for the regular main table stored in the database table structure and the therefrom corresponding content of rule to be turned Configuration file of the metaplasia into XML format.
- 14. multi-source heterogeneous data verification system as claimed in claim 10, it is characterised in that：The execution module, including：Resolution unit, for parsing the configuration file of the corresponding XML format of the verification task according to verification task；Generation unit, configuration file after being parsed for execution, and generate check results.
- 15. multi-source heterogeneous data verification system as claimed in claim 14, it is characterised in that：The resolution unit, including：Analysis of object subelement, for the configuration file of XML format to be parsed into specific Java Bean objects；Subelement is obtained, for obtaining database linkage information, centralized database information, more data from Java Bean objects The verification content of storehouse information.
- 16. multi-source heterogeneous data verification system as claimed in claim 15, it is characterised in that：The database linkage information bag Include type of database, IP, port, username and password；The centralized database information includes the verification of individual character section, single table multi-field association verification associates verification with multilist multi-field；The multiple database information includes the association verification of multilist multi-field；The database linkage information, centralized database information, the verification content of multiple database information include table name to be verified, Field name and associate field relation.
- 17. multi-source heterogeneous data verification system as claimed in claim 14, it is characterised in that：The check results include single The check results of database check results, multiple database check results and database linkage information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711053946.4A CN107992519A (en) | 2017-10-31 | 2017-10-31 | The multi-source heterogeneous data verification system and method for a kind of smart grid-oriented big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711053946.4A CN107992519A (en) | 2017-10-31 | 2017-10-31 | The multi-source heterogeneous data verification system and method for a kind of smart grid-oriented big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107992519A true CN107992519A (en) | 2018-05-04 |
Family
ID=62031065
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711053946.4A Pending CN107992519A (en) | 2017-10-31 | 2017-10-31 | The multi-source heterogeneous data verification system and method for a kind of smart grid-oriented big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107992519A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108804613A (en) * | 2018-05-30 | 2018-11-13 | 国网山东省电力公司经济技术研究院 | A kind of Various database real time fusion system and its fusion method |
| CN109299074A (en) * | 2018-10-30 | 2019-02-01 | 珠海伟诚科技股份有限公司 | A kind of data verification method and system based on templating data base view |
| CN109597828A (en) * | 2018-09-29 | 2019-04-09 | 阿里巴巴集团控股有限公司 | A kind of off-line data checking method, device and server |
| CN112001710A (en) * | 2020-09-07 | 2020-11-27 | 山东钢铁集团日照有限公司 | Big data reading and integrating system in steel product production process |
| CN112632052A (en) * | 2020-12-29 | 2021-04-09 | 中通服公众信息产业股份有限公司 | Heterogeneous data sharing method and intelligent sharing system |
| CN112989433A (en) * | 2021-02-24 | 2021-06-18 | 上海中通吉网络技术有限公司 | Verification method based on Java Bean verification rule and related equipment |
| CN113760681A (en) * | 2021-03-10 | 2021-12-07 | 中科天玑数据科技股份有限公司 | Unified SQL (structured query language) -based multi-source heterogeneous data quality verification method and system |
| CN114936150A (en) * | 2022-04-27 | 2022-08-23 | 浙江浩瀚能源科技有限公司 | Big data stream synchronization and monitoring test method, device and storage medium |
| US11693884B2 (en) | 2020-03-04 | 2023-07-04 | Hcl Technologies Limited | Adapative system for processing distributed data files and a method thereof |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120310904A1 (en) * | 2011-06-01 | 2012-12-06 | International Business Machine Corporation | Data validation and service |
| CN103020318A (en) * | 2013-01-10 | 2013-04-03 | 北京锐安科技有限公司 | Method for maintenance of database tables in database |
| CN105095329A (en) * | 2014-11-20 | 2015-11-25 | 北京航天金盾科技有限公司 | Method for checking population data |
| CN105321047A (en) * | 2015-11-10 | 2016-02-10 | 中国电力科学研究院 | Multi-dimensional verification method for schedule plan data |
| US20160063050A1 (en) * | 2014-08-28 | 2016-03-03 | Joerg Schoen | Database Migration Consistency Checker |
-
2017
- 2017-10-31 CN CN201711053946.4A patent/CN107992519A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120310904A1 (en) * | 2011-06-01 | 2012-12-06 | International Business Machine Corporation | Data validation and service |
| CN103020318A (en) * | 2013-01-10 | 2013-04-03 | 北京锐安科技有限公司 | Method for maintenance of database tables in database |
| US20160063050A1 (en) * | 2014-08-28 | 2016-03-03 | Joerg Schoen | Database Migration Consistency Checker |
| CN105095329A (en) * | 2014-11-20 | 2015-11-25 | 北京航天金盾科技有限公司 | Method for checking population data |
| CN105321047A (en) * | 2015-11-10 | 2016-02-10 | 中国电力科学研究院 | Multi-dimensional verification method for schedule plan data |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108804613A (en) * | 2018-05-30 | 2018-11-13 | 国网山东省电力公司经济技术研究院 | A kind of Various database real time fusion system and its fusion method |
| CN109597828A (en) * | 2018-09-29 | 2019-04-09 | 阿里巴巴集团控股有限公司 | A kind of off-line data checking method, device and server |
| CN109597828B (en) * | 2018-09-29 | 2023-04-25 | 蚂蚁金服（杭州）网络技术有限公司 | Offline data checking method, device and server |
| CN109299074A (en) * | 2018-10-30 | 2019-02-01 | 珠海伟诚科技股份有限公司 | A kind of data verification method and system based on templating data base view |
| CN109299074B (en) * | 2018-10-30 | 2022-03-04 | 珠海伟诚科技股份有限公司 | Data verification method and system based on templated database view |
| US11693884B2 (en) | 2020-03-04 | 2023-07-04 | Hcl Technologies Limited | Adapative system for processing distributed data files and a method thereof |
| CN112001710A (en) * | 2020-09-07 | 2020-11-27 | 山东钢铁集团日照有限公司 | Big data reading and integrating system in steel product production process |
| CN112632052A (en) * | 2020-12-29 | 2021-04-09 | 中通服公众信息产业股份有限公司 | Heterogeneous data sharing method and intelligent sharing system |
| CN112632052B (en) * | 2020-12-29 | 2022-10-28 | 中通服公众信息产业股份有限公司 | Heterogeneous data sharing method and intelligent sharing system |
| CN112989433A (en) * | 2021-02-24 | 2021-06-18 | 上海中通吉网络技术有限公司 | Verification method based on Java Bean verification rule and related equipment |
| CN113760681A (en) * | 2021-03-10 | 2021-12-07 | 中科天玑数据科技股份有限公司 | Unified SQL (structured query language) -based multi-source heterogeneous data quality verification method and system |
| CN114936150A (en) * | 2022-04-27 | 2022-08-23 | 浙江浩瀚能源科技有限公司 | Big data stream synchronization and monitoring test method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107992519A (en) | The multi-source heterogeneous data verification system and method for a kind of smart grid-oriented big data | |
| CN103838672B (en) | The automated testing method of a kind of all-purpose financial statements and device | |
| CN104778124B (en) | A kind of software application automated testing method | |
| CN105320690B (en) | A kind of statistics list rapid generation and system based on metadata | |
| CN104572122B (en) | A kind of generating means and method of software application data | |
| CN101867501B (en) | Method and system for automatically testing consistence of SNMP (Simple Network Management Protocol) interface information model | |
| CN108628748B (en) | Automatic test management method and automatic test management system | |
| CN107579836A (en) | A kind of parameter verification method and apparatus and NM server | |
| CN107145622A (en) | A kind of configurable product model detection method for quality | |
| CN108959068A (en) | Software interface testing method, device and storage medium | |
| CN111651431A (en) | Database service oriented management flow standardization method | |
| CN107886296B (en) | Collaborative auditing method between heterogeneous PDM systems | |
| CN107783902A (en) | A kind of Selenium automated testing methods and system from coding | |
| CN105760296A (en) | Automation testing control method, device and terminal | |
| CN110209584A (en) | A kind of automatic generation of test data and relevant apparatus | |
| CN114123482A (en) | Main plant station information joint debugging decoupling method | |
| CN105930344B (en) | A kind of database application system quick development platform based on product development process | |
| CN107797910A (en) | A kind of evaluation method of dispatch automated system software quality | |
| CN111190814B (en) | Method and device for generating software test case, storage medium and terminal | |
| CN104615928B (en) | Measurement and examination method, verification system and server | |
| CN109471791A (en) | A kind of demand response communication consistency testing system and method | |
| CN109800402A (en) | A kind of facility information processing method and system for process simulation | |
| CN112732237B (en) | Method and system for constructing code-free development technology model | |
| CN104123394B (en) | Processing method for the report file of database | |
| CN106649078B (en) | Application interface test method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: No. 15, Qinghe Xiaoying East Road, Haidian District, Beijing Applicant after: CHINA ELECTRIC POWER RESEARCH INSTITUTE Co.,Ltd. Applicant after: STATE GRID CORPORATION OF CHINA Address before: No. 15, Qinghe Xiaoying East Road, Haidian District, Beijing Applicant before: CHINA ELECTRIC POWER RESEARCH INSTITUTE Co.,Ltd. Applicant before: State Grid Corporation of China |
|
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20180906 Address after: No. 15, Qinghe Xiaoying East Road, Haidian District, Beijing Applicant after: CHINA ELECTRIC POWER RESEARCH INSTITUTE Co.,Ltd. Applicant after: STATE GRID CORPORATION OF CHINA Applicant after: STATE GRID FUJIAN ELECTRIC POWER Co.,Ltd. Applicant after: STATE GRID SHANGHAI MUNICIPAL ELECTRIC POWER Co. Address before: No. 15, Qinghe Xiaoying East Road, Haidian District, Beijing Applicant before: CHINA ELECTRIC POWER RESEARCH INSTITUTE Co.,Ltd. Applicant before: State Grid Corporation of China |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180504 |