CN109766352A - The method and system that a kind of pair of heterogeneous data source is uniformly processed - Google Patents
The method and system that a kind of pair of heterogeneous data source is uniformly processed Download PDFInfo
- Publication number
- CN109766352A CN109766352A CN201811379218.7A CN201811379218A CN109766352A CN 109766352 A CN109766352 A CN 109766352A CN 201811379218 A CN201811379218 A CN 201811379218A CN 109766352 A CN109766352 A CN 109766352A
- Authority
- CN
- China
- Prior art keywords
- data
- request end
- module
- uniformly processed
- target database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses the method and system that a kind of pair of heterogeneous data source is uniformly processed, method is comprising steps of acquisition request end data；By request end data acquisition user information and user is authenticated；Analysis request end data determines the target database of request end data；Request end data are converted into the corresponding dialect of target database；Creation and the connection of target database simultaneously execute request end data by the dialect after conversion and return to implementing result to request end；System includes data acquisition module, data resolution module, permission configuration module, routing resolution module, syntax conversion module and execution module；The present invention is based on MySql binary signaling protocol communications, by the way that standard Sql sentence is converted into target database dialect, it realizes using unified Sql92 standard sentence as entrance, operate variety classes database and returned data, cost of implementation and technical difficulty that the heterogeneous data source of integration across database is uniformly processed are effectively reduced, the efficiency that heterogeneous data source is uniformly processed is improved.
Description
Technical field
The present invention relates to database data processing technology fields, are uniformly processed more particularly, to a kind of pair of heterogeneous data source
Method and system.
Background technique
Database is many kinds of on the market at present, such as Oracle, MySql, SqlServer, Hbase, Hive, and every kind
Database has the syntactic property of itself, in the today for having marched toward big data era, has many system data needs to be merged into
Row operation, can only just can be carried out operation by the way that data pick-up to be merged into after same database at present, and process is complicated, and technology is difficult
Degree is big, and cost of implementation is high；Therefore need one kind can be by the method for the sql grammatical operations different data sources of unified standard.
Summary of the invention
To solve the above-mentioned problems, the invention proposes the method and system that a kind of pair of heterogeneous data source is uniformly processed, bases
In the standard communication protocol of setting, multitype database is accessed with a kind of standard syntax, realizes being uniformly processed for heterogeneous data source.
To achieve the goals above, the present invention adopts the following technical scheme:
Specifically, the method that a kind of pair of heterogeneous data source is uniformly processed, comprising the following steps:
S1. acquisition request end data；
S2. by request end data acquisition user information and user is authenticated；
S3. analysis request end data determines the target database of request end data；
S4. request end data are converted into the corresponding dialect of target database；
S5. it creates the connection with target database and executes request end data by the dialect after conversion and return and execute knot
Fruit is to request end.
Further, the step S1 further include:
S11. the standard communication protocol communicated with request end is set；
S12. the request end data that acquisition request end is sent；
S13. by the standard communication protocol analysis request end data and user information is obtained.
Further, the request end data is the Sql sentence of Sql92 standard.
Further, the standard communication protocol set is MySql binary signaling agreement.
Further, the step S2 further include:
S21. the permissions list of user is obtained by the user information obtained；
S22. action type is judged by the request end data obtained, and judges whether user has correlation according to permissions list
Permission returns to lack of competence and prompts to request end if lack of competence, if having permission, executes S3.
Further, the step S3 further include: parse the request end data, the table letter in acquisition request end data
Breath, according to table acquisition of information configuration file, finds the database information where corresponding table, determines target database.
Further, the step S5 further include:
S51. the connection with target database is created；
S52. request end data is executed by the dialect after conversion and returns to implementing result；
S53. it obtains implementing result and implementing result is encapsulated as to the data packet of standard communication protocol；
S54. request end is returned to by the standard communication protocol data packet that standard communication protocol encapsulates step S53.
Specifically, the system that a kind of pair of heterogeneous data source is uniformly processed, the system comprises data acquisition modules, data solution
Analyse module, permission configuration module, routing resolution module, syntax conversion module and execution module；
The data acquisition module is used to carry out data communication by the standard agreement of setting and request end, obtains external defeated
Enter data；
The data resolution module obtains user information for parsing to the data that data acquisition module is got；
The permission configuration module is used to obtain user right according to the user information got and parses input data
Operational order simultaneously judges whether user has corresponding authority；
The routing resolution module is used to obtain the target database information of input data, determines the number of targets of input data
According to source；
The syntax conversion module is used to be converted into corresponding target data according to the target data source of input data
Library dialect data simultaneously return to the dialect data after conversion；
The execution module is used to obtain the dialect data after conversion and creates the connection with target database, passes through conversion
Dialect data afterwards execute corresponding operating and return the result to request end.
Compared with prior art, the beneficial effects of the present invention are:
Based on MySql binary signaling protocol communication, realizes user authority management, number of targets is determined by Sql sentence
According to library；Based on standard Sql sentence is converted into target database dialect, realize using unified Sql92 standard sentence as entrance,
Variety classes database and returned data are operated, the cost of implementation that the heterogeneous data source of integration across database is uniformly processed is effectively reduced
And technical difficulty, improve the efficiency that heterogeneous data source is uniformly processed.
Detailed description of the invention
Fig. 1 is the method flow diagram that a kind of pair of heterogeneous data source of the invention is uniformly processed；
Fig. 2 is the system structure diagram that a kind of pair of heterogeneous data source of the invention is uniformly processed；
Fig. 3 is the embodiment of the present invention flow chart.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.The present invention being usually described and illustrated herein in the accompanying drawings is implemented
The component of example can be arranged and be designed with a variety of different configurations.
Therefore, the detailed description of the embodiment of the present invention provided in the accompanying drawings is not intended to limit below claimed
The scope of the present invention, but be merely representative of selected embodiment of the invention.Based on the embodiments of the present invention, this field is common
Technical staff's every other embodiment obtained without creative efforts belongs to the model that the present invention protects
It encloses.
It should also be noted that similar label and letter indicate similar terms in following attached drawing, therefore, once a certain Xiang Yi
It is defined in a attached drawing, does not then need that it is further defined and explained in subsequent attached drawing, term " includes ", " packet
Containing " or any other variant thereof is intended to cover non-exclusive inclusion, so that including the process of a series of elements, side
Method, article or equipment not only include those elements, but also including other elements that are not explicitly listed, or further include
For elements inherent to such a process, method, article, or device.In the absence of more restrictions, by sentence " including one
It is a ... " limit element, it is not excluded that there is also in addition in the process, method, article or apparatus that includes the element
Identical element.
As shown in Figure 1, the method that a kind of pair of heterogeneous data source is uniformly processed, comprising the following steps:
S1. acquisition request end data；
S2. by request end data acquisition user information and user is authenticated；
S3. analysis request end data determines the target database of request end data；
S4. request end data are converted into the corresponding dialect of target database；
S5. it creates the connection with target database and executes request end data by the dialect after conversion and return and execute knot
Fruit is to request end.
As shown in Fig. 2, the system that a kind of pair of heterogeneous data source is uniformly processed, system includes data acquisition module, data solution
Analyse module, permission configuration module, routing resolution module, syntax conversion module and execution module；
Data acquisition module is used to carry out data communication by the standard agreement of setting and request end, obtains external input number
According to；
Data resolution module obtains user information for parsing to the data that data acquisition module is got；
Permission configuration module is used to obtain user right according to the user information got and parses the operation of input data
It orders and judges whether user has corresponding authority；
Routing resolution module is used to obtain the target database information of input data, determines the target data of input data
Source；
Syntax conversion module is used to be converted into corresponding target database side according to the target data source of input data
Speech data simultaneously return to the dialect data after conversion；
Execution module is used to obtain the dialect data after conversion and creates the connection with target database, after conversion
Dialect data execute corresponding operating and return the result to request end.
As shown in figure 3, step S1 further include:
S11. the standard communication protocol that setting built-in system is communicated with request end, standard communication protocol are MySql binary system
Communication protocol；
S12. request end is drivingly connected service by the jdbc of MySql, what data acquisition module acquisition request end was passed to
Sql92 standard Sql sentence；
S13. data resolution module obtains user information by MySql binary signaling protocol analysis Sql sentence.
Step S2 further include:
S21. permission configuration module connects right discriminating system by the user information obtained, obtains the permissions list of user；
S22. permission configuration module by obtain Sql action statement before character, judgement be select, create,
The operation such as drop, delete, update, then judges whether user has corresponding authority according to the permissions list got, if nothing
Permission then returns to lack of competence and prompts to request end, if having permission, executes S3.
Step S3 further include: analysis request end data, routing resolution module are obtained by the Sql syntax parsing to input
Table information in Sql sentence finds the database letter where corresponding table by configuration file according to table acquisition of information configuration file
Breath, determines target database.
Step S4 is specially by Sql sentence and target database type transfer to syntax conversion module, syntax conversion module
Incoming Sql sentence is converted into dialect and the return of correspondence database.
Step S5 further include:
S51. execution module creation is connect with the jdbc of target database, can be led to by jdbc:phoenix connection Hbase
Cross jdbc:hive2 connection Hive；
S52. the dialect after conversion is put into the connection with target database and is executed, and obtain the execution knot of return
Fruit；
S53. the implementing result that will acquire is encapsulated as the byte packet of MySql agreement；
S54. packaged byte packet is sent to request end by way of nio.
In embodiment provided herein, it should be understood that disclosed method and system, it can also be by other
Mode realize.System embodiment described above is only schematical, and each box in flowchart or block diagram can be with
A part of a module, section or code is represented, a part of the module, section or code includes one or more
Executable instruction for implementing the specified logical function.In addition, each functional module in embodiments of the present invention can integrate
An independent part is formed together, is also possible to modules individualism, it can also be with two or more module collection
It is shaped to an independent part.If the function is realized in the form of software function module and is sold as independent product
Or it in use, can store in a computer readable storage medium.Based on this understanding, technical solution of the present invention
Substantially the part of the part that contributes to existing technology or the technical solution can be in the form of software products in other words
It embodies, which is stored in a storage medium, including some instructions are used so that a computer
Equipment (can be personal computer, server or the network equipment etc.) execute the method for the embodiment of the present invention whole or
Part steps.And storage medium above-mentioned include: USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only Memory),
Random access memory (RAM, Random Access Memory), magnetic or disk etc. be various to can store program code
Medium.
Claims (8)
1. the method that a kind of pair of heterogeneous data source is uniformly processed, which comprises the following steps:
S1. acquisition request end data；
S2. by request end data acquisition user information and user is authenticated；
S3. analysis request end data determines the target database of request end data；
S4. request end data are converted into the corresponding dialect of target database；
S5. it creates the connection with target database and request end data is executed by the dialect after conversion and returns to implementing result extremely
Request end.
2. the method that a kind of pair of heterogeneous data source according to claim 1 is uniformly processed, which is characterized in that the step S1
Further include:
S11. the standard communication protocol communicated with request end is set；
S12. the request end data that acquisition request end is sent；
S13. by the standard communication protocol analysis request end data and user information is obtained.
3. the method that a kind of pair of heterogeneous data source according to claim 1 is uniformly processed, which is characterized in that the request end
Data are the Sql sentence of Sql92 standard.
4. the method that a kind of pair of heterogeneous data source according to claim 2 is uniformly processed, which is characterized in that the setting
Standard communication protocol is MySql binary signaling agreement.
5. the method that a kind of pair of heterogeneous data source according to claim 1 is uniformly processed, which is characterized in that the step S2
Further include:
S21. the permissions list of user is obtained by the user information obtained；
S22. action type is judged by the request end data obtained, and judges whether user has correlative weight according to permissions list
Limit, if lack of competence, returns to lack of competence and prompts to request end, if having permission, execute S3.
6. the method that a kind of pair of heterogeneous data source according to claim 1 is uniformly processed, which is characterized in that the step S3
Further include: the request end data is parsed, the table information in acquisition request end data is looked for according to table acquisition of information configuration file
Database information to where corresponding table, determines target database.
7. the method that a kind of pair of heterogeneous data source according to claim 2 is uniformly processed, which is characterized in that the step S5
Further include:
S51. the connection with target database is created；
S52. request end data is executed by the dialect after conversion and returns to implementing result；
S53. it obtains implementing result and implementing result is encapsulated as to the data packet of standard communication protocol；
S54. request end is returned to by the standard communication protocol data packet that standard communication protocol encapsulates step S53.
8. the system that a kind of pair of heterogeneous data source is uniformly processed, using one kind as described in claim 1-7 any one to different
The method that structure data source is uniformly processed, which is characterized in that the system comprises data acquisition module, data resolution module, permissions
Configuration module, routing resolution module, syntax conversion module and execution module；
The data acquisition module is used to carry out data communication by the standard agreement of setting and request end, obtains external input number
According to；
The data resolution module obtains user information for parsing to the data that data acquisition module is got；
The permission configuration module is used to obtain user right according to the user information got and parses the operation of input data
It orders and judges whether user has corresponding authority；
The routing resolution module is used to obtain the target database information of input data, determines the target data of input data
Source；
The syntax conversion module is used to be converted into corresponding target database side according to the target data source of input data
Speech data simultaneously return to the dialect data after conversion；
The execution module is used to obtain the dialect data after conversion and creates the connection with target database, after conversion
Dialect data execute corresponding operating and return the result to request end.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811379218.7A CN109766352A (en) | 2018-11-19 | 2018-11-19 | The method and system that a kind of pair of heterogeneous data source is uniformly processed |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811379218.7A CN109766352A (en) | 2018-11-19 | 2018-11-19 | The method and system that a kind of pair of heterogeneous data source is uniformly processed |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109766352A true CN109766352A (en) | 2019-05-17 |
Family
ID=66449618
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811379218.7A Pending CN109766352A (en) | 2018-11-19 | 2018-11-19 | The method and system that a kind of pair of heterogeneous data source is uniformly processed |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109766352A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110633292A (en) * | 2019-09-19 | 2019-12-31 | 上海依图网络科技有限公司 | Query method, device, medium, equipment and system for heterogeneous database |
| CN110781205A (en) * | 2019-09-12 | 2020-02-11 | 成都四方伟业软件股份有限公司 | JDBC-based database direct-checking method, device and system |
| CN111008211A (en) * | 2019-12-06 | 2020-04-14 | 北京百分点信息科技有限公司 | Visual interface creating method and device, readable storage medium and electronic equipment |
| CN111125218A (en) * | 2019-12-13 | 2020-05-08 | 成都安恒信息技术有限公司 | Database compatibility method based on protocol analysis and compatibility proxy device thereof |
| CN111488391A (en) * | 2020-04-09 | 2020-08-04 | 中国银行股份有限公司 | Data processing method and device |
| CN111858760A (en) * | 2020-07-13 | 2020-10-30 | 中国工商银行股份有限公司 | Data processing method and device for heterogeneous database |
| CN112087524A (en) * | 2020-09-21 | 2020-12-15 | 烟台云朵软件有限公司 | Unified access method and device for multi-source heterogeneous medical data |
| CN112905567A (en) * | 2021-03-23 | 2021-06-04 | 杭州沃趣科技股份有限公司 | Database replacement method, device, system and medium based on network protocol conversion |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107066499A (en) * | 2016-12-30 | 2017-08-18 | 江苏瑞中数据股份有限公司 | The data query method of multi-source data management and visualization system is stored towards isomery |
-
2018
- 2018-11-19 CN CN201811379218.7A patent/CN109766352A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107066499A (en) * | 2016-12-30 | 2017-08-18 | 江苏瑞中数据股份有限公司 | The data query method of multi-source data management and visualization system is stored towards isomery |
Non-Patent Citations (2)
| Title |
|---|
| 姚莉等: "《数据库系统实践指南》", 31 December 2017 * |
| 笑天居士: "Cobar的架构与实践", 《CSND博客HTTPS://BLOG.CSDN.NET/XTJSXTJ/ARTICLE/DETAILS/15809999》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110781205A (en) * | 2019-09-12 | 2020-02-11 | 成都四方伟业软件股份有限公司 | JDBC-based database direct-checking method, device and system |
| CN110633292A (en) * | 2019-09-19 | 2019-12-31 | 上海依图网络科技有限公司 | Query method, device, medium, equipment and system for heterogeneous database |
| CN110633292B (en) * | 2019-09-19 | 2022-06-21 | 上海依图网络科技有限公司 | Query method, device, medium, equipment and system for heterogeneous database |
| CN111008211A (en) * | 2019-12-06 | 2020-04-14 | 北京百分点信息科技有限公司 | Visual interface creating method and device, readable storage medium and electronic equipment |
| CN111008211B (en) * | 2019-12-06 | 2023-04-11 | 北京百分点科技集团股份有限公司 | Visual interface creating method and device, readable storage medium and electronic equipment |
| CN111125218A (en) * | 2019-12-13 | 2020-05-08 | 成都安恒信息技术有限公司 | Database compatibility method based on protocol analysis and compatibility proxy device thereof |
| CN111488391A (en) * | 2020-04-09 | 2020-08-04 | 中国银行股份有限公司 | Data processing method and device |
| CN111858760A (en) * | 2020-07-13 | 2020-10-30 | 中国工商银行股份有限公司 | Data processing method and device for heterogeneous database |
| CN111858760B (en) * | 2020-07-13 | 2024-03-22 | 中国工商银行股份有限公司 | Data processing method and device for heterogeneous database |
| CN112087524A (en) * | 2020-09-21 | 2020-12-15 | 烟台云朵软件有限公司 | Unified access method and device for multi-source heterogeneous medical data |
| CN112905567A (en) * | 2021-03-23 | 2021-06-04 | 杭州沃趣科技股份有限公司 | Database replacement method, device, system and medium based on network protocol conversion |
| CN112905567B (en) * | 2021-03-23 | 2023-04-07 | 杭州沃趣科技股份有限公司 | Database replacement method, device, system and medium based on network protocol conversion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109766352A (en) | The method and system that a kind of pair of heterogeneous data source is uniformly processed | |
| CN111259036B (en) | Cross-library and cross-table query method, device, server and storage medium | |
| CN110443059B (en) | Data protection method and device | |
| US9336203B2 (en) | Semantics-oriented analysis of log message content | |
| CN107292170B (en) | Method, device and system for detecting SQL injection attack | |
| CN106649810A (en) | Ajax-based news webpage dynamic data grabbing method and system | |
| CN103559444A (en) | Sql (Structured query language) injection detection method and device | |
| CN113064583B (en) | Multistage page route jump method, device, computer equipment and storage medium | |
| CN111683066A (en) | Heterogeneous system integration method and device, computer equipment and storage medium | |
| US8489631B2 (en) | Distributing a query | |
| CN109861998B (en) | Plug-in type dynamic analysis system and method based on Beidou short message protocol | |
| CN112860727A (en) | Data query method, device, equipment and medium based on big data query engine | |
| CN112084167A (en) | Authority filtering method and device and storage medium | |
| EP4024251A1 (en) | Method for verifying vulnerabilities of network devices using cve entries | |
| CN109241157B (en) | Data calling method and device, communication equipment and storage medium | |
| CN112235124B (en) | Method and device for configuring pico-cell, storage medium and electronic device | |
| CN114416776A (en) | Data self-defined query statistical method | |
| CN106020923A (en) | SELinux strategy compiling method and system | |
| CN110489207A (en) | Verification method, apparatus and computer readable storage medium | |
| CN112883088B (en) | Data processing method, device, equipment and storage medium | |
| CN108845995A (en) | Data processing method, device, storage medium and electronic device | |
| CN111259039A (en) | Database operation method, device, equipment and computer readable storage medium | |
| CN116991929A (en) | Micro-service system based on big hospital data | |
| CN114764330A (en) | Data blood margin analysis method and device, electronic equipment and computer readable storage medium | |
| CN113934430A (en) | Data retrieval analysis method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190517 |