CN113051272A

CN113051272A - Database table structure construction method and device, computer equipment and storage medium

Info

Publication number: CN113051272A
Application number: CN202110338932.7A
Authority: CN
Inventors: 肖茂林
Original assignee: Shenzhen Hongtu Chuangcheng Technology Co ltd
Current assignee: Shenzhen Hongtu Chuangcheng Technology Co ltd
Priority date: 2021-03-30
Filing date: 2021-03-30
Publication date: 2021-06-29

Abstract

The embodiment of the invention discloses a method and a device for constructing a database table structure, computer equipment and a storage medium. The method comprises the following steps: formulating an acquisition strategy and sending the acquisition strategy to an application server so that the application server acquires user behavior data and application transmission data to obtain intermediate data, and specifically, setting an actual acquisition strategy by the application server in a switch mode according to the acquisition strategy; intercepting the access data by the application server by adopting a byte enhancement technology to obtain intercepted data; the application server acquires user behavior data and application transmission data of the intercepted data according to an actual acquisition strategy to obtain intermediate data; aggregating the intermediate data; filtering the intermediate data based on the application database access interface to obtain target data; and analyzing according to the target data and constructing a database table structure. By implementing the method provided by the embodiment of the invention, a database table structure can be more comprehensively, automatically, intelligently and completely constructed, and the safety is high.

Description

Database table structure construction method and device, computer equipment and storage medium

Technical Field

The invention relates to a data processing method, in particular to a database table structure construction method, a database table structure construction device, computer equipment and a storage medium.

Background

When auditing a database and identifying sensitive data of the database, database table structure information is needed, and generally includes information such as a database IP (Internet Protocol) or a domain name, a database name, a table name, a field name, and the like.

The existing methods for constructing a database table structure mainly include two methods, one is to use network traffic to analyze a protocol related to a database and extract an SQL (Structured Query Language) statement, and then construct database table structure information by analyzing SQL syntax and semantics, and the other is to use an authorized database account to connect to the database to pull the table structure information. For the first mode, when the bypass mode is used for deploying and extracting network traffic, due to the deployment of port mirror image or optical splitting equipment and the limitation of the physical position of the deployment, a large amount of equipment is required to be deployed when a large-scale network is deployed, and the management becomes extremely difficult; when the agent mode is used for deploying and extracting network flow, the design of a network architecture is changed, the complexity of the network architecture is improved, the difficulty of network operation and maintenance is improved, the efficiency of troubleshooting is reduced, and the reliability and the availability of the whole network are reduced; whether the deployment is performed in the bypass mode or the proxy mode, the deployment is limited by the physical limitations of the port mirror image and the optical splitter, the deployment and implementation cannot be performed in the cloud environment, and the access of a data acquisition object, namely a database, cannot be managed and controlled through an interactive data acquisition strategy, such as an application interface, an interface field, an acquisition frequency, a data acquisition amount and the like; for an open-source database such as mysql, maridb, PostgreSQL and the like, SQL statements and returned data can be extracted during network traffic analysis, but for a non-open-source database such as oracle, DB2, dreams and the like, because a private protocol is not used to disclose data encoding rules thereof, the executed SQL statements and transmitted data need to be extracted and decoded, but existing products using the network traffic analysis database protocol technology are difficult to completely extract the SQL statements and the returned data; the construction of the database table structure information is incomplete due to the fact that the extracted SQL statements are incomplete; when the database table structure information is constructed, the constructed database table structure is incomplete because the analyzed SQL statement does not contain information such as field types, field lengths and the like. The second method is that an account is established and authorized for each connected database, and then the account and a password are input into the product to be connected for connection, if hundreds of database system connections are performed, a challenge is presented to database operation and maintenance personnel and product maintenance personnel, and database identification and database desensitization products in the market at present belong to the type; because of the security requirement, the account password of the database needs to be modified periodically, so how to modify the account password of the database periodically and synchronize the database with the product safely also becomes a big problem of safe operation and maintenance, and if the security protection measures are not made, the account password is possibly leaked, so that a security event of data leakage is caused.

Therefore, it is necessary to design a new method, which has no influence on the existing network architecture, ensures the reliability and availability of the network, reduces the difficulty of network operation and maintenance, reduces the efficiency of troubleshooting, constructs the database table structure more comprehensively, automatically, intelligently and completely, and has high security.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provides a database table structure construction method, a database table structure construction device, computer equipment and a storage medium.

In order to achieve the purpose, the invention adopts the following technical scheme: the database table structure construction method comprises the following steps:

an acquisition strategy is formulated and issued to an application server, so that the application server acquires user behavior data and application transmission data to obtain intermediate data;

aggregating the intermediate data;

filtering the intermediate data based on an application database access interface to obtain target data;

analyzing according to the target data and constructing a database table structure;

the method comprises the following steps of making an acquisition strategy, issuing the acquisition strategy to an application server, and acquiring user behavior data and application transmission data by the application server to obtain intermediate data, wherein the steps comprise:

an acquisition strategy is formulated and issued to an application server, so that the application server adopts a switch mode to set an actual acquisition strategy according to the acquisition strategy; intercepting the access data by the application server by adopting a byte enhancement technology to obtain intercepted data; and acquiring user behavior data and application transmission data for the intercepted data by the application server according to the actual acquisition strategy to obtain intermediate data.

The further technical scheme is as follows: the filtering the intermediate data based on the application database access interface to obtain target data, comprising:

extracting data meeting preset conditions in the intermediate data to obtain first data;

classifying the first data to obtain second data and third data;

analyzing the second data and the third data to obtain target data;

the preset condition comprises that an application interface is a database access interface type.

The further technical scheme is as follows: the classifying the first data to obtain second data and third data includes:

and classifying the SQL statement of the first data and response information returned after query to obtain second data and third data, wherein the second data comprises the SQL statement information, and the third data comprises the response information returned after query.

The further technical scheme is as follows: the analyzing the second data and the third data to obtain target data includes:

performing lexical analysis and syntactic analysis on the second data to obtain a lexical analysis result and a syntactic analysis result;

associating the lexical analysis result and the grammatical analysis result to obtain associated information;

extracting database table structure information of the third data to obtain table structure information;

wherein the target data includes the association information and the table structure information.

The further technical scheme is as follows: the constructing of the database table structure according to the target data comprises:

merging single target data by taking a single database as a unit to obtain a merging result;

and carrying out deduplication processing and association analysis on the merged result to obtain a database table structure.

The further technical scheme is as follows: after the database table structure is constructed according to the target data, the method further comprises the following steps:

and dynamically managing the database table structure.

The further technical scheme is as follows: the dynamically managing the database table structure includes:

and carrying out duplicate removal on the target data obtained by the single analysis and the information stored in the database, and then updating the target data into the database.

The invention also provides a database table structure construction device, which comprises:

the data acquisition unit is used for formulating an acquisition strategy and issuing the acquisition strategy to the application server so as to acquire user behavior data and application transmission data by the application server to obtain intermediate data;

the aggregation unit is used for aggregating the intermediate data;

the filtering unit is used for filtering the intermediate data based on an application database access interface to obtain target data;

the construction unit is used for analyzing according to the target data and constructing a database table structure;

the data acquisition unit is used for formulating an acquisition strategy and sending the acquisition strategy to the application server so that the application server sets an actual acquisition strategy in a switching mode according to the acquisition strategy; intercepting the access data by the application server by adopting a byte enhancement technology to obtain intercepted data; and acquiring user behavior data and application transmission data for the intercepted data by the application server according to the actual acquisition strategy to obtain intermediate data.

The invention also provides computer equipment which comprises a memory and a processor, wherein the memory is stored with a computer program, and the processor realizes the method when executing the computer program.

The invention also provides a storage medium storing a computer program which, when executed by a processor, is operable to carry out the method as described above.

Compared with the prior art, the invention has the beneficial effects that: the invention sets the data acquisition strategy at the management terminal, updates the data acquisition strategy to the application client, generates an actual acquisition strategy by combining the acquisition strategy with the application client deployed on the application server in a switch mode, intercepts the data of a specific interface by adopting a byte code enhancement technology, acquires the corresponding data by combining the actual acquisition strategy, realizes the purpose of acquiring user behavior data and application transmission data in a control mode, is more comprehensive, simple and flexible to acquire, analyzes and constructs a database table structure based on the comprehensively acquired data, realizes no influence on the existing network architecture, ensures the reliability and the availability of the network, reduces the difficulty of network operation and maintenance, reduces the efficiency of fault removal, constructs the database table structure more comprehensively, automatically, intelligently and completely, and has high safety.

The invention is further described below with reference to the accompanying drawings and specific embodiments.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic view of an application scenario of a database table structure construction method provided by an embodiment of the present invention;

fig. 2 is a schematic flow chart of a database table structure construction method according to an embodiment of the present invention;

FIG. 3 is a schematic view of a sub-flow of a database table structure construction method according to an embodiment of the present invention;

FIG. 4 is a schematic sub-flow diagram of a database table structure construction method according to an embodiment of the present invention;

FIG. 5 is a schematic view of a sub-flow of a database table structure construction method according to an embodiment of the present invention;

FIG. 6 is a schematic sub-flow diagram of a database table structure construction method according to an embodiment of the present invention;

FIG. 7 is a flowchart illustrating a method for building a database table structure according to another embodiment of the present invention;

FIG. 8 is a schematic block diagram of a database table structure building apparatus according to an embodiment of the present invention;

FIG. 9 is a schematic block diagram of a filtering unit of a database table structure building apparatus according to an embodiment of the present invention;

FIG. 10 is a schematic block diagram of an analysis subunit of the database table structure building apparatus provided by the embodiment of the present invention;

fig. 11 is a schematic block diagram of a building unit of a database table structure building apparatus according to an embodiment of the present invention;

FIG. 12 is a schematic block diagram of a database table structure constructing apparatus according to another embodiment of the present invention;

FIG. 13 is a schematic block diagram of a computer device provided by an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

Referring to fig. 1 and fig. 2, fig. 1 is a schematic view of an application scenario of a database table structure construction method according to an embodiment of the present invention. Fig. 2 is a schematic flowchart of a database table structure construction method according to an embodiment of the present invention. The database table structure construction method is applied to a management end which is a server and is communicated with an agent installed on an application server, namely an application client, the management end is an independent server, the management end is used for communicating and managing the application client and comprises data interaction, wherein an acquisition strategy set by the management end is pushed to the application client in real time, an acquisition switch arranged in the application client on the application server is combined for setting an actual acquisition strategy, when a user terminal initiates access to the application server and the accessed data passes through a specified interface of the application client, the application client intercepts the data by adopting a byte enhancement technology and acquires the intercepted data by adopting the actual acquisition strategy, so that the management end extracts table structure information of the acquired data and generates a corresponding database table structure, and after data are acquired each time, target data can be obtained through analysis, and the constructed database is dynamically managed.

It should be noted that fig. 1 only illustrates one application server and one management-side server, and in the actual operation process, a plurality of application servers and a plurality of management-side servers may be simultaneously executed.

Fig. 2 is a schematic flow chart of a database table structure construction method according to an embodiment of the present invention. As shown in fig. 2, the method includes the following steps S110 to S140.

And S110, making an acquisition strategy, and sending the acquisition strategy to an application server so that the application server acquires user behavior data and application transmission data to obtain intermediate data.

In this embodiment, the intermediate data refers to a result obtained by screening and extracting user behavior data and transmission data of intercepted data by using a specific policy.

In an embodiment, referring to fig. 3, the step S110 may include steps S111 to S113.

And S111, making an acquisition strategy, and sending the acquisition strategy to an application server so that the application server sets an actual acquisition strategy by adopting a switch mode according to the acquisition strategy.

The acquisition strategy comprises at least one data acquisition mode of data acquisition based on user identification configuration, data acquisition based on application service configuration, data acquisition based on an application interface, data acquisition according to fields of the application interface, data acquisition based on configured frequency and data acquisition based on configured single data acquisition quantity.

The method has the advantages that the acquisition range is limited by a specific network area and a specific deployment position relative to the acquisition of network traffic, the acquisition range is limited, the embodiment deploys on the application service host in a client-side application client-side mode during deployment, the deployment is simple, the acquisition range is not limited, the deployment is not influenced by physical positions or cloud environments, and a management end with unified management performs unified configuration management on the application client-side; the management terminal can configure single or combined data acquisition strategies based on application services, user identifications, application interfaces, application interface fields, acquisition frequency and data contents to acquire user behavior data and application transmission data, controllability of data acquisition is achieved, and the strategies of the application client terminal for data acquisition can be changed by changing the acquisition strategies at the management terminal, so that controllability of data acquisition is achieved. The data acquisition strategy can be rapidly configured, fixed point data acquisition and development are not needed, manpower is saved, the development period is greatly shortened, and the work and project efficiency is improved; when in deployment, the application service host is deployed in a client mode, and a management end with unified management performs unified configuration management on the application client; the method has no influence on the architecture of an application system, has no influence on the existing network architecture, cannot reduce the reliability and the usability of the network, and cannot cause the difficulty in operation and maintenance of the network to be improved; the efficiency of troubleshooting is not reduced.

Specifically, the data acquisition mode for performing data acquisition based on the user identifier configuration refers to whether to acquire application transmission data based on the user identifier configuration, and specifies that a single user, multiple users or all users may or may not acquire user behavior data and application transmission data, where the user identifier includes an accessed user account, an accessed IP, an accessed mac, an accessed browser, and the like.

The data acquisition mode for acquiring data based on the application service configuration refers to whether user behavior data and application transmission data are acquired based on the application service configuration, and it is specified that a single or multiple or all application services may or may not acquire the user behavior data and the application transmission data.

The data acquisition mode for acquiring data based on the application interface refers to that whether the application interface acquires application transmission data or not, and a single or a plurality of or all application interfaces are designated to acquire user behavior data and application transmission data or not.

The data acquisition mode for acquiring data according to the field of the application interface refers to whether the field configuration of the application interface acquires user behavior data and application transmission data, the control is mainly performed through the interface field on the interface name, and a single or a plurality of or all application interface fields are appointed to acquire the user behavior data and the application transmission data or not.

The data acquisition mode for acquiring data based on the configured frequency is controlled by the time period for acquiring data.

The data acquisition mode for carrying out data acquisition based on the configured single data acquisition amount is mainly controlled by the number of data to be acquired.

Whether the application transmission data is collected or not is configured in any combination of the data collection modes, for example, behavior data of a certain user accessing a certain interface is allowed to be collected or user behavior data and application transmission data of a certain specified application service are not allowed to be collected, wherein the specified application service is accessed by all the users; thereby forming an acquisition strategy.

In this embodiment, the actual acquisition policy refers to a policy actually used for acquiring data, and combines the acquisition policy transmitted by the client and the set switch mode to form a plurality of data acquisition policies, and the data acquisition policies are sequentially executed according to the set priorities.

Specifically, the application client deployed on the application server sets the actual acquisition policy in a switch mode according to the acquisition policy.

In this embodiment, the switch mode includes at least one of a switch for setting data acquisition based on an application service, a switch for setting data acquisition based on a user identifier, a switch for setting data acquisition based on an application interface, and a switch for setting data acquisition based on a field of an application interface; the switch mode also includes a period threshold for data acquisition and a filtering rule based on the acquisition amount of single data.

Specifically, setting the mode of the switch for data acquisition based on the application service means that when the state is enabled, the application client opens the data acquisition function of the application service, which is a precondition that the acquisition function can normally work; when the state is closed, the application client closes the acquisition function of the application service, and all the acquisition functions of the application service are closed.

Setting a mode of a switch for data acquisition based on the user identifier means that the application client opens a data acquisition function based on the user identifier when the state is enabled; when the state is closed, the application client closes the data acquisition function based on the user identification, but other acquisition switches and data acquisition are not affected, and only user behavior data and application transmission data are not acquired based on the user identification any more.

Setting the mode of a switch for data acquisition based on the application interfaces means that when the state is enabled, data acquisition is performed on all the application interfaces by default, and acquisition or non-acquisition can be defined for data of a designated interface, wherein the designated interface can be a single interface or a plurality of interfaces or a mark of combining all interface names with acquisition or non-acquisition; when the state is off, the data acquisition function of the application interface based on the application service is closed by default, but data of a designated interface such as single or multiple or all interface names can be acquired in combination with acquired marks, and other acquisition switches and data acquisition are not affected at the moment.

Setting the mode of the switch for data acquisition based on the field of the application interface means that when the state is enabled, data acquisition is performed on all interfaces and all fields by default, and data acquisition is not performed on the designated interfaces and designated fields such as single or multiple or all interface names plus the field names of the corresponding interfaces and non-acquired marks; when the state is off, the data acquisition function based on the application interface field is turned off, but other acquisition switches and data acquisition are not affected, and only user behavior data and application transmission data are not acquired based on the application interface field any more, but the data of the appointed interface and appointed field, such as single or multiple or all interface names, the field name of the corresponding interface and the acquired mark, are acquired, and at the moment, other acquisition switches and data acquisition are not affected.

The period threshold value of data acquisition is to control the data to be acquired by counting the time period of the acquired data, and the acquired data main body is a single application service, an application interface and an interface field in the service; the filtering rule based on the acquisition amount of single data is used for filtering data number extraction of single application service, application interfaces and interface fields in the service, and the filtering is carried out in the manners of pre-data extraction, post-data extraction, random extraction, data extraction from a certain specific data point and the like.

The management terminal and the application client terminal establish a bidirectional communication channel, can send instructions to the application client terminal at any time, the application client terminal can execute according to the instructions and return execution results, and the application client terminal can also actively send information such as the state of the application client terminal to the management terminal through the bidirectional communication channel. During initial installation, after the application client and the management terminal establish communication, the management terminal can push acquisition strategies according to an application client group or a single application client, when the data acquisition strategies are updated, strategy pushing can be carried out in real time, and the application client executes different data acquisition logics according to different acquisition instructions in the acquisition strategies aiming at the received data strategies, so that the function of managing and controlling data acquisition is achieved.

And S112, intercepting the access data by the application server by adopting a byte enhancement technology to obtain the intercepted data.

In this embodiment, the intercepted data refers to data passing through a specified interface of the application client.

Specifically, an application client deployed on an application server intercepts access data by using a byte enhancement technology to obtain intercepted data.

Specifically, after the Java bytecode is generated, the generated Java bytecode is dynamically modified during the JVM loading according to rules defined in the Java bytecode, and the content of the enhanced function is added, so that access data is intercepted according to the content of the enhanced function, and intercepted data is obtained.

By using a bytecode enhancement technology, after the Java bytecode is generated, according to rules defined in the Java bytecode, the generated Java bytecode is dynamically modified when a JVM (Java Virtual Machine) loads, and a required field is added or a method function is added or a new class and interface are inherited to be implemented. The dynamically added fields, or method functions, or inheritance to realize new classes and interfaces are all enhanced functions, and the enhanced functions mainly refer to the functions of automatically extracting request parameters and return results of the method functions.

Before extracting the relevant data, it is first determined that the data of interest must pass through the method function of the class in the course of one access by the user. After determining the method functions, acquiring data before and after calling the method functions by using a bytecode enhancement technology, and acquiring an application system to be acquired according to actual requirements, where the method functions include Object first (method) and/or Object second (method);

firstly, extracting the main content of the method (string parameter) as a function of the name of the method name random string, such as sdfsdfsd (parameter); the method (string parameter) is then modified as follows: method (String parameter) support

bef or (), and

}

sdfsdfsd(paramer)；

after(){

}

}。

finally, the data collected in bef or () and after () will be stored in a variable in the first object, which is enhanced. The value in this variable can then be passed to bef in first method (String parameter) or to the after method, which concatenates the user's data throughout the lifetime of an access.

In the extracted data, there are different types of data, user identification information such as a user account, an IP, a mac, a browser, etc., user access time, an access interface, an access result, access data, access times, etc., and there are protocol fields and content data such as http and RPC (Remote Procedure Call) protocol field names, and corresponding information in the fields; the method comprises the steps of applying access time in transmission, accessing an interface, accessing a protocol, all fields in the protocol, contents corresponding to all fields in the protocol, all transmitted request data, all transmitted response data, database names, database IP (Internet protocol), database ports, database categories and the like, wherein the data are dispersed in different places in the access process of a user.

When the access data stream and the interface of user access and task scheduling call the Agent passing through the application client, the application client intercepts the flowing data by using the byte code enhancement technology, and the user behavior data and the application transmission data which are obtained from the byte code enhancement technology and analyzed by the application layer protocol are also acquired, so that the protocol does not need to be analyzed and the content is restored, and the data acquisition is not influenced by the encryption protocol and the private protocol. The user behavior data acquisition of a buried point and a second switch is not needed, so that the operation maintenance of the application and the influence on an application system are reduced, and the normal operation of the application can be further ensured; the method has no influence on the architecture of an application system and no influence on the existing network architecture.

S113, the application server collects user behavior data and application transmission data for the intercepted data according to the actual collection strategy to obtain intermediate data.

Specifically, the application client deployed on the application server acquires user behavior data and application transmission data of the intercepted data according to the actual acquisition strategy to obtain intermediate data.

Specifically, the intercepted data stream is subjected to data analysis according to an actual acquisition strategy, and required user behavior data and application transmission data are acquired.

In this embodiment, user behavior data and application transmission data are acquired for intercepted data according to different data acquisition logics in the actual acquisition strategy; wherein the different data acquisition logic performs data acquisition according to a set priority.

In addition, the user behavior data includes user identification (such as a user account, an access IP, an access mac, an access browser, and the like), acquisition time, access behavior of the user, an access interface, the number of access times, the amount of access data, and the like. The transmission data includes user identification, access time, access interface, access protocol, all fields in the protocol, contents corresponding to all fields in the protocol, all request data transmitted, all response data transmitted, database name, database IP, database port, database category, and the like.

The user behavior data range which can be collected is all applications running based on the JVM, and the user behavior data which can be collected is a user identifier; collecting time; user access behavior such as access to that service, access to that application interface, other custom user behavior events; the number of accesses; access data volume, etc.; the transmission data that can be collected is user identification, time of data transmission, protocol field and content such as http extracted and RPC (Remote Procedure Call) protocol field name and corresponding field content, request data transmitted, response data transmitted, etc.

In this embodiment, the priority is:

when a plurality of strategies are executed, processing is carried out according to the priority from high to low, and processing is carried out according to a data acquisition mode for carrying out data acquisition based on application service configuration, a data acquisition mode for carrying out data acquisition based on configured frequency, a data acquisition mode for carrying out data acquisition based on configured single data acquisition quantity, a data acquisition mode for carrying out data acquisition based on an application interface, a data acquisition mode for carrying out data acquisition according to fields of the application interface and a data acquisition mode for carrying out data acquisition based on user identification configuration.

When a plurality of strategies conflict, processing according to high priority, from high to low, according to a data acquisition mode of performing data acquisition based on application service configuration, a data acquisition mode of performing data acquisition based on configured frequency, a data acquisition mode of performing data acquisition based on configured single data acquisition amount, a data acquisition mode of performing data acquisition based on an application interface, a data acquisition mode of performing data acquisition according to fields of the application interface, and a data acquisition mode of performing data acquisition based on user identification configuration.

When the executed data acquisition strategy logic has a problem, the application client does not execute the data acquisition instruction and returns a failure result to the management terminal.

And S120, aggregating the intermediate data.

In this embodiment, the collected intermediate data is transmitted to a designated receiving location, and the collected intermediate data is aggregated, specifically, the collected intermediate data may be transmitted to a data cache space, and all the collected intermediate data are aggregated in the data cache space.

And S130, filtering the intermediate data based on the access interface of the application database to obtain target data.

In this embodiment, the target data includes associated information and table structure information, where the associated information refers to information associated with a database table structure, that is, information associated with a data table, a data field, and the like; the table structure information includes, but is not limited to, database names, table names, field types, field lengths, primary keys, indexes, and the like.

Because the data acquisition is not influenced by the encryption protocol and the private protocol and the acquired data is comprehensive and controllable, more comprehensive SQL sentences and returned data and information can be extracted, and the structure of the constructed database table is more complete.

In an embodiment, referring to fig. 4, the step S130 may include steps S131 to S133.

S131, extracting data meeting preset conditions in the intermediate data to obtain first data;

In this embodiment, the first data refers to intermediate data acquired when the acquisition policy is a data acquisition mode for acquiring data based on an application interface, and the application interface is a database access interface type. Because the collected data has corresponding collection strategies and set switch modes, the related data of the application interface which is the type of the database access interface can be quickly and accurately searched.

S132, classifying the first data to obtain second data and third data.

In this embodiment, the second data includes SQL statement information, and the third data includes response information returned after the query.

Specifically, the SQL statement of the first data and the response information returned after the query are classified to obtain second data and third data. When the acquired intermediate data is data of which the application interface is a database access interface type, the accessed data generally comprises SQL statements and response information returned after the data port is queried, and can be classified according to the characteristics formed by the SQL statements, so that the classification can be quickly and accurately performed.

And S133, analyzing the second data and the third data to obtain target data.

In an embodiment, referring to fig. 5, the step S133 may include steps S1331 to S1333.

And S1331, performing lexical analysis and syntactic analysis on the second data to obtain a lexical analysis result and a syntactic analysis result.

In this embodiment, the lexical analysis result refers to a result obtained by identifying keywords, operators, comments, spaces, placeholders, and the like included in the SQL statement; the syntactic analysis result refers to information such as database names, data table names, field names and the like obtained by performing syntactic analysis on the structure identified by the lexical analysis.

Specifically, the lexical analysis and the syntactic analysis may be performed by using an existing analysis method, and are not described herein again.

And S1332, associating the lexical analysis result and the syntactic analysis result to obtain associated information.

In this embodiment, the associated information refers to information related to the structure of the database table, i.e., information related to the data table, the data field, and the like.

Specifically, the lexical analysis results and the syntactic analysis results are associated in a unified manner, information related to the database table structure is extracted based on a single database, and the syntactic analysis results are analyzed based on the lexical analysis results, so that a certain association relationship exists between the lexical analysis results and the syntactic analysis results, for example, the syntactic analysis results are analyzed by which keywords of the lexical analysis results, and the syntactic analysis results corresponding to the keywords can be associated correspondingly.

And S1333, extracting the database table structure information of the third data to obtain the table structure information.

In the present embodiment, the table structure information includes, but is not limited to, a database name, a table name, a field type, a field length, a primary key, an index, and the like.

The third data refers to response information returned after query, and the response information returned after query all includes contents such as database name, table name, field type, field length, primary key, index and the like, so that the third data can be directly extracted.

And S140, analyzing according to the target data and constructing a database table structure.

In this embodiment, the database table structure refers to a table structure formed by extracting a database IP or domain name, a database name, a table name, and a field name according to user behavior data and transmission data of an application.

In an embodiment, referring to fig. 6, the step S140 may include steps S141 to S142.

And S141, merging the single target data by taking the single database as a unit to obtain a merging result.

In this embodiment, the merged result refers to a result formed by merging all the target data in units of a single database, where the single database is currently accessed based on the application interface.

Specifically, with a single database as a unit, relevant information in a database table structure is directly extracted aiming at relevant information obtained by SQL statement analysis and table structure information, and the extracted information is merged.

And S142, carrying out duplicate removal processing and association analysis on the combined result to obtain a database table structure.

In this embodiment, duplicate removal processing is performed on duplicate information in the merged result, and association analysis is performed on the merged result to form a database table structure.

The duplicate removal mechanism is carried out based on a single database as a unit, initial data of a table structure consisting of a database IP or a domain name, a database name, a table name and a field name is stored, when data are stored subsequently, information of the database IP, the database name, the table name, the field name and the like is compared in sequence, if the information is not repeated, data updating is carried out, if the information is repeated, duplicate removal operation is carried out, and the duplicate removal operation covers or discards duplicate data.

The association analysis is carried out on the basis of a single database as a unit, the single database is used as main directory information, a table name based on the lower surface of the single database is obtained and used as second directory information, and information such as a field name, a field length, a main key and the like under the table name is obtained and used as third directory information, so that the database table structure information is constructed.

Because the data acquisition is not influenced by an encryption protocol and a private protocol and the acquired data is comprehensive, manageable and controllable, more comprehensive SQL sentences and returned data and information can be extracted, and the structure of the constructed database table is more complete; because the authorized database account is not required to be connected to the database pull table structure information, the structure information is dynamically constructed by collecting related data, operation and maintenance management on the database account is not required, and the account safety problem does not exist.

By implementing the method of the embodiment of the invention, the existing database system does not need to be authorized and connected, the safe operation and maintenance of the database and the safety of the account number can be better ensured, the database table structure can be automatically sorted and updated according to the deployment of the service system and the change of the service, the database table structure can be more comprehensively, automatically, intelligently and completely constructed, the safety is high, the implementation process has no influence on the existing network architecture, the reliability and the usability of the network are ensured, the difficulty in the operation and the maintenance of the network is reduced, and the efficiency of fault removal is reduced.

According to the database table structure construction method, the data acquisition strategy is set at the management end and is updated to the application client, the application client generates the actual acquisition strategy by combining the acquisition strategy in a switch mode, the data of a specific interface is intercepted by adopting a byte code enhancement technology, and the corresponding data is acquired by combining the actual acquisition strategy, so that the user behavior data and the transmission data can be acquired in a control mode, the acquisition is more comprehensive, simple and flexible, the database table structure is analyzed and constructed based on the comprehensively acquired data, the existing network architecture is not influenced, the reliability and the usability of the network are ensured, the network operation and maintenance difficulty is reduced, the fault removal efficiency is reduced, the database table structure is constructed more comprehensively and completely, and the safety is high.

Fig. 7 is a schematic flowchart of a method for building a database table structure according to another embodiment of the present invention. As shown in FIG. 7, the database table structure construction method of the present embodiment includes steps S210-S250. Steps S210 to S240 are similar to steps S110 to S140 in the above embodiments, and are not described herein again. The added step S250 in the present embodiment is explained in detail below.

And S250, dynamically managing the database table structure.

Specifically, the target data obtained by the single analysis and the information stored in the database are updated into the database after being deduplicated.

In the embodiment, the table structure information after single analysis is stored in a structured database; and comparing the table structure information analyzed once with the information stored in the database, and removing repeated information for storage without storage or selecting to cover storage. The table structure information analyzed in a single time is compared with the information stored in the database, and the updated information is stored in a storage. Note that, here, the database table structure is already constructed, and when data is subsequently collected, the constructed database table structure needs to be updated and managed, so that an effect of dynamically managing the database table structure is achieved.

Fig. 8 is a schematic block diagram of an apparatus 300 for constructing a database table structure according to an embodiment of the present invention. As shown in fig. 8, the present invention further provides a database table structure constructing apparatus 300 corresponding to the above database table structure constructing method. The database table structure construction apparatus 300 includes a unit for executing the above-described database table structure construction method, and the apparatus may be configured in a server. Specifically, referring to fig. 8, the database table structure building apparatus 300 includes a data acquisition unit 301, a gathering unit 302, a filtering unit 303, and a building unit 304.

The data acquisition unit 301 is configured to formulate an acquisition policy, and send the acquisition policy to the application server, so that the application server acquires user behavior data and application transmission data to obtain intermediate data; an aggregation unit 302, configured to aggregate the intermediate data; a filtering unit 303, configured to filter the intermediate data based on an application database access interface to obtain target data; a constructing unit 304, configured to analyze and construct a database table structure according to the target data; the data acquisition unit 301 is configured to formulate an acquisition policy, and send the acquisition policy to an application server, so that the application server sets an actual acquisition policy in a switching mode according to the acquisition policy; intercepting the access data by the application server by adopting a byte enhancement technology to obtain intercepted data; and acquiring user behavior data and application transmission data for the intercepted data by the application server according to the actual acquisition strategy to obtain intermediate data.

In one embodiment, as shown in fig. 9, the filtering unit 303 includes a data extracting sub-unit 3031, a classifying sub-unit 3032, and an analyzing sub-unit 3033.

A data extraction subunit 3031, configured to extract data meeting a preset condition from the intermediate data to obtain first data; a classification subunit 3032, configured to classify the first data to obtain second data and third data; an analyzing subunit 3033, configured to analyze the second data and the third data to obtain target data; the preset condition comprises that an application interface is a database access interface type.

In an embodiment, the classification subunit 3032 is configured to classify the SQL statement and the response information returned after the query on the first data to obtain second data and third data, where the second data includes the SQL statement information, and the third data includes the response information returned after the query.

In an embodiment, referring to fig. 10, the analysis subunit 3033 includes a sentence analysis module 30331, an association module 30332, and an information extraction module 30333.

A sentence analysis module 30331, configured to perform lexical analysis and syntactic analysis on the second data to obtain a lexical analysis result and a syntactic analysis result; an association module 30332, configured to associate the lexical analysis result and the syntactic analysis result to obtain association information; an information extraction module 30333, configured to extract database table structure information of the third data to obtain table structure information; wherein the target data includes the association information and the table structure information.

In one embodiment, referring to fig. 11, the construction unit 304 includes a merging subunit 3041 and a de-weighting subunit 3042.

A merging subunit 3041, configured to merge single target data with a single database as a unit, so as to obtain a merged result; and a duplicate removal subunit 3042, configured to perform duplicate removal processing and association analysis on the merged result to obtain a database table structure.

Fig. 12 is a schematic block diagram of an apparatus 300 for constructing database table structures according to another embodiment of the present invention. As shown in fig. 12, the database table structure building apparatus 300 of the present embodiment is added with a dynamic management unit 305 on the basis of the above-described embodiment.

A dynamic management unit 305, configured to dynamically manage the database table structure.

In an embodiment, the dynamic management unit 305 is configured to update the target data obtained by the single analysis and the information stored in the database after deduplication.

It should be noted that, as can be clearly understood by those skilled in the art, the specific implementation processes of the database table structure constructing apparatus 300 and each unit may refer to the corresponding descriptions in the foregoing method embodiments, and for convenience and brevity of description, no further description is provided here.

The above-described database table structure constructing apparatus 300 may be implemented in the form of a computer program that can be run on a computer device as shown in fig. 13.

Referring to fig. 13, fig. 13 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 may be a server.

Referring to fig. 13, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.

The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer programs 5032 include program instructions that, when executed, cause the processor 502 to perform a database table structure construction method.

The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.

The internal memory 504 provides an environment for running the computer program 5032 in the non-volatile storage medium 503, and when the computer program 5032 is executed by the processor 502, the processor 502 can be caused to execute a database table structure construction method.

The network interface 505 is used for network communication with other devices. Those skilled in the art will appreciate that the architecture shown in fig. 13 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing device 500 to which the disclosed aspects apply, as a particular computing device 500 may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

Wherein the processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps:

an acquisition strategy is formulated and issued to an application server, so that the application server acquires user behavior data and application transmission data to obtain intermediate data; aggregating the intermediate data; filtering the intermediate data based on an application database access interface to obtain target data; analyzing according to the target data and constructing a database table structure;

the method comprises the following steps of making an acquisition strategy, issuing the acquisition strategy to an application server, and acquiring user behavior data and application transmission data by the application server to obtain intermediate data, wherein the steps comprise: an acquisition strategy is formulated and issued to an application server, so that the application server adopts a switch mode to set an actual acquisition strategy according to the acquisition strategy; intercepting the access data by the application server by adopting a byte enhancement technology to obtain intercepted data; and acquiring user behavior data and application transmission data for the intercepted data by the application server according to the actual acquisition strategy to obtain intermediate data.

In an embodiment, when implementing the step of filtering the intermediate data based on the application database access interface to obtain the target data, the processor 502 specifically implements the following steps:

extracting data meeting preset conditions in the intermediate data to obtain first data; classifying the first data to obtain second data and third data; analyzing the second data and the third data to obtain target data;

In an embodiment, when the processor 502 implements the step of classifying the first data to obtain the second data and the third data, the following steps are specifically implemented:

In an embodiment, when the processor 502 implements the step of analyzing the second data and the third data to obtain the target data, the following steps are specifically implemented:

performing lexical analysis and syntactic analysis on the second data to obtain a lexical analysis result and a syntactic analysis result; associating the lexical analysis result and the grammatical analysis result to obtain associated information; extracting database table structure information of the third data to obtain table structure information;

In an embodiment, when the processor 502 implements the step of constructing the database table structure according to the target data, the following steps are specifically implemented:

merging single target data by taking a single database as a unit to obtain a merging result; and carrying out deduplication processing and association analysis on the merged result to obtain a database table structure.

In one embodiment, after the step of constructing a database table structure from the target data is performed, the processor 502 further performs the following steps:

and dynamically managing the database table structure.

In an embodiment, when implementing the step of dynamically managing the database table structure, the processor 502 specifically implements the following steps:

It should be understood that in the embodiment of the present Application, the Processor 502 may be a Central Processing Unit (CPU), and the Processor 502 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It will be understood by those skilled in the art that all or part of the flow of the method implementing the above embodiments may be implemented by a computer program instructing associated hardware. The computer program includes program instructions, and the computer program may be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.

Accordingly, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program, when executed by a processor, causes the processor to perform the steps of:

In an embodiment, when the processor executes the computer program to implement the step of filtering the intermediate data based on the application database access interface to obtain the target data, the following steps are specifically implemented:

In an embodiment, when the processor executes the computer program to realize the step of classifying the first data to obtain the second data and the third data, the following steps are specifically realized:

In an embodiment, when the processor executes the computer program to implement the step of analyzing the second data and the third data to obtain the target data, the following steps are specifically implemented:

In an embodiment, when the step of constructing a database table structure according to the target data is implemented by executing the computer program, the processor specifically implements the following steps:

In one embodiment, after executing the computer program to implement the step of constructing a database table structure from the target data, the processor further implements the steps of:

and dynamically managing the database table structure.

In an embodiment, when the step of dynamically managing the database table structure is implemented by executing the computer program, the processor specifically implements the following steps:

The storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk, which can store various computer readable storage media.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, various elements or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented.

The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be merged, divided and deleted according to actual needs. In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a terminal, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention.

While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. The database table structure construction method is characterized by comprising the following steps:

aggregating the intermediate data;

2. The database table structure construction method according to claim 1, wherein the filtering the intermediate data based on the application database access interface to obtain target data comprises:

classifying the first data to obtain second data and third data;

analyzing the second data and the third data to obtain target data;

3. The database table structure construction method according to claim 2, wherein the classifying the first data to obtain a second data and a third data comprises:

4. The database table structure construction method according to claim 3, wherein the analyzing the second data and the third data to obtain target data comprises:

5. The database table structure construction method according to claim 1, wherein the constructing a database table structure from the target data comprises:

6. The database table structure construction method according to any one of claims 1 to 5, further comprising, after constructing the database table structure from the target data:

and dynamically managing the database table structure.

7. The database table structure building method of claim 6, wherein the dynamically managing the database table structure comprises:

8. Database table structure construction equipment, its characterized in that includes:

the aggregation unit is used for aggregating the intermediate data;

9. A computer device, characterized in that the computer device comprises a memory, on which a computer program is stored, and a processor, which when executing the computer program implements the method according to any of claims 1 to 7.

10. A storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.