CN113377420B - Inter-system data docking method, device, equipment and storage medium - Google Patents
Inter-system data docking method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113377420B CN113377420B CN202110637040.7A CN202110637040A CN113377420B CN 113377420 B CN113377420 B CN 113377420B CN 202110637040 A CN202110637040 A CN 202110637040A CN 113377420 B CN113377420 B CN 113377420B
- Authority
- CN
- China
- Prior art keywords
- data
- data file
- file
- source system
- column number
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000003032 molecular docking Methods 0.000 title claims abstract description 19
- 238000003860 storage Methods 0.000 title claims abstract description 18
- 230000000977 initiatory effect Effects 0.000 claims abstract description 8
- 230000015654 memory Effects 0.000 claims description 30
- 238000004590 computer program Methods 0.000 claims description 14
- 238000012545 processing Methods 0.000 claims description 12
- 238000004422 calculation algorithm Methods 0.000 claims description 10
- 230000005540 biological transmission Effects 0.000 claims description 8
- 238000000586 desensitisation Methods 0.000 claims description 8
- 230000008676 import Effects 0.000 claims description 8
- 230000008569 process Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000001514 detection method Methods 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 5
- 238000012546 transfer Methods 0.000 description 5
- 210000001503 joint Anatomy 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000013478 data encryption standard Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000004801 process automation Methods 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 208000025174 PANDAS Diseases 0.000 description 1
- 208000021155 Paediatric autoimmune neuropsychiatric disorders associated with streptococcal infection Diseases 0.000 description 1
- 240000000220 Panda oleosa Species 0.000 description 1
- 235000016496 Panda oleosa Nutrition 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001010 compromised effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013479 data entry Methods 0.000 description 1
- 230000001627 detrimental effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 238000007711 solidification Methods 0.000 description 1
- 230000008023 solidification Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- General Engineering & Computer Science (AREA)
- Bioethics (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present specification relates to the technical field of data docking, and provides a method, a device, equipment and a storage medium for data docking between systems, where the method includes: timing is based on a preset the sequence of operations is such that, initiating an operation request to a designated page element of a corresponding page on a source system; acquiring data returned by the source system for the operation request; storing the data as a data file in a specified format; and transmitting the data files to a target system in batches. The present description may enable inter-system data interfacing without the source system providing an API interface for the target system.
Description
Technical Field
The present disclosure relates to the field of data interfacing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for interfacing data between systems.
Background
The IT department in banking industry uses a large amount of management information systems (MANAGEMENT INFORMATION SYSTEM, abbreviated as MIS), such as resource management systems, configuration management systems, operation and maintenance management systems developed by each department, etc., each system has ITs own data, and needs to use the data of other systems, but when one system is developed, the need of supplying data to other systems is not necessarily considered, even what need will be in the future is not known, so that IT is easy to form a data island.
For example, a new development of an operation and maintenance management system by a certain department requires the data of two old systems, namely a resource management system and a configuration management system, to be used as the calculation basis in near real time, so that the new system needs to be in data docking with the old system. However, the old system is developed earlier, the requirement of data supply for the new system is not considered, and a specific application programming interface (Application Programming Interface, abbreviated as API) interface is not reserved, so that the data docking between the new system and the old system is difficult to realize.
Disclosure of Invention
An objective of the embodiments of the present disclosure is to provide a method, an apparatus, a device, and a storage medium for inter-system data interfacing, so as to implement inter-system data interfacing in a case that a source system does not provide an API interface for a target system.
In order to achieve the above object, in one aspect, an embodiment of the present disclosure provides a method for interfacing data between systems, including:
timing based on a preset operation sequence, initiating an operation request to a designated page element of a page on a source system;
acquiring data returned by the source system for the operation request;
Storing the data as a data file in a specified format;
And transmitting the data files to a target system in batches.
In a preferred embodiment, before the batch transferring the data file to the target system, the method further includes:
And carrying out data integrity detection on the data row and/or the data column of the data file.
In a preferred embodiment, the detecting the data integrity of the data row and/or the data column of the data file includes:
confirming whether the data column number of the data file is equal to the data column number of the historical data file or not;
when the data column number of the data file is equal to the data column number of the historical data file, determining a difference value between the data line number of the data file and the data line number of the historical data file;
Determining an absolute value of a ratio of the difference to a number of data columns of the data file;
Determining whether the absolute value is greater than a preset threshold;
And when the absolute value is not greater than a preset threshold value, confirming that the data of the data file is complete.
In a preferred embodiment, the method further comprises:
when the data column number of the data file is not equal to the data column number of the historical data file or the absolute value is larger than a preset threshold value, confirming that the data of the data file is incomplete;
deleting the data file and reacquiring the data file.
In a preferred embodiment, before the timing initiates the operation request to the specified page element of the corresponding page on the source system based on the preset operation sequence, the method further includes:
providing a user name and a password ciphertext for the source system so that the source system can carry out login authentication after decryption; and the source system uses a decryption algorithm after code confusion processing when decrypting the user name and the cipher text.
In a preferred embodiment, before transmitting the data file to the target system, the method further comprises:
and carrying out data desensitization processing on the data file.
In a preferred embodiment, the specified format comprises a plain text format.
On the other hand, the embodiment of the specification also provides a data docking device between systems, which comprises:
The robot flow automation module is used for initiating an operation request to a designated page element of a corresponding page on a source system at regular time based on a preset operation sequence and acquiring data returned by the source system for the operation request;
the file transmission service module is used for storing the data into a data file with a specified format;
And the batch import module is used for batch transmitting the data files to a target system.
In another aspect, embodiments of the present disclosure further provide a computer device including a memory, a processor, and a computer program stored on the memory, which when executed by the processor, performs the instructions of the above method.
In another aspect, embodiments of the present disclosure also provide a computer storage medium having stored thereon a computer program which, when executed by a processor of a computer device, performs instructions of the above method.
As can be seen from the technical solutions provided by the embodiments of the present disclosure, the embodiments of the present disclosure may initiate an operation request to a specified page element of a corresponding page on a source system based on a preset operation sequence, and acquire data returned by the source system for the operation request, thereby implementing automatic simulation of a manual operation source system. On the basis, the data can be stored as data files with specified formats and transmitted to the target system in batches, so that under the condition that the source system does not provide an API interface for the target system, the data butt joint between the source system and the target system is realized, namely, the light-weight and quick data butt joint between the source system and the target system is realized.
Drawings
In order to more clearly illustrate the embodiments of the present description or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some of the embodiments described in the present description, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. In the drawings:
FIG. 1 illustrates a flow chart of a method of inter-system data interfacing in some embodiments of the present description;
FIG. 2 illustrates a flow chart of data integrity detection in some embodiments of the present description;
FIG. 3 illustrates a block diagram of an inter-system data docking apparatus in some embodiments of the present disclosure;
fig. 4 illustrates a block diagram of a computer device in some embodiments of the present description.
[ Reference numerals description ]
1. A data docking device;
11. An RPA module;
12. a file transfer service module;
13. A batch importing module;
21. A source system;
22. a source system database;
31. a target system;
32. A target system database;
402. A computer device;
404. A processor;
406. A memory;
408. a driving mechanism;
410. An input/output interface;
412. an input device;
414. An output device;
416. A presentation device;
418. a graphical user interface;
420. A network interface;
422. A communication link;
424. a communication bus.
Detailed Description
In order to make the technical solutions in the present specification better understood by those skilled in the art, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.
Embodiments of the present description relate to inter-system (e.g., MIS inter-system, etc.) data docking techniques. Wherein, data docking refers to: for the a-system and the B-system, if the B-system is to use the full data of the a-system or a part of the full data (which is generally used for data analysis of data of a plurality of systems), the a-system can actively transmit the data to the B-system in a normalized manner, or the B-system can automatically acquire the required data from the a-system in a normalized manner, so that the process is called data docking. For ease of description, in the present description, the a-system may be referred to as a source system (or data provider), and the B-system may be referred to as a target system (or data demander). In the embodiment of the present specification, the source system may be a system that adopts a browser and server architecture in general.
In some cases, the source system may not reserve a particular API interface for the target system. In other cases, the source system reserves a specific API interface for the target system, but sets a limit (e.g., 20 seconds) on the call interface frequency based on the performance capacity of the front-end server; as such, interfacing only a few thousand pieces of data through an API may take tens of hours, which makes it difficult to meet the data timeliness requirement. As such, in these cases, it is difficult to achieve data interfacing between the new and old systems between the source system and the target system.
In view of this, to achieve data interfacing of the source system and the target system in this case, the present embodiments provide an improved inter-system data interfacing method that can be applied to any suitable computer device side. Referring to fig. 1, in some embodiments of the present disclosure, the inter-system data docking method may include the following steps:
S101, initiating an operation request to a designated page element of a corresponding page (or called a graphical user interface) on a source system at fixed time based on a preset operation sequence.
S102, acquiring data returned by the source system for the operation request.
S103, storing the data into a data file with a specified format.
S104, transmitting the data files to a target system in batches.
In the embodiment of the specification, the operation request can be initiated to the designated page element of the corresponding page on the source system based on the preset operation sequence, and the data returned by the source system for the operation request is obtained, so that the automatic simulation of the manual operation source system is realized. On the basis, the data can be stored as data files with specified formats and transmitted to the target system in batches, so that under the condition that the source system does not provide an API interface for the target system, the data butt joint between the source system and the target system is realized, namely, the light weight (without modifying the source system) and agility (the API interface is provided for the target system relative to the source system, but the requirement of data timeliness is difficult to meet due to limited use) between the source system and the target system are realized.
In the embodiment of the present specification, step S101 and step S102 may be implemented based on a robotic process automation (Robotic Process Automation, RPA) technology. The RPA technology is a business process automation technology based on a software robot, wherein the robot is only a virtual concept, and does not need an entity robot, and is only an execution body or an execution unit of a flow. For example, such an executable may be referred to as an RPA module in embodiments of the present description.
By simulating a series of operations performed by a person on a computer device, the RPA module can automatically implement all operations of the person on the computer (such as clicking a mouse, copying, pasting, data entry, web navigation, opening a file, etc.), and can continuously repeat operations according to a certain rule. Because its core is to simulate human operation, it applies pressure to the source system as well as normal operation of a person on the source system, generally without excessive load, and without significant impact on the source system. Therefore, in the embodiment of the present disclosure, the process of initiating an operation request to a specified page element of a corresponding page on a source system and acquiring data returned by the source system for the operation request based on a preset operation sequence at regular time is a process of automatically simulating manual operation of the source system and automatically acquiring required data.
In the embodiment of the present disclosure, the operation sequence is all operations and the ordering thereof required to implement data interfacing. For example, in one embodiment, taking all files under a specified path in the source system as an example. The sequence of operations may include: a series of operations of logging in the source system, a series of operations of opening a specified path, a series of operations of reading all files under the specified path, a series of operations of saving the read files as a specified format, and the like. The operation sequence may be obtained in advance by the following steps 1to 4. Since the entire operation process involved in the operation sequence typically involves jumping between pages, and each page requires an operation-specific page element to jump to the next page. Thus, it is necessary to initiate an operation to a specified page element of a corresponding page on the source system. It can be seen that the operation requests in the embodiments of the present specification are actually operation request sequences, that is, a plurality of operation requests are sequentially operated.
And step 1, the solidification manual execution obtains the flow of the target data from the page of the source system. Firstly, all operation steps on a source system page when manual execution is required to be explicitly performed are ensured to be successfully performed according to sequential operation of the steps, and then all step lists are fixed.
In some embodiments of the present description, there are two methods for obtaining all operations, method 1: the engineer responsible for implementing the data docking interviews the user who manually executes the process, rubs and records the operation process of the user on the source system page, decomposes the process into steps, and lists the steps into a list of tables, thereby obtaining all the operations. Method 2: the list is automatically formed by recording and storing each step of operation in the background when the user manually operates by a software program.
And 2, identifying page elements operated in manual execution. I.e., identify the page elements (i.e., objects of operation) operated upon by the person performing each step, such as multi-boxes, drop-down boxes, buttons, and the like.
There are two approaches as to how page elements can be identified. Method 1: on the basis of forming the list, an engineer responsible for data butt joint observes the operation process of the user on the source system page, visually judges and records what page elements are operated by the user at each step, and therefore identification is completed. Method 2: and recording and storing each operation in the background when the user manually operates by a software program, and automatically judging the page elements of each operation according to the webpage HTML codes of the page elements related to the operation and forming a list, thereby completing identification.
And 3, identifying the operation on the page element in manual execution. And identifying what operation is performed on each page element in manual execution, for example, clicking operation is performed on a button element, and pull-down operation and selection operation are performed on a pull-down frame element.
There are two ways how to identify the operation on the page element. Method 1: on the basis of the step 2, an engineer responsible for data docking visually judges and records what operation is performed on each page element (such as clicking, dragging and the like with a left mouse button) by a user in the operation process of the source system page by the sightseeing and hearing user. Method 2: each step of operation is recorded and stored in the background when a user manually executes the operation through a software program, and according to a message hook (hook) of an operating system, the operation of each step on a corresponding page element is automatically judged, and a list is formed according to the operation.
And 4, writing an RPA module to automatically simulate the manual operation flow. The RPA module refers to a computer program developed specifically for this purpose. Specifically, the RPA module may be implemented using a programming language such as python or java, so that all of the above manual steps are automated.
In addition, in order to realize continuous and automatic transmission of the data obtained by the RPA module to the transmission destination system, parameter configurations such as a storage path for storing the data obtained by the RPA module, a frequency of configuring the operation of the RPA module (for example, a time interval required for each execution of the RPA is not smaller than a set to be executed once an hour or once a day, etc.), a frequency of configuring the import batch operation (for example, a time interval required for each execution of the RPA or once a day, etc., which should be lower than the frequency of the RPA operation), webdriver parameters (for example, a timeout time of page loading, an IP address of a source system, a page size) and the like are also required to be preset.
In the embodiment of the present disclosure, the data returned by the source system for the operation request is obtained, that is, the data is read from the database of the source system based on the operation request. The data obtained from the source system may be saved as a data file in a plain text format (e.g., comma separated value (Comma-SEPARATED VALUES, CSV) format, etc.). Therefore, the data file can be used commonly among various systems and programs, and the problem that a target system cannot be identified due to the problem of text format is solved. Furthermore, while the data acquired from the source system may be stored locally on the RPA module, this may be detrimental to direct import into the target system. Therefore, the RPA module can automatically upload the data file in the plain text format to a file uploading service module such as an FTP server. Thus, batch import of data files to the target system (e.g., batch import to the target system database) can be achieved through written bat or shell scripts.
In other embodiments of the present disclosure, the data integrity check may also be performed on the data rows and/or the data columns of the data file before the data file is transferred to the target system in bulk, so as to ensure that the data transferred to the target system is complete data.
In some embodiments of the present disclosure, as shown in fig. 2, the detecting the data integrity of the data row and/or the data column of the data file may include the following steps:
s201, determining the number of data columns of the data file. Because the data in the format data files such as CSV are arranged in a matrix, the number of data columns and the number of data rows of the data files can be calculated.
S202, confirming whether the data column number of the data file is equal to the data column number of the historical data file. Step S203 is performed when equal, otherwise step S207 and subsequent steps are performed.
Because the data in the format data files such as CSV are arranged in a matrix, in many cases, the number of columns of the data file obtained each time is fixed. Therefore, the number of columns corresponding to the statistical history data file can be used as a reference value for evaluating whether the data file obtained each time is abnormal.
And S203, when the data column number of the data file is equal to the data column number of the historical data file, determining a difference value between the data line number of the data file and the data line number of the historical data file.
S204, determining an absolute value of the ratio of the difference value to the data column number of the data file.
I.e. can pass the formulaThe absolute value v is calculated. Wherein c 1 is the number of data lines of the currently obtained data file, and c 0 is the number of data lines of the history data file. The larger the absolute value is, the larger the difference between the data line number of the currently obtained data file and the data line number of the historical data file is; accordingly, the greater the probability that the currently obtained data file is an abnormal data file.
S205, determining whether the absolute value is larger than a preset threshold.
And S206, when the absolute value is not greater than a preset threshold value, confirming that the data of the data file is complete. Therefore, no reacquisition is required. Of course, in this case, it is also possible to generate a log of processing success and automatically notify the user by means of e-mail or the like; the log format of the log may be, for example, time-stamped for the success flag bit.
And S207, when the data column number of the data file is unequal to the data column number of the historical data file or the absolute value is larger than a preset threshold value, confirming that the data of the data file is incomplete. Therefore, re-acquisition is required to ensure the integrity of the data file.
S208, deleting the data file and reacquiring the data file.
In other embodiments of the present disclosure, login authentication is generally required to be performed by logging into the source system before an operation request is initiated to a specified page element of a corresponding page on the source system based on a preset operation sequence at a fixed time. Thus, the source system may be provided with a user name and password ciphertext (i.e., the user name and password are both ciphertext) for login authentication after decryption; and the source system uses a decryption algorithm after code confusion processing when decrypting the user name and the cipher text. Therefore, an attacker can be prevented from knowing the encryption algorithm through decompilation and other means, so that the user name and the password plaintext cannot be known, and the data security is improved.
Code confusion in this specification refers to: for the program P to be protected (e.g., decryption algorithm, etc.), the obfuscated P ' and P ' output are consistent, but it is difficult for an attacker (attacker refers to a person attempting to illegally learn information, such as a hacker attempting to infer the ciphertext by acquiring the decryption algorithm) to learn key information (e.g., decryption algorithm) in the program through P '. The code confusion in the embodiment of the specification can be static code confusion or dynamic code confusion, and can be specifically selected according to actual application requirements. Static confusion, typically, is a flattening control flow, a complicating control flow, etc., where the code is fixed before the program is executed, and dynamic confusion refers to that the code is still changing in execution, such as code self-modification, etc. In some embodiments of the present description, the encryption algorithm may employ an asymmetric encryption algorithm such as a data encryption standard (Data Encryption Standard, DES), an advanced encryption standard (Advanced Encryption Standard, AES), or the like.
Some field information in a data file may contain sensitive information, which is defined as: information that would expose natural persons and/or organizations to economic or other losses once compromised. The source system may provide such sensitive information, but the target system generally does not require sensitive information. Therefore, in order to further improve the data security, before the data file is transferred to the target system, the data file may be subjected to data desensitization processing, that is, before the data file is transferred to the target system, the data file may be read into the memory (for example, a manner such as a csv_read method of pandas may be used), and the sensitive information may be deleted from the data file according to the preset data desensitization processing logic and stored as a new file. Which field attributes are listed as sensitive information can be pre-specified in the data desensitization processing logic, and can be configured by the user. For example, in one embodiment of the present disclosure, the field attribute columns "specific use of IT asset", "intranet IP address", etc. may be designated as sensitive information.
In addition, in order to avoid that the number of columns of normal data files becomes smaller due to deletion of sensitive information. In some embodiments of the present disclosure, an empty column may be inserted in the home position of the field attribute column corresponding to the deletion sensitive information, and the column name of the empty column is set to be the column name of the deleted column, so that the data structure of the data file after desensitization is kept consistent with the data structure of the data file before desensitization, that is, the column numbers are equal, and the column names are the same.
In other embodiments of the present disclosure, the lifecycle detection of the data file may also be performed periodically, i.e., periodically to determine whether the data file has expired, prior to bulk transfer of the data file to the target system. Wherein expiration refers to: the file has been imported into the database of the target system. Thus, in bulk transfer, the transferred data files are unexpired data files. And the expired data files can be deleted to reduce the occupation of storage resources.
While the process flows described above include a plurality of operations occurring in a particular order, it should be apparent that the processes may include more or fewer operations, which may be performed sequentially or in parallel (e.g., using a parallel processor or a multi-threaded environment).
Corresponding to the method, the embodiment of the specification also provides a data docking device between systems. Referring to fig. 3, in some embodiments of the present disclosure, the data docking device 1 may initiate an operation request to a specified page element of a corresponding page on the source system 21 at a timing based on a preset operation sequence; acquiring data returned by the source system 21 for the operation request (namely acquiring data from a source system database 22 corresponding to the source system 21); storing the data as a data file in a specified format; the data files are transferred in batches to the target system 31 (i.e. to the target system database 32 corresponding to the target system 31) for use by the target system 31.
With continued reference to fig. 3, the data docking device 1 may include an RPA module 11, a file transfer service module 12, and a batch import module 13. Wherein:
the RPA module 11 can be used for initiating an operation request to a designated page element of a corresponding page on a source system based on a preset operation sequence at fixed time and acquiring data returned by the source system for the operation request;
a file transfer service module 12 for storing the data as a data file of a specified format;
A batch import module 13, configured to batch transmit the data file to a target system.
In some embodiments of the present description, the RPA module 11 may also be used to: and before the batch transmission of the data file to a target system, performing data integrity detection on data rows and/or data columns of the data file.
In some embodiments of the present disclosure, the performing data integrity detection on the data row and/or the data column of the data file includes:
confirming whether the data column number of the data file is equal to the data column number of the historical data file or not;
when the data column number of the data file is equal to the data column number of the historical data file, determining a difference value between the data line number of the data file and the data line number of the historical data file;
Determining an absolute value of a ratio of the difference to a number of data columns of the data file;
Determining whether the absolute value is greater than a preset threshold;
And when the absolute value is not greater than a preset threshold value, confirming that the data of the data file is complete.
In some embodiments of the present description, the RPA module 11 may also be used to: when the data column number of the data file is not equal to the data column number of the historical data file or the absolute value is larger than a preset threshold value, confirming that the data of the data file is incomplete; deleting the data file and reacquiring the data file.
In some embodiments of the present description, the RPA module 11 may also be used to: before an operation request is initiated to a designated page element of a corresponding page on a source system based on a preset operation sequence at the timing, a user name and a password ciphertext are provided for the source system so that the source system can carry out login authentication after decryption; and the source system uses a decryption algorithm after code confusion processing when decrypting the user name and the cipher text.
In some embodiments of the present description, the RPA module 11 may also be used to: and before the data file is transmitted to a target system, performing data desensitization processing on the data file.
In some embodiments of the present description, the specified format comprises a plain text format.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when implemented in the present specification.
Embodiments of the present description also provide a computer device. As shown in fig. 4, in some embodiments of the present description, the computer device 402 may include one or more processors 404, such as one or more Central Processing Units (CPUs) or Graphics Processors (GPUs), each of which may implement one or more hardware threads. The computer device 402 may also include any memory 406 for storing any kind of information, such as code, settings, data, etc., and in a particular embodiment, a computer program on the memory 406 and executable on the processor 404, which when executed by the processor 404, may perform the instructions of the inter-system data interfacing method described in any of the embodiments above. For example, and without limitation, memory 406 may include any one or more of the following combinations: any type of RAM, any type of ROM, flash memory devices, hard disks, optical disks, etc. More generally, any memory may store information using any technique. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent fixed or removable components of computer device 402. In one case, when the processor 404 executes associated instructions stored in any memory or combination of memories, the computer device 402 can perform any of the operations of the associated instructions. The computer device 402 also includes one or more drive mechanisms 408 for interacting with any memory, such as a hard disk drive mechanism, optical disk drive mechanism, and the like.
Computer device 402 may also include an input/output interface 410 (I/O) for receiving various inputs (via an input device 412) and for providing various outputs (via an output device 414). One particular output mechanism may include a presentation device 416 and an associated graphical user interface 418 (page). In other embodiments, input/output interface 410 (I/O), input device 412, and output device 414 may not be included, but merely as a computer device in a network. Computer device 402 may also include one or more network interfaces 420 for exchanging data with other devices via one or more communication links 422. One or more communication buses 424 couple the above-described components together.
The communication link 422 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the internet), a point-to-point connection, etc., or any combination thereof. Communication link 422 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to some embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processor to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processor, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processor to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processor to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computer device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computer device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present description embodiments may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present embodiments may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The embodiments of the specification may also be practiced in distributed computing environments where tasks are performed by remote processors that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments. In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the embodiments of the present specification. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.
Claims (7)
1. A method of interfacing data between systems, comprising:
Timing, based on a preset operation sequence, initiating an operation request to a designated page element of a corresponding page on a source system;
acquiring data returned by the source system for the operation request;
storing the data into a data file in a plain text format;
Detecting the data integrity of the data row and/or the data column of the data file; the detecting the data integrity of the data row and/or the data column of the data file comprises the following steps: confirming whether the data column number of the data file is equal to the data column number of the historical data file or not; when the data column number of the data file is equal to the data column number of the historical data file, determining a difference value between the data line number of the data file and the data line number of the historical data file; determining an absolute value of a ratio of the difference to a number of data columns of the data file; determining whether the absolute value is greater than a preset threshold; when the absolute value is not greater than a preset threshold value, confirming that the data of the data file is complete;
And transmitting the data files to a target system in batches.
2. The method of inter-system data interfacing of claim 1, further comprising:
when the data column number of the data file is not equal to the data column number of the historical data file or the absolute value is larger than a preset threshold value, confirming that the data of the data file is incomplete;
deleting the data file and reacquiring the data file.
3. The method of inter-system data interfacing according to claim 1, wherein before the timing initiates the operation request to the specified page element of the corresponding page on the source system based on the preset operation sequence, further comprising:
providing a user name and a password ciphertext for the source system so that the source system can carry out login authentication after decryption; and the source system uses a decryption algorithm after code confusion processing when decrypting the user name and the cipher text.
4. The method of inter-system data interfacing of claim 1, further comprising, prior to transmitting the data file to a target system:
and carrying out data desensitization processing on the data file.
5. An inter-system data docking apparatus, comprising:
The robot flow automation module is used for initiating an operation request to a designated page element of a corresponding page on a source system at regular time based on a preset operation sequence and acquiring data returned by the source system for the operation request;
the file transmission service module is used for storing the data into a data file in a plain text format;
the batch import module is used for batch transmitting the data files to a target system; the method further comprises the following steps before the batch transmission of the data files to a target system: detecting the data integrity of the data row and/or the data column of the data file; the detecting the data integrity of the data row and/or the data column of the data file comprises the following steps: confirming whether the data column number of the data file is equal to the data column number of the historical data file or not; when the data column number of the data file is equal to the data column number of the historical data file, determining a difference value between the data line number of the data file and the data line number of the historical data file; determining an absolute value of a ratio of the difference to a number of data columns of the data file; determining whether the absolute value is greater than a preset threshold; and when the absolute value is not greater than a preset threshold value, confirming that the data of the data file is complete.
6. A computer device comprising a memory, a processor, and a computer program stored on the memory, characterized in that the computer program, when being executed by the processor, performs the instructions of the method according to any of claims 1-4.
7. A computer storage medium having stored thereon a computer program, which, when executed by a processor of a computer device, performs the instructions of the method according to any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110637040.7A CN113377420B (en) | 2021-06-08 | 2021-06-08 | Inter-system data docking method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110637040.7A CN113377420B (en) | 2021-06-08 | 2021-06-08 | Inter-system data docking method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113377420A CN113377420A (en) | 2021-09-10 |
| CN113377420B true CN113377420B (en) | 2024-08-02 |
Family
ID=77576538
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110637040.7A Active CN113377420B (en) | 2021-06-08 | 2021-06-08 | Inter-system data docking method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113377420B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110147397A (en) * | 2019-04-10 | 2019-08-20 | 管南风 | System docking method, apparatus, management system and terminal device, storage medium |
| CN110830591A (en) * | 2019-11-22 | 2020-02-21 | 北京浪潮数据技术有限公司 | Information synchronization method, system, electronic equipment and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10649752B2 (en) * | 2016-09-28 | 2020-05-12 | International Business Machines Corporation | Sharing data and applications across computing systems |
| CN111435380B (en) * | 2019-01-14 | 2024-04-16 | 顺丰科技有限公司 | Page cross-domain interaction method, system, device and storage device |
| CN112256773A (en) * | 2019-07-22 | 2021-01-22 | 沅圣科技股份有限公司 | Enterprise management system docking method, electronic device and storage medium |
-
2021
- 2021-06-08 CN CN202110637040.7A patent/CN113377420B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110147397A (en) * | 2019-04-10 | 2019-08-20 | 管南风 | System docking method, apparatus, management system and terminal device, storage medium |
| CN110830591A (en) * | 2019-11-22 | 2020-02-21 | 北京浪潮数据技术有限公司 | Information synchronization method, system, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113377420A (en) | 2021-09-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110597925B (en) | Cross-chain data processing method and device based on block chain | |
| US9716692B2 (en) | Technology-agnostic application for high confidence exchange of data between an enterprise and third parties | |
| US10496843B2 (en) | Systems and method for automatically securing sensitive data in public cloud using a serverless architecture | |
| US20160299918A1 (en) | Device Control Using a Secure Decentralized Transactional Ledger | |
| CN113438219B (en) | Playback transaction identification method and device based on blockchain all-in-one machine | |
| EP3842980B1 (en) | System and method for automatically securing sensitive data in public cloud using a serverless architecture | |
| CN108334436A (en) | Optimization method, device, equipment and the computer readable storage medium of application software | |
| CN111597567B (en) | Data processing method, data processing device, node equipment and storage medium | |
| CN112365001B (en) | Model generation method, device and server | |
| US10831754B2 (en) | Using metadata to take action on an SMS message on a proprietary system | |
| CN111444090B (en) | Contract testing method, device, electronic device and storage medium in blockchain | |
| CN107392018A (en) | Application program hulling method and device | |
| CN108880868A (en) | BFD keep alive Packet transmission method, device, equipment and machine readable storage medium | |
| CN114741736A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111199486A (en) | Data processing method and device based on block chain network and storage medium | |
| CN113377420B (en) | Inter-system data docking method, device, equipment and storage medium | |
| US12072848B1 (en) | Systems and methods for managing personalized life information | |
| KR20220142210A (en) | Adaptive transaction processing node and method for blockchain | |
| KR102095046B1 (en) | Distributed Ledger Device and Distributed Ledger Method for Sharing User Information using Blockchain | |
| US20240249013A1 (en) | Data protection through collaborative association | |
| CN106657092B (en) | Service processing method and device based on SS L/T L S | |
| CN112732676B (en) | Block chain-based data migration method, device, equipment and storage medium | |
| CN109740327A (en) | Identity identifying method, apparatus and system | |
| CN116244062A (en) | Data processing method and device, electronic equipment and storage medium | |
| KR20230080677A (en) | High-speed blockchain system and method for processing an information using the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |