CN112486967A - Data acquisition method, terminal equipment and storage medium - Google Patents
Data acquisition method, terminal equipment and storage medium Download PDFInfo
- Publication number
- CN112486967A CN112486967A CN202011358838.XA CN202011358838A CN112486967A CN 112486967 A CN112486967 A CN 112486967A CN 202011358838 A CN202011358838 A CN 202011358838A CN 112486967 A CN112486967 A CN 112486967A
- Authority
- CN
- China
- Prior art keywords
- data
- request information
- storage medium
- acquisition method
- data acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a data acquisition method, a terminal device and a storage medium, wherein the method comprises the following steps: s1: receiving data request information, verifying the address of the data request information, and adding the data request information into a request queue when the address of the data request information conforms to a preset address rule; s2: extracting data request information according to the storage sequence of the data request information in the request queue, and acquiring corresponding data from a database corresponding to the data request information through a unified service port; s3: judging whether the acquired data accord with a preset data rule, if so, entering S4, otherwise, ending; s4: processing the acquired data; s5: and carrying out persistence operation on the processed data. The invention realizes the flexibility and the easy use of data acquisition by the access support, the pluggable component and the control of the acquisition threshold value of various storage media.
Description
Technical Field
The present invention relates to the field of data acquisition, and in particular, to a data acquisition method, a terminal device, and a storage medium.
Background
With the continuous development and promotion of hospital information work, hospital information construction has been extended outwards from the traditional hospital internal business system, and cross-industry cooperation and social resource integration have become one of the working key points of the next-stage hospital information construction. Various information of the hospital needs to be integrated with various social resources such as banks, communication operators, media, superior institutions, peer medical institutions and the like, and rich and convenient medical and health services are provided for patients.
As an emerging software architecture, the micro-service can convert a large single application program and service into dozens of micro-services. By using the technical characteristics of the micro-service, a standard interface can be established for each data item, the data interface is subjected to secondary packaging, standardized and unified issuing and access management, and services such as user data and message queues are pushed.
However, the existing microservice collecting technology can only improve the sharing efficiency of data resources of each party to a certain extent, and still needs to face the following problems:
1. flexible configuration and access cannot be achieved for different storages.
2. In the face of different data rules of different partners, customized service development is still required.
3. With the increase of the number of the partners, the development of the interface for acquiring the micro-service becomes huge and redundant, and the unified management is difficult.
4. The acquisition progress can not be effectively monitored, and the problems of data loss and repeated acquisition exist.
Disclosure of Invention
In order to solve the above problems, the present invention provides a data acquisition method, a terminal device, and a storage medium.
The specific scheme is as follows:
a method of data acquisition comprising the steps of:
s1: receiving data request information, verifying the address of the data request information, and adding the data request information into a request queue when the address of the data request information conforms to a preset address rule;
s2: extracting data request information according to the storage sequence of the data request information in the request queue, and acquiring corresponding data from a database corresponding to the data request information through a unified service port;
s3: judging whether the acquired data accord with a preset data rule, if so, entering S4, otherwise, ending;
s4: processing the acquired data;
s5: and carrying out persistence operation on the processed data.
Further, step S2 includes configuring port parameters of the unified service port, where the port parameters include a port number, a request data size, and an access size.
Further, the step of receiving the data request information comprises the step of requesting the identity authentication of the user; and carrying out security verification when receiving the data in the database.
Further, the data rules include data format, data set size, and data encoding.
Further, the data processing in step S4 includes:
s41: analyzing and converting the data into unified standard structure data;
s42: filtering the converted standard structure data according to a filtering rule;
s43: converting the data according to the content of the data, so that the converted data conforms to the safety rule and the service rule;
s44: and carrying out standardized reconstruction on the field names of the data according to the source of the data and the stored fields.
Further, the persistence operation includes storing the received data in the storage medium.
Further, the storage medium includes a relational database and a non-relational database; the storage medium is connected with a pluggable configuration, and the configuration parameters of the connection storage medium comprise a connection address, a port number, a user name, a password, a minimum connection number and connection waiting timeout.
Further, step S5 includes counting data stored in the storage medium, and recording and reminding an error log.
A data acquisition terminal device includes a processor, a memory, and a computer program stored in the memory and operable on the processor, where the processor executes the computer program to implement the steps of the method described above in the embodiments of the present invention.
A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out the steps of the method as described above for an embodiment of the invention.
The invention adopts the technical scheme and has the beneficial effects that:
1. the flexibility and the usability of data acquisition are realized through access support, pluggable components and acquisition threshold control of various storage media (relational databases and non-relational databases).
2. Through the standardization of the interface rule and the data transmission rule, the problem of joint debugging caused by rule confusion of multi-party butt joint is solved.
3. The problem that task confusion caused by expansion of services and increase of butt-joint parties is difficult to manage is solved by carrying out unified management on multiple different requests and different addresses.
4. The data acquisition process is recorded in real time, the integrity of the data is guaranteed, and meanwhile, an optimization suggestion can be provided for a subsequent configuration scheme.
Drawings
Fig. 1 is a flowchart illustrating a first embodiment of the present invention.
Detailed Description
To further illustrate the various embodiments, the invention provides the accompanying drawings. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the embodiments. Those skilled in the art will appreciate still other possible embodiments and advantages of the present invention with reference to these figures.
The invention will now be further described with reference to the accompanying drawings and detailed description.
The first embodiment is as follows:
an embodiment of the present invention provides a data acquisition method, as shown in fig. 1, the method includes the following steps:
s1: and receiving the data request information, verifying the address of the data request information, and adding the data request information into the request queue when the address of the data request information conforms to a preset address rule.
In this embodiment, when receiving the data request information, the method further includes authenticating the identity of the requesting user who sent the data request information, and the subsequent steps are performed only if the authentication is successful. Authentication, for example, a user may be registered in advance, and the registered user can transmit request information.
S2: and extracting the data request information according to the storage sequence of the data request information in the request queue, and acquiring corresponding data from a database corresponding to the data request information through the unified service port.
The unified service port may be an internal or external port, and parameters of the unified service port need to be configured in advance before the unified service port is used, where the parameters of the unified service port in this embodiment include a port number, a request data size, an access amount size, and the like. Other port parameters may be configured in other embodiments, and are not limited herein.
The database for acquiring data may be an external database or an internal database, which is not limited herein.
In order to ensure the security of data, besides the authentication of the requesting user who sends the data request information when receiving the data request information, the security check of the received data in the database is also included, for example, it is set that the data sent by the database needs to include an authorization code.
S3: and judging whether the acquired data accord with a preset data rule, if so, entering S4, and if not, ending.
And the data rule is set for standardizing the format of the received data, so that the subsequent processing difficulty is reduced. The data rules set in this embodiment include data format, data set size, data encoding, and the like. Other rules may be set in other embodiments, and are not limited herein.
S4: and carrying out data processing on the acquired data.
The data processing process comprises the following steps:
s41: and analyzing the data to convert the data into unified standard structure data.
Because the obtained data may have different structures, such as xml, blob, custom structures, etc., different analysis methods are set for different structures, and the data with different structures are converted into unified standard structure data.
S42: and filtering the converted standard structure data according to a filtering rule.
The filtering can be made according to specific service requirements, such as filtering null values, illegal, non-service related data, and the like.
S43: and converting the data according to the content of the data, so that the converted data conforms to the safety rule and the business rule.
The data may include sensitive data, dirty data, and the like which do not conform to the security rule or the business rule, and therefore, such data needs to be converted, for example, the identification number is replaced by the symbol "X".
S44: and carrying out standardized reconstruction on the field names of the data according to the source of the data and the stored fields.
Since the data from different data sources often have different corresponding field names, for example, the fields representing the sex of the school student may be various, such as "male", "male" and "boy", so that standardized reconstruction of various field names representing one content is required. This can also cut off the structural association between the final data and the source data, further ensuring the security and concealment of the data.
S5: and carrying out persistence operation on the processed data.
The persistence operation includes storing the received data in a storage medium.
Storage media include, but are not limited to, relational databases (mysql, oracle, sqlserver.) and non-relational databases (hbase, redis, elastic search.).
The storage medium is connected with a pluggable configuration, and the configuration parameters of the connection storage medium include but are not limited to a connection address, a port number, a user name, a password, a minimum connection number and a connection waiting timeout time.
Furthermore, in order to ensure the integrity of the stored data, the method also comprises the steps of counting the data stored in the storage medium and simultaneously recording an error log and reminding.
Example two:
the invention further provides a data acquisition terminal device, which comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the processor executes the computer program to realize the steps of the method embodiment of the first embodiment of the invention.
Further, as an executable scheme, the data acquisition terminal device may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The data acquisition terminal equipment can comprise, but is not limited to, a processor and a memory. Those skilled in the art will understand that the above-mentioned constituent structure of the data acquisition terminal device is only an example of the data acquisition terminal device, and does not constitute a limitation on the data acquisition terminal device, and may include more or less components than the above, or combine some components, or different components, for example, the data acquisition terminal device may further include an input/output device, a network access device, a bus, and the like, which is not limited in this embodiment of the present invention.
Further, as an executable solution, the Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, and the like. The general processor may be a microprocessor or the processor may be any conventional processor, etc., and the processor is a control center of the data acquisition terminal device and connects various parts of the whole data acquisition terminal device by using various interfaces and lines.
The memory can be used for storing the computer program and/or the module, and the processor can realize various functions of the data acquisition terminal equipment by running or executing the computer program and/or the module stored in the memory and calling data stored in the memory. The memory can mainly comprise a program storage area and a data storage area, wherein the program storage area can store an operating system and an application program required by at least one function; the storage data area may store data created according to the use of the mobile phone, and the like. In addition, the memory may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
The invention also provides a computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out the steps of the above-mentioned method of an embodiment of the invention.
The integrated module/unit of the data acquisition terminal device can be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as a stand-alone product. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), software distribution medium, and the like.
While the invention has been particularly shown and described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A data acquisition method is characterized by comprising the following steps:
s1: receiving data request information, verifying the address of the data request information, and adding the data request information into a request queue when the address of the data request information conforms to a preset address rule;
s2: extracting data request information according to the storage sequence of the data request information in the request queue, and acquiring corresponding data from a database corresponding to the data request information through a unified service port;
s3: judging whether the acquired data accord with a preset data rule, if so, entering S4, otherwise, ending;
s4: processing the acquired data;
s5: and carrying out persistence operation on the processed data.
2. The data acquisition method according to claim 1, characterized in that: step S2 further includes configuring port parameters of the unified service port, where the port parameters include a port number, a request data size, and an access size.
3. The data acquisition method according to claim 1, characterized in that: when receiving the data request information, the authentication of the requesting user is included; and carrying out security verification when receiving the data in the database.
4. The data acquisition method according to claim 1, characterized in that: the data rules include data format, data set size, and data encoding.
5. The data acquisition method according to claim 1, characterized in that: the process of data processing in step S4 includes:
s41: analyzing and converting the data into unified standard structure data;
s42: filtering the converted standard structure data according to a filtering rule;
s43: converting the data according to the content of the data, so that the converted data conforms to the safety rule and the service rule;
s44: and carrying out standardized reconstruction on the field names of the data according to the source of the data and the stored fields.
6. The data acquisition method according to claim 1, characterized in that: the persistence operation includes storing the received data in a storage medium.
7. The data acquisition method of claim 6, wherein: the storage medium comprises a relational database and a non-relational database; the storage medium is connected with a pluggable configuration, and the configuration parameters of the connection storage medium comprise a connection address, a port number, a user name, a password, a minimum connection number and connection waiting timeout.
8. The data acquisition method of claim 6, wherein: step S5 further includes counting data stored in the storage medium, and recording and reminding an error log.
9. A data acquisition terminal equipment characterized in that: comprising a processor, a memory and a computer program stored in the memory and running on the processor, the processor implementing the steps of the method according to any one of claims 1 to 8 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: the computer program when executed by a processor implementing the steps of the method as claimed in any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011358838.XA CN112486967A (en) | 2020-11-27 | 2020-11-27 | Data acquisition method, terminal equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011358838.XA CN112486967A (en) | 2020-11-27 | 2020-11-27 | Data acquisition method, terminal equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112486967A true CN112486967A (en) | 2021-03-12 |
Family
ID=74936232
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011358838.XA Pending CN112486967A (en) | 2020-11-27 | 2020-11-27 | Data acquisition method, terminal equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112486967A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108153682A (en) * | 2017-12-12 | 2018-06-12 | 西安交通大学 | A kind of method that flash translation layer (FTL) address of cache is carried out using flash memory internal concurrency |
| CN109344148A (en) * | 2018-10-16 | 2019-02-15 | 万达信息股份有限公司 | A kind of data acquisition management system and method |
| US20190304210A1 (en) * | 2016-05-16 | 2019-10-03 | Wi-Tronix, Llc | Real-Time Data Acquisition and Recording Data Sharing System |
| CN110658980A (en) * | 2018-06-29 | 2020-01-07 | 华为技术有限公司 | Data processing method and device and storage system |
| CN110990790A (en) * | 2019-11-29 | 2020-04-10 | 腾讯科技（深圳）有限公司 | Data processing method and equipment |
| CN111243748A (en) * | 2019-12-30 | 2020-06-05 | 湖南中医药大学 | Needle pushing health data standardization system |
| CN111949142A (en) * | 2020-08-10 | 2020-11-17 | 智业软件股份有限公司 | Time input method, terminal equipment and storage medium |
-
2020
- 2020-11-27 CN CN202011358838.XA patent/CN112486967A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190304210A1 (en) * | 2016-05-16 | 2019-10-03 | Wi-Tronix, Llc | Real-Time Data Acquisition and Recording Data Sharing System |
| CN108153682A (en) * | 2017-12-12 | 2018-06-12 | 西安交通大学 | A kind of method that flash translation layer (FTL) address of cache is carried out using flash memory internal concurrency |
| CN110658980A (en) * | 2018-06-29 | 2020-01-07 | 华为技术有限公司 | Data processing method and device and storage system |
| CN109344148A (en) * | 2018-10-16 | 2019-02-15 | 万达信息股份有限公司 | A kind of data acquisition management system and method |
| CN110990790A (en) * | 2019-11-29 | 2020-04-10 | 腾讯科技（深圳）有限公司 | Data processing method and equipment |
| CN111243748A (en) * | 2019-12-30 | 2020-06-05 | 湖南中医药大学 | Needle pushing health data standardization system |
| CN111949142A (en) * | 2020-08-10 | 2020-11-17 | 智业软件股份有限公司 | Time input method, terminal equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110008045B (en) | Method, device and equipment for aggregating microservices and storage medium | |
| WO2022021696A1 (en) | Multi-information source-based whole-process blockchain system | |
| US9600602B2 (en) | Combined deterministic and probabilistic matching for data management | |
| CN107133309B (en) | Method and device for storing and querying process example, storage medium and electronic equipment | |
| WO2021159762A1 (en) | Data relationship extraction method and apparatus, electronic device, and storage medium | |
| CN105786998A (en) | Database middleware system and method for processing data through database middleware system | |
| WO2020164211A1 (en) | Data transmission method and apparatus, terminal device and medium | |
| CN111949693B (en) | Data processing device, data processing method, storage medium and electronic equipment | |
| CN112084179B (en) | Data processing method, device, equipment and storage medium | |
| CN112527420A (en) | Interface data flow processing method and device, computer equipment and medium | |
| US11797567B1 (en) | Rapid hyperledger onboarding platform | |
| US11568397B2 (en) | Providing a financial/clinical data interchange | |
| CN111177246A (en) | Service data processing method and device | |
| CN113779609B (en) | Data management method, device, electronic equipment and storage medium | |
| CN110009518A (en) | Card read/write method and device based on block chain | |
| CN112905636A (en) | Data manipulation method, server, and computer-readable medium | |
| CN111863178A (en) | Method, device, medium and electronic device for issuing medical report | |
| CN110598007B (en) | Bill file processing method, device, medium and electronic equipment | |
| CN111625532A (en) | Data blood relationship processing method and device, computer equipment and storage medium | |
| CN112053058B (en) | Index model generation method and device | |
| CN112486967A (en) | Data acquisition method, terminal equipment and storage medium | |
| CN112905617B (en) | Data writing method, server and computer readable storage medium | |
| CN113392138B (en) | Statistical analysis method, device, server and storage medium for private data | |
| US20170364293A1 (en) | Method and apparatus for data processing | |
| CN111369375A (en) | Social relationship determination method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 361000 unit 404, No. 24, guanri Road, phase II, software park, Xiamen, Fujian Applicant after: XIAMEN ZHIYE SOFTWARE ENGINEERING CO.,LTD. Address before: 361000 No.24 guanri Road, Siming District, Xiamen City, Fujian Province Applicant before: XIAMEN ZHIYE SOFTWARE ENGINEERING CO.,LTD. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210312 |