CN114138877A

CN114138877A - Method, device and equipment for realizing theme data service based on micro-service architecture

Info

Publication number: CN114138877A
Application number: CN202111243557.4A
Authority: CN
Inventors: 许琮维; 徐奥; 王振众
Original assignee: Hangxiao Steel Structure Co Ltd
Current assignee: Hangxiao Steel Structure Co Ltd
Priority date: 2021-10-25
Filing date: 2021-10-25
Publication date: 2022-03-04

Abstract

The application discloses a method, a device and equipment for realizing theme data service based on micro-service architecture. The method comprises the following steps: receiving and storing configuration information from a user terminal, setting and storing a plurality of invokable microservices, and providing the invokable microservices to the outside; reading task configuration information in the configuration information, and acquiring bottom layer service data based on the task configuration information; and reading the index configuration information in the configuration information, and calling the callable microservice to perform index distributed calculation on the index configuration information and the bottom-layer service data to obtain a calculation result. According to the method for realizing the theme data service, data are collected and managed in a unified mode based on a micro-service architecture, the standardized and reusable theme data service is provided for an upper-layer service system through automatic data service construction, code redundancy is effectively reduced, background development cost is reduced, and generation efficiency, normalization and accuracy of the data service are improved.

Description

Method, device and equipment for realizing theme data service based on micro-service architecture

Technical Field

The application relates to the technical field of data processing, in particular to a method, a device and equipment for realizing a theme data service based on a micro-service architecture.

Background

With the vigorous development of information technology, enterprises have also come up with the digital revolution of production management and marketing mode. The data service collects and processes the business theme data in the bottom information system based on the business requirements, and generates an interface to the outside so as to meet the data calling requirements of the upper application and enable the data to circulate on a complete business chain.

At present, the internal data service of an enterprise mainly adopts a customized development mode, business personnel put forward actual data requirements based on business logic, information personnel manage a bottom database and perform one-time data service development and deployment, and then data are provided to an upper business department in a data interface mode. However, the customized development brings about a large waste of resources, and the maintainability and reusability thereof are poor.

The services provided by data service construction companies on the market at present cannot perform unified modeling and standard definition on data, for data services with the same business theme, the corresponding relation between services and fields needs to be defined for multiple times, and the problems of complicated data service definition codes, low expansibility and the like exist. In addition, the bottom layer data is complicated, the number of information personnel is limited, the data extraction requirements of large upper layer business quantity and large change are difficult to respond in time, and the problem of inaccurate data caused by non-uniform index statistical calibers is difficult to trace.

Therefore, how to manage underlying heterogeneous multi-source business data, formulate a uniform index specification, automatically and extendably construct reusable data services for business topics, reduce code redundancy while considering calling efficiency, reduce development cost and operation efficiency becomes a problem to be solved in the field.

Disclosure of Invention

The application aims to provide a method, a device and equipment for realizing theme data service based on a micro-service architecture. The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. This summary is not an extensive overview and is intended to neither identify key/critical elements nor delineate the scope of such embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

According to an aspect of an embodiment of the present application, a method for implementing a theme data service based on a micro-service architecture is provided, including:

receiving and storing configuration information from a user terminal, setting and storing a plurality of invokable microservices, and providing the invokable microservices to the outside;

reading task configuration information in the configuration information, and acquiring bottom layer service data based on the task configuration information;

and reading the index configuration information in the configuration information, and calling the callable microservice to perform index distributed calculation on the index configuration information and the bottom-layer service data to obtain a calculation result.

In some embodiments of the present application, the receiving and storing configuration information from a user terminal, setting and storing a plurality of callable microservices, and providing callable microservices to the outside includes:

receiving and storing configuration information from a user terminal, setting and storing a plurality of micro-services which can be called;

receiving a micro-service calling request;

verifying the micro-service calling request, realizing micro-service inquiry, and recommending the inquired micro-service;

and receiving the inquired micro service and sending the inquired micro service to the outside through a calling interface.

In some embodiments of the present application, the task configuration information includes subject information and data sources; the reading of the task configuration information in the configuration information and the obtaining of the underlying service data based on the task configuration information include:

reading the binding relationship between the subject information and the data source and between the subject information and the data source in the task configuration information;

generating bottom layer business data by taking the subject information as granularity based on the binding relationship; the underlying business data includes ETL task profile data.

In some embodiments of the present application, the metric configuration information includes composite metric information and derived metric information; the callable microservice comprises a grouping function and a calculating function; the reading of the index configuration information in the configuration information and the calling of the callable microservice to perform index distributed computation on the index configuration information and the underlying business data to obtain a computation result includes:

reading composite index information and derivative index information in the index configuration information;

and calling the grouping function and the calculation function in the callable microservice, and performing distributed calculation on the composite index information and the derived index information to obtain a calculation result.

According to another aspect of the embodiments of the present application, there is provided a theme data service implementation apparatus based on a micro service architecture, including:

the micro-service calling module is used for receiving and storing configuration information from the user terminal, setting and storing a plurality of calling micro-services and providing the calling micro-services to the outside;

the ETL task execution module is used for reading task configuration information in the configuration information and acquiring bottom layer service data based on the task configuration information;

and the distributed computing engine module is used for reading the index configuration information in the configuration information and calling the callable micro-service to perform index distributed computing on the index configuration information and the bottom layer service data to obtain a computing result.

In some embodiments of the present application, the micro service invocation module includes a callable micro service unit, a registry unit, and an application service unit;

the calling microservice unit is used for receiving and storing configuration information from a user terminal, setting and storing a plurality of calling microservices;

the application service unit is used for receiving a micro-service calling request;

the registration center unit is used for verifying the micro-service calling request, realizing micro-service inquiry and recommending the inquired micro-service;

the application service unit is also used for receiving the inquired micro-service and sending the inquired micro-service to the outside through a calling interface.

In some embodiments of the present application, the task configuration information includes subject information and data sources; the ETL task execution module comprises:

the first reading unit is used for reading the binding relationship between the subject information and the data source and between the subject information and the data source in the task configuration information;

the generating unit is used for generating bottom layer business data by taking the subject information as granularity based on the binding relationship; the underlying business data includes ETL task profile data.

In some embodiments of the present application, the metric configuration information includes composite metric information and derived metric information; the callable microservice comprises a grouping function and a calculating function; the distributed computing engine module comprises:

the second reading unit is used for reading the composite index information and the derived index information in the index configuration information;

and the computing unit is used for calling the grouping function and the computing function in the callable microservice, and performing distributed computing on the composite index information and the derived index information to obtain a computing result.

The technical scheme provided by one aspect of the embodiment of the application can have the following beneficial effects:

according to the method for realizing the theme data service based on the micro-service architecture, the data are uniformly collected and managed based on the micro-service architecture, the standardized and reusable theme data service is provided for an upper-layer service system through automatic data service construction, code redundancy is effectively reduced, background development cost is reduced, and generation efficiency, normalization and accuracy of the data service are improved.

Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the application, or may be learned by the practice of the embodiments.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a flow chart of a method for implementing a subject data service based on a micro service architecture according to an embodiment of the present application;

fig. 2 shows a flowchart of step S10 in fig. 1;

fig. 3 shows a flowchart of step S20 in fig. 1;

fig. 4 shows a flowchart of step S30 in fig. 1;

FIG. 5 is a block diagram illustrating a theme data service implementation apparatus based on micro-service architecture according to another embodiment of the present application;

FIG. 6 is a block diagram illustrating the structure of the microservice invocation module of FIG. 5;

FIG. 7 shows a block diagram of the ETL task execution module of FIG. 5;

FIG. 8 is a block diagram showing the structure of the distributed computing engine module of FIG. 5;

FIG. 9 is a block diagram illustrating a subject data service implementation system based on micro service architecture according to another embodiment of the present application;

FIG. 10 is a block diagram of an electronic device according to another embodiment of the present application;

FIG. 11 shows a schematic diagram of a computer-readable storage medium of another embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is further described with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It will be understood by those within the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The main problem is to define a specific Business process according to actual management requirements, a data service is one of core components of various data platforms, data in a bottom information system is collected and processed, a data interface is generated, and data support is provided for upper-layer applications such as Business Intelligence (BI) and auxiliary decision making.

Referring to fig. 1, an embodiment of the present application provides a method for implementing a theme data service based on a microservice architecture, which may include steps S10 to S30. The method comprises the following steps:

s10, receiving and storing the configuration information from the user terminal, setting and storing a plurality of the invokable microservices, and providing the invokable microservices to the outside.

The invokable microservice refers to a low-coupling reusable microservice which can be invoked independently, and mainly comprises service contents such as data source registration, theme and index definition, external data injection, index grouping processing, interface creation, data query and the like.

The invokable microservices may include, for example, data source configuration microservices, topic configuration microservices, data injection microservices, packet processing microservices, interface configuration microservices, and data query microservices, among others.

The data source configuration micro-service is used for receiving the configuration information of the database, the data file and the data interface type data source, storing the configuration information to a back-end MySQL database and carrying out unified management on heterogeneous data sources; for a database type data source, storing metadata such as a database type, a name, a JDBC connection address, a user name, a password and the like; for a data file type data source, storing metadata such as a file name, uploading time, file size and the like; and for the data interface type data source, storing metadata such as an interface address, a request mode and the like.

And the theme configuration micro-service is used for receiving the theme and the related index configuration information thereof, storing the theme and the related index configuration information to a back-end MySQL database, and performing unified definition and management on the business theme.

The data injection micro-service is used for receiving the data file and the data interface type data to perform instant data injection and storing the data to the HBase database; matching fields with the same name as the header of the file and performing one-time data injection on a data file type data source; for a data interface type data source, an operator completes data injection once when calling an interface to perform write operation once.

The packet processing microservice receives the grouping logic of the derived indexes needing packet statistical analysis and calls the Map object to perform data grouping;

the interface configuration microservice is used for receiving information such as data interface names, access parameters, return values, SQL query statements and the like, storing the information into a rear-end MySQL database, defining the interface return values and uniformly managing the data interfaces;

and the data query microservice is used for reading the interface configuration stored in the MySQL database, screening, querying and extracting data through the HBase data query language, and returning the data through the data interface.

Referring to fig. 2, in some embodiments, step S10 may include the following steps:

s101, receiving and storing configuration information from a user terminal, and setting and storing a plurality of invokable microservices.

The configuration information may include information of data source, theme, data service, etc., which is configured and transmitted by the user through a user terminal, such as a smart phone, a personal computer, etc.

S102, receiving a micro-service calling request.

The micro-service calling request comprises the relevant information of the micro-service required to be called. For example, identification information of various microservices such as a data source configuration microservice, a theme configuration microservice, a data injection microservice, a packet processing microservice, an interface configuration microservice, and a data query microservice, etc. may be included.

S103, the micro-service call request is verified, micro-service query is realized, and the queried micro-service is recommended.

After the micro-service calling request is verified, the corresponding micro-service is inquired and recommended according to the identification information in the micro-service calling request.

And S104, receiving the inquired micro-service and sending the inquired micro-service to the outside through a calling interface.

By calling the callable microservice, the operations of data source registration, theme data related configuration acquisition and storage, data injection, query and the like can be realized.

In certain embodiments, step S10 may include the steps of:

theme configuration step: receiving the subject name and the service description information and storing the subject name and the service description information to a back-end MySQL database;

an atomic index configuration step: receiving information such as index names, service descriptions, data types and the like and storing the information in a back-end MySQL database, wherein the optional index is an object ID (identity) ROW which is an object unique identifier and corresponds to the unique identifier in a data source;

a composite index configuration step: the method comprises the following steps that the composite index is formed by calculating an atomic index through a certain operation rule, and the information such as composite index name, service description, data type and index definition code is received and stored in a back-end MySQL database;

a step of configuring derived indexes: the method comprises the steps of obtaining a derivative index, obtaining a service description, a data type, a grouping function, a calculating function and the like, wherein the derivative index is formed by screening and aggregating an atomic index or a composite index under a certain limiting condition, and the information such as the name, the service description, the data type, the grouping function, the calculating function and the like of the derivative index is received and stored in a back-end MySQL database.

In some embodiments, step S10 includes steps for implementing dynamic microservice discovery, microservice configuration, microservice metadata and traffic management, providing service available instance queries and call validation. Specifically, an application side initiating a service call calls the micro-service through the RPC and provides RestAPI to the outside. Sending service information and reporting the health state; monitoring a server message, receiving registration information and registering a service; acquiring registered service information in a long polling mode, and sending a service request; receiving a client instruction, writing the client instruction into a log, interacting with a server through a consistency module and ensuring log sequential replication; the commands are processed in sequence and the results are returned to the client.

S20, reading the task configuration information in the configuration information, and acquiring the bottom layer service data based on the task configuration information.

The task configuration information comprises theme information and a data source; referring to fig. 3, in some embodiments, step S20 may include the following steps:

s201, reading the binding relationship between the subject information and the data source and between the subject information and the data source in the task configuration information.

For example, based on task configuration, the underlying business data bound by the subject atomic index is extracted to the HBase database regularly. And reading the binding relation between the stored theme and the data source by inquiring the theme configuration information in the MySQL database.

S202, generating bottom layer business data by taking the subject information as granularity based on the binding relationship; the underlying business data includes ETL task profile data.

For example, based on the binding relationship, the ETL task configuration file is generated with the granularity of a subject, and a data source and a write address are specified. And starting a timing task, matching the object identifiers of the source database and the target database, extracting data from the service system database to a back-end data storage database, and setting the default task period as days.

S30, reading the index configuration information in the configuration information, and calling the callable micro-service to perform index distributed calculation on the index configuration information and the bottom layer service data to obtain a calculation result.

The index configuration information comprises composite index information and derived index information; the callable microservices include grouping functions and computation functions.

In some embodiments, as shown with reference to fig. 4, step S30 may include the following steps:

s301, reading the composite index information and the derived index information in the index configuration information.

For example, by querying the index configuration information in the MySQL database, the calculation functions of the composite and derived indexes are read.

S302, the grouping function and the calculating function in the micro-service capable of being called are called, and distributed calculation is carried out on the composite index information and the derived index information to obtain a calculating result.

For example, in some embodiments, objects under all topics in the data storage database are traversed, distributed computation is performed on the composite index and the derived index based on the computation function, and the computation result is stored in the data storage database.

According to the theme data service implementation method based on the micro-service architecture, the heterogeneous multi-source service data at the bottom layer are uniformly collected and managed based on the micro-service architecture, the data standard is defined in a standardized manner through the theme standardized modeling, and the standardized and reusable theme data service is provided for the upper-layer service system through the automatic data service construction, so that the code redundancy is effectively reduced, the background development cost is reduced, and the generation efficiency, the normalization and the accuracy of the data service are improved.

Referring to fig. 5, another embodiment of the present application provides a theme data service implementation apparatus based on a micro service architecture, including:

Referring to FIG. 6, in some embodiments, the microservice invocation module includes a callable microservice unit, a registry unit, and an application service unit;

the registration center unit is used for verifying the service calling request, realizing micro-service inquiry and recommending the inquired micro-service;

In some embodiments, the task configuration information includes topic information and data sources; as shown in fig. 7, the ETL task execution module includes:

In some embodiments, the metric configuration information includes composite metric information and derived metric information; the callable microservice comprises a grouping function and a calculating function; as shown in fig. 8, the distributed computing engine module includes:

Another embodiment of the present application provides a system for implementing a theme data service based on a micro-service architecture, including:

Referring to fig. 9, in the system of this embodiment, the micro service invoking module includes an Nginx gateway, an application server, a micro service unit, and a registry. And the ETL task execution module comprises an ETL task executor, a MySQL database and a business system heterogeneous data source. And the distributed computing engine module comprises a Pheonix Sql server, a Spark computing engine and an HBase database.

The Nginx gateway receives the calling request and forwards the request based on the polling mode load balance;

the application server receives the request forwarded by the Nginx gateway, finds the micro-service to be called through the registration center, and calls the micro-service to be called through the RPC;

when the configuration type micro service is called, such as data source configuration, theme configuration, data interface configuration micro service and the like, the configuration information is stored in the MySQL database;

the ETL task executor regularly reads the binding relationship between the theme in the MySQL database and the data source, and extracts data from the data source to the HBase database;

the Spark calculation engine reads the index configuration, traverses all objects under the HBase database, performs index calculation based on a calculation function in the configuration and stores a calculation result in the HBase database;

when the data interface generated by the configuration is called, the application server calls the data query micro-service, reads the interface configuration in the Mysql server, and uses SQL statements to operate the HBase database through the Phoenix SQL server, so as to query and return the data in the database, thereby completing the interface data query service.

Another embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the program to implement the method for implementing a theme data service based on a micro service architecture described in any of the foregoing embodiments. As shown in fig. 10, the electronic device 10 may include: the system comprises a processor 100, a memory 101, a bus 102 and a communication interface 103, wherein the processor 100, the communication interface 103 and the memory 101 are connected through the bus 102; the memory 101 stores a computer program that can be executed on the processor 100, and the processor 100 executes the computer program to perform the method provided by any of the foregoing embodiments of the present application.

The Memory 101 may include a high-speed Random Access Memory (RAM), and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.

The bus 102 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 101 is used for storing a program, and the processor 100 executes the program after receiving an execution instruction, and the method disclosed in any of the foregoing embodiments of the present application may be applied to the processor 100, or implemented by the processor 100.

Processor 100 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 100. The Processor 100 may be a general-purpose Processor, and may include a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the method in combination with the hardware.

The electronic device provided by the embodiment of the application and the method provided by the embodiment of the application have the same inventive concept and have the same beneficial effects as the method adopted, operated or realized by the electronic device.

Another embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the method for implementing a theme data service based on a micro service architecture described in any of the foregoing embodiments. Referring to fig. 11, the computer-readable storage medium is an optical disc 20, on which a computer program (i.e., a program product) is stored, and when the computer program is executed by a processor, the computer program performs the method of any of the foregoing embodiments.

It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory, or other optical and magnetic storage media, which are not described in detail herein.

The computer-readable storage medium provided by the above-mentioned embodiments of the present application and the method provided by the embodiments of the present application have the same advantages as the method adopted, executed or implemented by the application program stored in the computer-readable storage medium.

It should be noted that:

the term "module" is not intended to be limited to a particular physical form. Depending on the particular application, a module may be implemented as hardware, firmware, software, and/or combinations thereof. Furthermore, different modules may share common components or even be implemented by the same component. There may or may not be clear boundaries between the various modules.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose devices may also be used with the examples based on this disclosure. The required structure for constructing such a device will be apparent from the description above. In addition, this application is not directed to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present application as described herein, and any descriptions of specific languages are provided above to disclose the best modes of the present application.

It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

The above-mentioned embodiments only express the embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A theme data service implementation method based on a micro-service architecture is characterized by comprising the following steps:

2. The method of claim 1, wherein receiving and storing configuration information from a user terminal, setting and storing a plurality of callable microservices, and providing callable microservices externally comprises:

receiving a micro-service calling request;

3. The method of claim 1, wherein the task configuration information includes topic information and data sources; the reading of the task configuration information in the configuration information and the obtaining of the underlying service data based on the task configuration information include:

4. The method of claim 1, wherein the metric configuration information comprises composite metric information and derived metric information; the callable microservice comprises a grouping function and a calculating function; the reading of the index configuration information in the configuration information and the calling of the callable microservice to perform index distributed computation on the index configuration information and the underlying business data to obtain a computation result includes:

5. A theme data service implementation device based on a micro-service architecture is characterized by comprising:

6. The apparatus of claim 5, wherein the microservice invocation module comprises a callable microservice unit, a registry unit, and an application service unit;

7. The apparatus of claim 5, wherein the task configuration information comprises topic information and data sources; the ETL task execution module comprises:

8. The apparatus of claim 5, wherein the metric configuration information comprises composite metric information and derived metric information; the callable microservice comprises a grouping function and a calculating function; the distributed computing engine module comprises:

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the method of any one of claims 1-4.

10. A computer-readable storage medium, on which a computer program is stored, characterized in that the program is executed by a processor to implement the method according to any of claims 1-4.