CN113626447B

CN113626447B - Civil aviation data management platform and method

Info

Publication number: CN113626447B
Application number: CN202111184316.7A
Authority: CN
Inventors: 程华; 张逸; 毛健; 李旺; 张超; 陈健; 郑雄杰; 刘智; 张扬; 梅刚; 彭文韬; 付俊超
Original assignee: Civil Aviation Electronic Technology Co ltd; Minhang Chengdu Information Technology Co ltd
Current assignee: Civil aviation Chengdu Electronic Technology Co., Ltd; MINHANG CHENGDU INFORMATION TECHNOLOGY CO.,LTD.
Priority date: 2021-10-12
Filing date: 2021-10-12
Publication date: 2022-02-22
Anticipated expiration: 2041-10-12
Also published as: CN113626447A

Abstract

The application provides a civil aviation data management platform and a method, and the platform comprises: the switching center is in communication connection with an external system, and the management center is used for receiving the civil aviation service data acquired by the switching center; the treatment center comprises a theme treatment module, and the theme treatment module is used for classifying the civil aviation service data based on a preset theme division rule; and the storage center comprises a preposed subject area, and the preposed subject area is used for storing the subject data obtained after the treatment center classifies the civil aviation service data. In the embodiment of the application, the classification of civil aviation business data can be realized through the theme management module of the management center, so that the associated business entities can be classified into one theme, the associated theme belongs to one theme domain, the data is managed more scientifically through the method, and more convenient service can be provided through the platform.

Description

Civil aviation data management platform and method

Technical Field

The application relates to the technical field of data management, in particular to a civil aviation data management platform and a method.

Background

With the development and progress of aerospace technology, aviation business is increased remarkably at present, and a large amount of data is generated by a civil aviation air administration in the production operation and function management processes. Meanwhile, ecological partners of the civil aviation air administration, such as aviation departments and airports, can also generate a large amount of data in service operation. Data is increasingly becoming a core asset supporting the operation of various systems. In order to improve the management of data of each system and improve the service efficiency and quality of airport ground traffic, the data of each system is necessarily combed and gathered, so that the data management platform is developed at the discretion. However, the current data management platform can only collect and store data, has a single function, and cannot provide more convenient services.

Disclosure of Invention

An object of the embodiment of the application is to provide a civil aviation data management platform and a method, so that data can be classified, more scientific management on the data can be completed, and more convenient service can be provided.

The invention is realized by the following steps:

in a first aspect, an embodiment of the present application provides a civil aviation data management platform, including: the switching center is in communication connection with an external system, the external system comprises an upstream system and a downstream system, and the switching center is used for acquiring civil aviation service data of the upstream system and sending the treated data to the downstream system; the governance center is used for receiving the civil aviation service data acquired by the switching center; the treatment center comprises a theme treatment module, and the theme treatment module is used for classifying the civil aviation service data based on a preset theme division rule; the hierarchy of the theme partitioning rule comprises a theme domain, a theme and a business entity; each of the topic domains includes at least one of the topics; each of the topics includes at least one of the business entities; and the storage center comprises a preposed subject area, and the preposed subject area is used for storing the subject data obtained after the treatment center classifies the civil aviation service data.

In the embodiment of the application, the classification of civil aviation business data can be realized through the theme management module of the management center, so that the associated business entities can be classified into one theme, the associated theme belongs to one theme domain, the data is managed more scientifically through the method, and more convenient service can be provided through the platform.

In combination with the technical solution provided by the first aspect, in some possible implementation manners, the abatement center further includes a multi-source processing module; the multi-source processing module is used for screening the civil aviation business data based on a preset priority rule so as to remove repeated data in the civil aviation business data; wherein the priority rule comprises an upstream system priority and/or a data field priority.

In the embodiment of the application, the treatment center further comprises a multi-source processing module, the repeated data can be screened through the multi-source processing module, the deviation data is integrated, and consistency and correctness of the final data are further guaranteed. Meanwhile, the multi-source processing module supports the configuration of system priority and/or data field priority, and meets the multi-source processing requirements in various scenes.

With reference to the technical solution provided by the first aspect, in some possible implementation manners, the multi-source processing module is further configured to filter the classified subject data based on the preset priority rule to obtain target subject data; the target subject data is the residual subject data after the repeated subject data of the classified subject data is removed; correspondingly, the storage center further comprises a detail subject area, and the detail subject area is used for storing the target subject data.

In this embodiment of the application, the multi-source processing module is further configured to screen the classified subject data based on a preset priority rule to obtain target subject data. By the method, only one piece of data exists in the same type of subject data, and uniqueness and integrity of the subject data in the detail subject area are further guaranteed.

With reference to the technical solution provided by the first aspect, in some possible implementation manners, the switching center obtains the civil aviation service data of the upstream system through a data probe, where the data probe includes: a message data probe, an API data probe and a file data probe; after the message data probe, the API data probe and the file data probe acquire the civil aviation service data, adding the civil aviation service data to an exchange queue of the exchange center; correspondingly, the storage center also comprises a source pasting area; the patch source area is used for storing data in the switching queue.

In the embodiment of the application, data in different formats can be acquired through the message data probe, the API data probe and the file data probe. In addition, the source pasting region is formed by an exchange queue, namely, the exchange queue is used as a memory cache high-performance database, so that the original data can be stored in a short time, and other modules of the platform can be conveniently utilized.

With reference to the technical solution provided by the first aspect, in some possible implementation manners, the data probe is further configured to add message header metadata to the civil aviation service data; the header metadata includes: the data ID, the organization of the data source system, the upstream system of the data source, the data acquisition source of the data probe butt joint and the time stamp for acquiring the civil aviation service data.

In the embodiment of the application, by adding the message header metadata, the traceability of the data can be ensured, and the management of subsequent data is facilitated.

With reference to the technical solution provided by the first aspect, in some possible implementation manners, the abatement center further includes a data processing module; the data processing module is used for carrying out data verification on the civil aviation business data through an APT tool before the theme management module classifies the civil aviation business data.

In the embodiment of the application, the intrusion of the codes can be reduced through the APT tool, so that a developer can automatically finish the generation of the verification codes and the data verification of civil aviation business data without perception.

With reference to the technical solution provided by the first aspect, in some possible implementation manners, the civil aviation data management platform further includes a calculation engine, and an internal bottom layer of the calculation engine adopts an open-source Flink framework; the processing channel of the computing engine adopts a pipeline-filter mode; the processing channel comprises a Source unit, a Processor unit and a Sink unit; the Source unit is used as an input end of the processing channel and used for receiving the civil aviation service data, the Processor unit is used as a filter for processing the data, and the Sink unit is used as an output end of the processing channel and used for storing or sharing the processed data.

In the embodiment of the application, all data processing processes of the computing engine can be realized through the Processor unit, one or more Processor units can be arranged in the pipeline-filter mode, and then in the development process of subsequent services, if a service needs to be added, only one Processor unit needs to be added.

With reference to the technical solution provided by the first aspect, in some possible implementation manners, the Processor unit is specifically configured to abstract the civil aviation service data to generate abstract data; converting the abstract data into a model object of an abstract theme model based on a preset mapping rule; wherein the abstract topic model comprises a root object; each root object corresponds to a theme, and the root objects comprise metadata, basic information and business entities; each abstract data corresponds to a business entity of the abstract topic model.

In the embodiment of the application, the civil aviation business data are abstracted to generate abstract data. And the abstract data is matched with the business entity of the abstract theme model, so that the data is uniformly processed, and the conversion from the data to the theme model is completed. In addition, the abstract theme model is in a tree form, and one root object represents one theme, so that external attributes can be defined for the root object. In addition, the method has strong service independence, high cohesion and good encapsulation. The abstract topic model may support constant extensions.

With reference to the technical solution provided by the first aspect, in some possible implementation manners, the Processor unit is further specifically configured to obtain a style file of the downstream system, abstract style data based on the style file to generate abstract style data, and convert the abstract style data into a model object of an abstract theme model based on the preset mapping rule; the abstract style data is data corresponding to the style file in the civil aviation service data; correspondingly, the Sink unit is further configured to send the subject data obtained by classifying the abstract style data by the Processor unit to the downstream system.

In the embodiment of the application, theme data is encapsulated through the style files of the downstream system, so that the configuration of different requirements is realized.

In a second aspect, an embodiment of the present application provides a civil aviation data management method, which is applied to a server, where the civil aviation data management platform provided in the embodiment of the first aspect is carried in the server, and the method includes: acquiring civil aviation service data of the upstream system through the switching center; receiving the civil aviation service data through the treatment center, and classifying the civil aviation service data based on a preset theme division rule; and storing the theme data after the treatment center classifies the civil aviation service data through the storage center.

In a third aspect, an embodiment of the present application provides a server, including: a processor and a memory, the processor and the memory connected; the memory is used for storing programs; the processor is configured to invoke a program stored in the memory to perform a method as provided in the above-described first aspect embodiment and/or in combination with some possible implementations of the above-described first aspect embodiment.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, performs the method as set forth in the above first aspect embodiment and/or in combination with some possible implementations of the above first aspect embodiment.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.

Fig. 1 is a schematic diagram of interaction between a civil aviation data management platform and an external system according to an embodiment of the present application.

Fig. 2 is a schematic diagram of interaction between another civil aviation data management platform and an external system according to an embodiment of the present application.

Fig. 3 is a schematic structural diagram of a Flink frame according to an embodiment of the present application.

Fig. 4 is a schematic structural diagram of an abstract message model according to an embodiment of the present application.

Fig. 5 is a schematic structural diagram of an abstract topic model according to an embodiment of the present application.

Fig. 6 is a schematic process diagram of a message egress according to an embodiment of the present application.

Fig. 7 is a flowchart illustrating steps of a method for managing civil aviation data according to an embodiment of the present disclosure.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

Referring to fig. 1, an embodiment of the present application provides a civil aviation data management platform. The platform includes: a switching center, a treatment center and a storage center.

Wherein the switching center is communicatively coupled to the external system.

The external system comprises an upstream system and a downstream system, and the switching center is used for acquiring civil aviation service data of the upstream system and sending the treated data to the downstream system.

It should be noted that the upstream system is a data outflow system, and when the transmission path of the data is from the system a to the civil aviation data management platform, the system a is the upstream system. The upstream system may be, but is not limited to, an ACDM (Airport Collaborative Decision-Making) system, a CDM (Collaborative Decision-Making) system, and an outbound system. The upstream system may also be some websites and servers corresponding to the application programs, and the present application is not limited.

And the downstream system is a data inflow system, and when the transmission path of the data is transmitted to the system B by the civil aviation data management platform, the system B is the downstream system. The downstream system may be, but is not limited to, a passenger service system, a flight monitoring system.

And the governance center is used for receiving the civil aviation service data acquired by the switching center. The governance center comprises a theme governance module, and the theme governance module is used for classifying the civil aviation business data based on preset theme dividing rules.

The hierarchy of the theme partitioning rule comprises a theme domain, a theme and a business entity. Each topic domain includes at least one topic; each topic includes at least one business entity. Namely, the business entities are divided into the same theme according to the correlation of the business entities, and then divided into the same theme domain according to the correlation of the theme.

Illustratively, the subject domain may include airport information, personnel information; themes may include flight themes, airport employee themes, passenger themes; the business entities include flight numbers, airplane takeoff times, passenger names, passenger genders, and the like, and the application is not limited.

The above division satisfies the following principles: 1. the same level of classification is at the same level of abstraction. 2. The classification is divided according to traffic relevance. 3. The classification satisfies orthogonality of traffic.

It should be noted that, orthogonality of services means that service entities under different topics are not intersected, and service entities under different topics do not affect each other. For example, the theme of the flight and the theme of the airport staff may both include the owner information, and in order to meet the orthogonality of the business, the owner information is attributed to the theme of the airport staff, so that the repeated appearance of the business entity is avoided, and the maintenance and the updating of the subsequent platform are facilitated through the mode.

And the storage center comprises a preposed subject area, and the preposed subject area is used for storing the subject data obtained after the treatment center classifies the civil aviation service data.

The leading subject area can be stored by using a non SQL (Not Only SQL, non-relational database) database elastic search (a search engine), which allows additional data, and no matter whether the data collected by the switching center is added, modified or deleted on the service, the leading subject area is the additional data, so that the historical track of the subject data after treatment can be completely stored. When the theme data is added, the Delta flag of each piece of theme data needs to be determined according to the data type, so as to mark the operation type of the data.

In summary, in the embodiment of the application, classification of civil aviation business data can be realized through the theme management module of the management center, so that the associated business entities are classified into one theme, the associated theme belongs to one theme domain, more scientific management of data is realized through the method, and more convenient service can be provided through the platform.

Referring to fig. 2, a civil aviation data management platform provided in the embodiment of the present application is described in detail below.

First, a switching center is described, which may specifically include a data acquisition module, a data distribution module, and a data synchronization module.

The data acquisition module is mainly used for acquiring civil aviation service data of an upstream system through the data probe. Data acquisition of different scenes can be realized by selecting different data probes, and data of different formats can be acquired by different data probes. The data probe may be deployed independent of the platform. According to the difference of the collected data source protocol, the data type and the performance requirement, the data probe provided by the embodiment of the application comprises: a message data probe, an API (Application Programming Interface) data probe, and a file data probe.

The data formats supported by the message data probe include, but are not limited to, XML (Extensible Markup Language), JSON (JavaScript Object Notation), message, and the like. The message data probe subscribes messages through the distributed real-time stream processing platform, can acquire the messages in real time once the messages arrive, and releases the acquired messages (civil aviation service data) to an exchange queue of the exchange center.

The API data probe faces to an HTTP (Hyper Text Transfer Protocol) Protocol class (WebService, RestFul) interface, can actively acquire data of a data source in a polling mode through a certain frequency, supports data formats including but not limited to XML, JSON, messages and the like, and immediately sends the acquired civil aviation service data to a switching queue of a switching center.

The data source oriented by the file data probe mainly aims at unstructured data, including pictures, texts, CSV (Comma-Separated Values), DBF (Digital Beam Forming), and other files. After being collected by the probe, the file data can be directly sent to a non-relational storage system, and can also be sent to an exchange queue as a message by reading the content of the file according to different file types.

In addition, the traceability of the data is ensured, so that the management of subsequent data is facilitated. The data probe is also used for adding message header metadata to the civil aviation business data.

The header metadata includes: data ID (Identity document), organization to which a data source system belongs, an upstream system of a data source, a data acquisition source to which the data probe is connected, and a timestamp for acquiring civil aviation service data.

The data ID adopts a Snow flash algorithm, and uniqueness of the data ID in the whole civil aviation data management platform is guaranteed. The organization to which the data source system belongs comprises an airport, an airline department, a department and the like. The acquisition data source to which the data probe is docked represents a data source, such as a website, to which the data probe is directly docked.

The data issuing module is responsible for issuing the data processed by the civil aviation data management platform to a downstream system needing the data in the forms of message pushing, service issuing, short message pushing and the like. The data publishing form adopts a pushing form, the data can be published to an exchange queue of an exchange center, and a downstream system obtains the data in a form of subscription information; if the service publishing form is adopted, the service is published to the outside in a micro-service mode.

The data synchronization module is mainly responsible for completing the synchronization of data. The method mainly comprises an incremental synchronization strategy and a full synchronization strategy.

An increment synchronization strategy: the storage center of the civil aviation data management platform records the complete time sequence of all records, so that the incremental synchronization can be completed by acquiring the data of incremental change through the time difference between two different tasks. For the incremental synchronization of the data source of the database, if the timestamp does not exist, the data change record of the whole database can be restored by combining the Log, so that the incrementally changed data is acquired to complete the incremental synchronization.

Full-scale synchronization strategy: the full-scale synchronization strategy controls the new full-scale data without considering the difference of the data in the two execution times.

Since the above-mentioned data synchronization method is well known in the art, it will not be described herein too much.

The abatement center is described below, and the abatement center may include a multi-source processing module and a data processing module in addition to the theme abatement module.

The data obtained by the switching center comes from the upstream system. For the data collected by the switching center, except for adding necessary message header metadata, the remaining part maintains the original data, and is called as original data. The data processing module is mainly used for processing the original data so as to improve the data quality. The data processing module may be specifically used for data verification, data filtering and data normalization.

The purpose of performing data verification is to avoid the entry of illegally invalid data, while also reducing unnecessary processing logic. The data verification logic is based on the plug-in mode, and can well cope with the change of the verification logic. In the embodiment of the present application, data verification is performed on the civil aviation business data before classification by an Annotation Processor Tool (APT). The APT tool employs an abstract syntax parse tree. The intrusion of codes can be reduced through the APT tool, so that a developer can automatically finish the generation of verification codes and the data verification of civil aviation business data without perception.

Data filtering is primarily from a business perspective, such as filtering data for a particular terminal or flight.

The data standardization is mainly a process of converting data of each upstream system into a uniform protocol object, the data after data standardization processing becomes more complete, and meanwhile, the ambiguity of the data is eliminated.

The multi-source processing module is used for screening the civil aviation business data based on a preset priority rule so as to remove repeated data in the civil aviation business data.

It should be noted that if the upstream system accessed by the data platform includes multiple pieces of data which send the same theme for the same service and the same scene, the data is considered to be a phenomenon of data duplication; if the data pushed by different upstream systems are different for the same service scene, the phenomenon of data deviation is considered to exist. And a multi-source processing function is introduced, so that the problems of data repetition and data deviation can be solved.

In the embodiment of the present application, the priority rule includes an upstream system priority and/or a data field priority.

For example, the upstream system priority may be understood as ranking different upstream systems, such as ranking system a higher than system B, and processing system a data and deleting system B data that is the same as system a after acquiring the same data from system a and system B.

Illustratively, the data field priority may be understood as ranking different business entities. For example, if the passenger information of system a is ranked higher than the passenger information of system B, the passenger information acquired from system B is deleted by processing the data of system a after the passenger information is acquired from both system a and system B.

In addition, when both the upstream system priority and the data field priority are included in the priority rule, the data field priority is prioritized. Whether the data is the unique data of the same business, the same scene and the same theme can be judged through the global identifier, and when one piece of multi-source data is processed, the historical information of the data is recorded so as to help the multi-source processing module to determine whether the data is processed.

Therefore, the repeated data can be screened through the multi-source processing module, deviation data can be integrated, and consistency and correctness of final data are guaranteed. Meanwhile, the multi-source processing module supports the configuration of system priority and/or data field priority, and meets the multi-source processing requirements in various scenes.

Optionally, the multi-source processing module is further configured to screen the classified subject data based on a preset priority rule to obtain target subject data; the target subject data is the subject data which is the rest of the classified subject data after the repeated subject data is removed.

The storage center will be explained below. In the embodiment of the present application, the storage center further includes a detail topic area and a patch source area.

The detail subject area is used for storing the target subject data. The detail subject area is stored by using NoSQL, and only one record appears for the same record of the same type under the support of multi-source processing, namely the record fuses the data of the preposed subject area, thereby ensuring the uniqueness and the integrity of the subject data of the detail subject area.

The patch source area is used for storing data in the switching queue. The source pasting region is formed by an exchange queue, namely, the exchange queue is used as a memory cache high-performance database, so that the original data can be stored in a short time, and other modules of the platform can be conveniently utilized.

Optionally, the storage center may further include: a classification area, a market-gathering area and a metadata area.

The filing region is used for storing original messages sent by an upstream system, the data are mainly used for data consanguineous relationship query, historical statistical analysis, massive data mining and the like, meanwhile, backup of the original data is provided, and recovery of the data is guaranteed under the condition that the data are lost. Since the historical data has the characteristics of long life cycle, various data formats, large data volume, no need of modification and the like, the data is stored according to the original message format except for adding the metadata tags.

The bazaar is created with the needs of the business system, and the data it needs comes from the subject area (including the pre-subject area and the detail subject area). The mart region follows a modeling method of a star model, data are divided into a fact table and a dimension table, subsequent multi-dimensional analysis is supported conveniently, and statistical analysis requirements of different scenes are met. And selecting MySQL for storage in the urban area. Over time, the data for the bazaar becomes more and more. Therefore, the mart areas can be backed up and cleared periodically, or the mart areas are subjected to database partitioning, and a Shardingsphere framework is introduced to realize access support of the database partitioning.

The metadata area is used for storing metadata of the data center, and includes three types of metadata, namely technical metadata, business metadata and management metadata. The stored metadata can be used for relationship query and data audit, and helps data production units to effectively maintain and manage data.

Optionally, the civil aviation data management platform further comprises an open center.

The open center is a channel that is open to the outside of the data, and includes a data query service and an OLAP (Online Analytical Processing) analysis service according to the difference between the data open capability and the service property.

Data queries support queries and retrieval of various conditions, including full text searches. The ElasticSearch is selected from the file classifying area and the theme area of the civil aviation data management platform, the joint query of semi-structure and structured data is supported, and Chinese word segmentation and full text retrieval are also supported. The market region of the civil aviation data management platform selects a relational database MySQL, and the MySQL is compatible with the support conditions of international SQL92 and SQL2003 standards.

OLAP analysis performs multi-dimensional analysis on each topic data, including supporting dimension-based drilling, index statistics, classification, aggregation and other statistical analysis operations.

Optionally, the civil aviation data management platform further comprises an asset center.

In order to better manage and master data, data asset combing needs to be carried out on data acquired, stored and processed by a civil aviation data management platform, and an asset center is established. The asset center follows data asset management practice, provides main data management and metadata management functions, defines data standards for a data platform, creates enterprise data assets facing the civil aviation field, and supports a user to perform data modeling on a market region according to the data standards and the metadata. The asset center also provides data quality management, and data quality reports can be generated as needed.

The data standards include a base class data standard and an index class data standard.

Basic data standards: the data processing method generally comprises a reference data and main data standard, a logic data model standard, a physical data model standard, a metadata standard, a public code and coding standard and the like, and is mainly represented by original attribute data which is not subjected to mathematical calculation and is directly generated by each production system of civil aviation, such as flight numbers, machine numbers, navigation departments, seat numbers, air lines and the like.

Index class data standard: generally, the method is divided into a basic index standard and a calculation index (also called a combined index) standard. The basic indexes generally do not contain dimension information and have specific business and economic meanings, such as delayed flight number and the like. The calculation index is usually calculated by more than two basic indexes, such as passenger seat rate, flight clearance normal rate and the like.

According to the relevant principle and method of data standardization, the business information of mainstream in the civil aviation industry is scientifically classified and coded, and a data standard specification facing to the aviation department direction of the civil aviation industry is established and can be used as the basis of information exchange and sharing.

Metadata is data describing data, and can be classified into technical metadata, business metadata, and management metadata according to usage.

The technical metadata describes data of related concepts, relations and rules in the technical field in the data system; the method comprises the following steps of definition of objects and data structures in a ground support data platform, mapping of source data to destination data, description of data conversion and the like.

The service metadata describes data of related concepts, relations and rules of the service field in the data system; including business terms, information classification, metrics, statistical calibers, etc.

The management metadata describes data of related concepts, relations and rules in the management field in the data system, and mainly comprises information of personnel roles, station responsibilities, management processes and the like.

Metadata management is an important basis for data asset management, and is the planning, enforcement, and control activities performed to obtain high-quality, integrated metadata. The ground support data platform supports the functions of collecting, auditing and basically managing the metadata of the theme zone and the urban area, and allows all the metadata to be inquired.

Optionally, the civil aviation data management platform further includes a management and control center.

The management and control center provides a management function of the data platform, and the user mainly operates the civil aviation data management platform through the management and control center. The system mainly aims at the whole platform to carry out management and monitoring, comprises functional modules such as configuration management, authority management, identity authentication, resource scheduling, operation and maintenance monitoring, cluster management, data backup and the like, and meets the operation requirements of managers and operation and maintenance personnel.

Optionally, the civil aviation data management platform further comprises a computing engine.

Referring to fig. 3, the computing engine is an autonomously developed "haina (haina) engine" whose bottom layer employs an open-source Flink framework. The basic unit of the Flink framework is Job. All data processing of the civil aviation data management platform is completed by Job. One Job corresponds to a physically deployed Environment (Environment), a plurality of jobs may share the same Environment, and one Job may include one to a plurality of flows, each Flow corresponding to one processing channel. The processing channel adopts a pipeline-filter mode; the processing channel comprises a Source unit, a Processor unit and a Sink unit.

The Source unit is used as an input end of the processing channel and used for receiving civil aviation service data. Either a web service (http) or a Message Queue (MQ) may be connected.

The Processor unit serves as a filter for processing data, and the logic for data processing is assumed by the Processor unit. Namely, the Processor unit is used for processing the civil aviation service data received by the Source unit.

The setting principle of the Processor unit can comprise: 1. the processing of the data stream on the service can be divided into a plurality of stages, and each Processor unit corresponds to one stage. 2. The side-effect and side-effect free responsibilities are separated into different Processor units. 3. Separating the traffic that needs to access the external system into different Processor units. 4. The short code of the Processor unit is ensured, and the real responsibility of the Processor unit is transferred to other classes. 5. The type parameters T and O corresponding to the upstream and downstream of each Processor unit, i.e. mapconnection or other interfaces, should adopt the model object defined by the civil aviation data management platform as much as possible instead of some basic types such as String. 6. The naming rule of each Processor unit adopts the mobile guest phrase and takes the Processor as a suffix of the class. For example, a flight message is split into a plurality of flight seats, and named as splitfighttostandsps processor. 7. External data required by each Processor unit is transferred through a constructor of the Processor unit.

And the Sink unit is used as an output end of the processing channel and used for storing or sharing the processed data. Message Queues (MQ) or Databases (DB) may be connected.

It can be seen that, in the embodiment of the present application, all data processing processes of the computing engine may be implemented by a Processor unit, one or more Processor units may be set in the pipeline-filter mode, and then, in the development process of subsequent services, if a service needs to be added, only one Processor unit needs to be added. For example, when a flight information extraction service from beijing to shanghai needs to be added, only one Processor unit needs to be added and a corresponding code is designed, and other codes do not need to be changed.

The above manner supports horizontal expansion of data computing capacity, that is, based on a Flink framework, multiple channels (corresponding to multiple flows) can be executed in parallel, each channel includes a Source unit, a Processor unit and a Sink unit, and different flows can process different service requirements in parallel.

In the embodiment of the application, the Processor unit is specifically configured to abstract the civil aviation service data to generate abstract data; and converting the abstract data into a model object of the abstract theme model based on a preset mapping rule.

It is understood that the abstract message model and the abstract topic model are pre-constructed. Wherein, the abstract message model can refer to fig. 4, and the abstract topic model can refer to fig. 5.

Since the data of the civil aviation data management platform are usually XML and JSON types, the two types of messages (civil aviation business data) are abstracted through an abstraction model. The MessageNodes and the MessageNodes form a message tree, and one MessageNode (abstract data) corresponds to a business entity of the abstract theme model. XMLNodes is an XML type message and JSONNodes is a JSON type message.

The abstract topic model includes a root object (RootEntity); each root object (RootEntity) corresponds to a theme, and the root objects comprise metadata (Meta), Basic information (Basic) and business entities (Business entity); each abstract data corresponds to a business entity of the abstract topic model.

Accordingly, each business entity can be further subdivided into leaf entity and branch entity to achieve continuous extension. For example, if the business entity is flight information, the business entity may be further subdivided, such as, for example, leafEntity represents a section, and branch represents a terminal.

The corresponding relation between the abstract message model and the abstract theme model is provided through a preset mapping rule. The preset mapping rule may be implemented in a mapping schema file. The file stores the business entity corresponding to each abstract message.

In addition, it should be noted that the abstract topic model only includes the corresponding relationship between the topic and the business entity, and the relationship between the topic and the topic domain is realized through the distribution of the database, as if the data of the same topic is stored in the same database.

Therefore, in the embodiment of the application, the civil aviation business data are abstracted to generate the abstract data. And the abstract data is matched with the business entity of the abstract theme model, so that the data is uniformly processed, and the conversion from the data to the theme model is completed. In addition, the abstract theme model is in a tree form, and one root object represents one theme, so that external attributes can be defined for the root object. In addition, the method has strong service independence, high cohesion and good encapsulation. The abstract topic model may support constant extensions.

Referring to fig. 6, the Processor unit is further specifically configured to obtain a style file of a downstream system, abstract the style data based on the style file to generate abstract style data, and convert the abstract style data into a model object of an abstract theme model based on a preset mapping rule.

The abstract style data is data corresponding to the style file in the civil aviation service data.

Correspondingly, the Sink unit is also used for sending the subject data (export data) classified by the Processor unit on the abstract style data to the downstream system.

For example, assume that the downstream system focuses on flight time, passenger information. The Processor unit only extracts the two types of data and constructs corresponding subject data to be sent to the downstream system. And the theme data is encapsulated through the style files of the downstream system, so that the configuration of different requirements is realized.

Referring to fig. 7, an embodiment of the present application further provides a civil aviation data management method, which is applied to a server on which the civil aviation data management platform provided in the foregoing embodiment is mounted, and the method includes steps S101 to S103.

Step S101: and acquiring civil aviation service data of an upstream system through the switching center.

Step S102: and receiving the civil aviation service data through the treatment center, and classifying the civil aviation service data based on a preset theme division rule.

Step S103: and storing the subject data after classifying the civil aviation service data by the storage center.

It should be noted that, since the processing procedure is already described in the foregoing embodiment, the description is not repeated here, and the same parts may be referred to each other.

Based on the same inventive concept, the embodiment of the application also provides a server. The server carries the civil aviation data management platform provided by the embodiment and can execute the civil aviation data management method. Structurally, the server comprises: a processor and a memory.

The processor and the memory are electrically connected, directly or indirectly, to enable data transmission or interaction, for example, the components may be electrically connected to each other via one or more communication buses or signal lines.

The processor may be an integrated circuit chip having signal processing capabilities. The Processor may also be a general-purpose Processor, for example, a Central Processing Unit (CPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a discrete gate or transistor logic device, or a discrete hardware component, which may implement or execute the methods, steps, and logic blocks disclosed in the embodiments of the present Application. Further, a general purpose processor may be a microprocessor or any conventional processor or the like.

The Memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and electrically Erasable Programmable Read-Only Memory (EEPROM). The memory is used for storing a program, and the processor executes the program after receiving the execution instruction.

It should be noted that, as those skilled in the art can clearly understand, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

Based on the same inventive concept, embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the computer program performs the methods provided in the above embodiments.

The storage medium may be any available medium that can be accessed by a computer or a data storage device including one or more integrated servers, data centers, and the like. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.

In addition, units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

Furthermore, the functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A civil aviation data management platform, comprising:

the switching center is in communication connection with an external system, the external system comprises an upstream system and a downstream system, and the switching center is used for acquiring civil aviation service data of the upstream system and sending the treated data to the downstream system;

the governance center is used for receiving the civil aviation service data acquired by the switching center; the treatment center comprises a theme treatment module, and the theme treatment module is used for classifying the civil aviation service data based on a preset theme division rule; the hierarchy of the theme partitioning rule comprises a theme domain, a theme and a business entity; each of the topic domains includes at least one of the topics; each of the topics includes at least one of the business entities;

the storage center comprises a preposed subject area, and the preposed subject area is used for storing subject data obtained after the treatment center classifies the civil aviation service data;

the civil aviation data management platform further comprises a calculation engine, wherein an open-source Flink framework is adopted in the internal bottom layer of the calculation engine; the basic unit of the Flink framework is Job, wherein one Job comprises at least one Flow, and each Flow corresponds to one processing channel; different flows may be used to process different services in parallel; the treatment channel adopts a pipeline-filter mode; the processing channel comprises a Source unit, a Processor unit and a Sink unit; the Source unit is used as an input end of the processing channel and used for receiving the civil aviation service data, and the Processor unit is used as a filter for processing the data; the Processor unit is additionally arranged to additionally arrange the business of the civil aviation data management platform, and the Sink unit is used as the output end of the processing channel and used for storing or sharing the processed data; the Processor unit is specifically used for abstracting the civil aviation service data to generate abstract data; converting the abstract data into a model object of an abstract theme model based on a preset mapping rule; wherein the abstract topic model comprises a root object; each root object corresponds to a theme, and the root objects comprise metadata, basic information and business entities; each abstract data corresponds to a business entity of the abstract theme model;

the treatment center also comprises a multi-source processing module; the multi-source processing module is used for screening the civil aviation business data based on a preset priority rule so as to remove repeated data in the civil aviation business data; wherein the priority rule comprises an upstream system priority and/or a data field priority.

2. The civil aviation data management platform of claim 1, wherein the multi-source processing module is further configured to filter the classified subject data based on the preset priority rule to obtain target subject data; the target subject data is the residual subject data after the repeated subject data of the classified subject data is removed;

correspondingly, the storage center further comprises a detail subject area, and the detail subject area is used for storing the target subject data.

3. The civil aviation data management platform of claim 1, wherein the switching center obtains the civil aviation service data of the upstream system through a data probe, and the data probe comprises: a message data probe, an API data probe and a file data probe;

after the message data probe, the API data probe and the file data probe acquire the civil aviation service data, adding the civil aviation service data to an exchange queue of the exchange center;

correspondingly, the storage center also comprises a source pasting area; the patch source area is used for storing data in the switching queue.

4. The civil aviation data management platform of claim 3, wherein the data probe is further configured to add header metadata to the civil aviation business data;

the header metadata includes: the data ID, the organization of the data source system, the upstream system of the data source, the data acquisition source of the data probe butt joint and the time stamp for acquiring the civil aviation service data.

5. The civil aviation data management platform of claim 1, wherein the abatement center further comprises a data processing module;

the data processing module is used for carrying out data verification on the civil aviation business data through an APT tool before the theme management module classifies the civil aviation business data.

6. The civil aviation data management platform of claim 1, wherein the Processor unit is further specifically configured to obtain a style file of the downstream system, abstract the style file based on the style file to generate abstract style data, and convert the abstract style data into a model object of an abstract theme model based on the preset mapping rule; the abstract style data is data corresponding to the style file in the civil aviation service data;

correspondingly, the Sink unit is further configured to send the subject data obtained by classifying the abstract style data by the Processor unit to the downstream system.

7. A civil aviation data management method, which is applied to a server, wherein the server carries the civil aviation data management platform as claimed in claim 1, and the method comprises the following steps:

acquiring civil aviation service data of the upstream system through the switching center;

receiving the civil aviation service data through the treatment center, and classifying the civil aviation service data based on a preset theme division rule;

and storing the theme data after the treatment center classifies the civil aviation service data through the storage center.