CN114201538B

CN114201538B - Data processing method and device, intelligent equipment and computer storage medium

Info

Publication number: CN114201538B
Application number: CN202010910100.3A
Authority: CN
Inventors: 陈瑶
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2020-09-02
Filing date: 2020-09-02
Publication date: 2024-09-10
Anticipated expiration: 2040-09-02
Also published as: CN114201538A

Abstract

The embodiment of the application discloses a data processing method and device, intelligent equipment and a computer storage medium, wherein the method comprises the following steps: analyzing the acquired data construction task, obtaining source data from source data environment equipment by executing a data selection script associated with the data construction task, storing files in the source data environment equipment, wherein the source data can be stored in various databases, is convenient to manage and read, and can be added with logs about the storage and the reading of each data in the source data when necessary; loading target data corresponding to the source data into a target data environment device by executing a data loading script associated with the data construction task; and feeding back a notification message about the loading of the target data to the target data environment device to the management platform so as to process the target data in the target data environment device. By adopting the application, the data construction time of the user is saved, and the automation requirement of data construction is met.

Description

Data processing method and device, intelligent equipment and computer storage medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus, an intelligent device, and a computer storage medium.

Background

With the development of computer technology and communication technology, people have stepped into a digital society, and people can generate various data in daily work, life and study and can also use various data. Various schemes and products concerning functions of data acquisition, storage, management, transmission, etc. are also widely used in various industries. Meanwhile, massive data can be stored into a cloud server through a cloud technology, and the data stored in the cloud server can be applied to industries of medical treatment, finance, banking, government affairs, games, internet of things and the like.

For scattered data, the data can be converted into a usable form by means of data construction, for example by means of a data table. The purpose of data construction by different data construction modes is to meet the requirements of different users on characteristic data. In order to construct data according to new data requirements, the existing technical scheme is that a user is required to manually select data from a data storage environment and then perform a series of manual operations such as data copying and restoring, so that time is consumed and efficiency is low.

Disclosure of Invention

The embodiment of the application provides a data processing method, a data processing device, intelligent equipment and a computer storage medium, which can automatically select data from a source data environment and load the data into a target data environment based on a script.

In one aspect, an embodiment of the present application provides a data processing method, including:

Acquiring a data construction task, wherein the data construction task is generated according to the detected data construction event;

Analyzing the data construction task, acquiring a data selection script related to the data construction task, and acquiring source data by executing the data selection script, wherein the data selection script is used for acquiring the source data from source data environment equipment;

Acquiring a data loading script associated with the data construction task, and loading target data into target data environment equipment by executing the data loading script, wherein the target data is obtained according to the source data;

and feeding back a notification message about the loading of the target data to the target data environment device to a management platform, wherein the notification message is used for instructing the management platform to respond to the data construction event and process the target data in the target data environment device.

On the other hand, the embodiment of the application also provides a data processing device, which comprises:

The acquisition module is used for acquiring a data construction task, wherein the data construction task is generated according to the detected data construction event;

The processing module is used for analyzing the data construction task, acquiring a data selection script related to the data construction task, and obtaining source data by executing the data selection script, wherein the data selection script is used for acquiring the source data from source data environment equipment; acquiring a data loading script associated with the data construction task, and loading target data into target data environment equipment by executing the data loading script, wherein the target data is obtained according to the source data;

And the notification module is used for feeding back a notification message about the loading of the target data to the target data environment equipment to the management platform, wherein the notification message is used for instructing the management platform to respond to the data construction event and process the target data in the target data environment equipment.

Correspondingly, the embodiment of the application also provides intelligent equipment, which is characterized by comprising a storage device and a processor, wherein the storage device stores a computer program, and the processor calls the computer program to realize the data processing method.

Accordingly, an embodiment of the present application also provides a computer storage medium having a computer program stored therein, where the above-mentioned data processing method is implemented when the computer program is executed by a processor.

According to the embodiment of the application, the data required by a part can be acquired from the source data environment equipment by automatically running the data selection script, and the acquired data can be automatically loaded into the target environment by automatically running the data loading script, so that the automatic requirement of data construction can be met, and the personalized data requirement of a user can be met. In the whole process, a user only needs to write part of scripts and configure the scripts to the data construction task, and the data construction task can be automatically selected and loaded by executing the data construction task through the application corresponding to the application, so that the selection and the loading do not need manual participation of the user, the time of the user is saved, and the automation requirement of the data construction is met.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic flow diagram of one of the data construction processes according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a framework for data construction in accordance with an embodiment of the present application;

FIG. 3 is a schematic diagram of the construction and management of a data processing related environment of an embodiment of the present application;

FIG. 4 is a flow chart of a data processing method according to an embodiment of the present application;

FIG. 5a is a flow chart of a data processing method according to an embodiment of the present application;

FIG. 5b is a flowchart of partitioning source data to facilitate synchronous extraction and conversion of data according to an embodiment of the present application;

FIG. 6 is a flow chart of an interactive method of data processing according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a configuration interface according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention;

Fig. 9 is a schematic structural diagram of an intelligent device according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

In the embodiment of the application, for the data construction task, after the data construction task is acquired, the source data can be automatically acquired from the source data environment equipment based on the data selection script and the data loading script indicated by the task configuration information in the data construction task, and the target data acquired according to the source data is automatically loaded into the target data environment equipment, so that a plurality of functional platforms or applications capable of acquiring data from the target data environment equipment can conveniently acquire the required target data. After the data selection script is executed, part of source data can be acquired from the source data environment equipment, the data which are needed for the data can be defined in the data selection script, and meanwhile, the data loading script can load target data corresponding to the selected source data into a target data environment for use, so that the automation requirement of data construction can be met, and the personalized data requirement of a user can be met. In the whole process, a user only needs to write a script, based on the written script, the automatic processing of data selection and loading can be realized through the application corresponding to the method, the manual participation of the user is not needed in the selection and loading process, the time of the user is saved, and the automatic requirement of data construction is met.

The servers (such as the intelligent devices mentioned in the embodiments and the devices included in the management platform) according to the embodiments of the present application may be independent physical servers, or may be a server cluster or a distributed system formed by a plurality of physical servers, or may be cloud servers that provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligent platforms. The terminal may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal and the server may be directly or indirectly connected through wired or wireless communication, and the present application is not limited herein.

The embodiment of the application can realize related data processing operation in the data construction process based on Cloud technology, wherein the Cloud technology refers to a hosting technology for unifying serial resources such as hardware, software, network and the like in a wide area network or a local area network to realize calculation, storage, processing and sharing of data. Cloud technology (Cloud technology) is based on the general terms of network technology, information technology, integration technology, management platform technology, application technology and the like applied by Cloud computing business models, and can form a resource pool, so that the Cloud computing business model is flexible and convenient as required. Cloud computing technology will become an important support. Background services of technical networking systems require a large amount of computing, storage resources, such as video websites, picture-like websites, and more portals. Along with the high development and application of the internet industry, each article possibly has an own identification mark in the future, the identification mark needs to be transmitted to a background system for logic processing, data with different levels can be processed separately, and various industry data needs strong system rear shield support and can be realized only through cloud computing.

The source data and the target data related to the application can be stored in the cloud by a cloud storage mode, and the source data environment equipment and the target data environment equipment can also be equipment in the cloud, such as certain servers for realizing the data storage function. Cloud storage (cloud storage) is a new concept that extends and develops in the concept of cloud computing, and a distributed cloud storage system (hereinafter referred to as a storage system for short) refers to a storage system that integrates a large number of storage devices (storage devices are also referred to as storage nodes) of various types in a network to work cooperatively through application software or application interfaces through functions such as cluster application, grid technology, and a distributed storage file system, so as to provide data storage and service access functions for the outside. The application can select the source data through the software or the application interfaces, and load the source data into a new data environment for different data users.

The storage method of the cloud storage system comprises the following steps: when creating logical volumes, each logical volume is allocated a physical storage space, which may be a disk composition of a certain storage device or of several storage devices. The client stores data on a certain logical volume, that is, the data is stored on a file system, the file system divides the data into a plurality of parts, each part is an object, the object not only contains the data but also contains additional information such as a data Identification (ID) and the like, the file system writes each object into a physical storage space of the logical volume, and the file system records storage position information of each object, so that when the client requests to access the data, the file system can enable the client to access the data according to the storage position information of each object.

The process of allocating physical storage space for the logical volume by the storage system specifically includes: physical storage space is divided into stripes in advance according to the set of capacity measures for objects stored on a logical volume (which measures tend to have a large margin with respect to the capacity of the object actually to be stored) and redundant array of independent disks (RAID, redundant Array of INDEPENDENT DISK), and a logical volume can be understood as a stripe, whereby physical storage space is allocated for the logical volume.

The Database (Database), which can be considered as an electronic filing cabinet, is a place for storing electronic files, and users can perform operations such as adding, inquiring, updating, deleting and the like on the data in the files. A "database" is a collection of data stored together in a manner that can be shared with multiple users, with as little redundancy as possible, independent of the application.

The Database management system (Database MANAGEMENT SYSTEM, DBMS) is a computer software system designed for managing databases, and generally has basic functions of storage, interception, security, backup and the like. The database management system may classify according to the database model it supports, e.g., relational, XML (Extensible Markup Language ); or by the type of computer supported, e.g., server cluster, mobile phone; or by the query language used, such as SQL (structured query language (Structured Query Language), XQuery, or by the energy impact emphasis, such as maximum-scale, maximum-speed, or other classification means, regardless of which classification means is used, some DBMSs can cross-category, for example, while supporting multiple query languages.

In the embodiment of the application, the data construction process is divided into a plurality of stages, mainly comprising selection (Select), extraction (Extract), conversion (Transform) and loading (Load). The selection (Select) refers to selection of source data, where the source data may be data in environments such as a database, a cache, a file, a network, and the like. A Select script (i.e., a data selection script) may be written to obtain data from the above-described types of source data. Extraction (Extract) refers to extracting data obtained from source data. Such as extracting data from certain fields in the database. An Extract script (i.e., a data extraction script) may also be written to Extract the desired data from the source data selected by the Select script. The conversion (Tranfrom) refers to converting the data extracted by the Extract script, and implements a specific conversion process according to the user-defined Tranfrom script (i.e., the data conversion script), where the conversion process of the data includes, for example, deletion of some unnecessary data items, conversion between some data items, and so on. Load refers to importing converted data into a target database or loading the converted data into a target data environment, so that a corresponding data demand end can read data from the target database or the target data environment. The data loading is implemented by a user-defined Load script (i.e., a data loading script).

In one embodiment, a schematic diagram of a data construction process is shown in fig. 1, where the data construction process is divided into 4 stages, and the data construction can be automatically completed in the 4 stages by using a user-defined script. It should be noted that, when the user does not define the custom script at a certain stage, no processing is performed on the data at this stage, for example, when the Extract script is not defined, the data extraction is not performed on the source data acquired through the Select script, and for example, when the Tranfrom script is not defined, no data conversion processing exists.

Referring to fig. 2 again, which is a schematic diagram of a framework related to data construction according to an embodiment of the present application, fig. 2 only illustrates one possible implementation, and the embodiment of the present application is not limited to a specific architecture, for example, in fig. 2, the management platform and the smart device for performing data processing are two different components, but in other embodiments, the management platform and the smart device for performing data processing may be the same device or belong to the same system.

In the source data environment device, various types of source data are stored, and in fig. 2, cc (configuration center) type source data, bc (business center) type source data, key (KEY SYSTEM) type source data, CMDB (configure management database, configuration management database) type source data, CKV (Cloud Key-Value) type source data, DB (database) type source data are shown, and of course, other types of source data may be included. The source data environment device may be a cloud server or a dedicated storage device.

In fig. 2, the core service module and the infrastructure module may belong to the same management platform, and are devices or device groups that provide specific services in the management platform. The management platform can select the required source data from the source data environment equipment on one hand, and can process the acquired source data through the data construction agent on the other hand and then load the processed source data into the target data environment equipment. The data construction agent may be installed in an intelligent device (for example, a server) in the management platform, or may be installed in a target data environment device, where the data acquired from the management platform is processed and then stored in CC source data, BC source data, key source data, CMDB source data, CKV source data, and DB source data configured in the target data environment device. The target data environment device may also be one of the intelligent devices (e.g., on a server) in the management platform.

The management platform performs data interaction with other external systems or platforms by providing an external interface. In fig. 2, the data interaction with the environment management system through the environment data construction interface, the data interaction with the automated test system through the service data construction interface, the data interaction with the account management system through the account management interface, and other special or general interfaces may exist for data interaction, which is not limited in this aspect of the application. The interactive data may include a data construction event initiated by a corresponding system or platform, and the management platform responds to the data construction event to provide the data loaded by the data construction agent in the target data environment device, for example, the target data environment including the CC source data, the BC source data, the Key source data, the CMDB source data, the CKV source data, and the DB source data in the target data environment device in fig. 2 is provided as interactive data to the environment management system.

Referring to fig. 3, a schematic diagram of construction and management of a data processing related environment according to an embodiment of the present application mainly includes: resource initialization, where a resource includes a machine, an installation package for a service, and the like. In this stage, the machine is initialized and prepared, the environment in which the system service operates is installed, and the installation package of each service and the mirror image of the container which need to be installed are prepared, and when a new environment and a target data environment are built, services such as mysql can be installed by downloading the installation package, or the services can be packaged in the mirror image in advance, and then the mirror image is downloaded and the container is started to complete the deployment of the services. Container arrangement and networking, and the installation and deployment of each service module are completed at this stage. Specifically, the mirror image is pulled and the container is started, and meanwhile, the configuration of each service is completed. And data construction, wherein configuration data, service data (such as an account number) and the like required by the service are constructed and generated at the stage. And checking the service state to check whether the running state of the service is normal. And (5) verifying the environment, namely verifying whether the whole environment is available or not through use cases, and whether the whole environment can be delivered to a user for use or not. The environment verification is mainly to verify the built target data environment, and the test case is operated in the environment in a test case mode, and the test case execution reaches the expectation, so that the environment is correctly available. The source data environment device mentioned later in the embodiment of the present application refers to a server and other devices loaded with an active data environment, the target data environment device refers to a built target data environment server and other devices, the data environment includes services and data, and corresponding data (source data or target data) is in mysql, file, ckv and the like.

For the data construction steps mentioned in fig. 3, in the embodiment of the present application, on the basis of fig. 2, the data construction process approximately includes: triggering of the data construction task. The triggering of tasks is typically performed by an interface created by the management platform or other third party system invoking data. Preparing a construction task, wherein the data construction core Server writes related scripts of data construction into a DB or a cache, creates the data construction task and simultaneously issues the task to the data construction Agent. The construction task is executed, the data construction Agent analyzes the data construction task, downloads the user-defined script required by the execution process, starts the execution of the task, and executes according to 4 stages, namely: select, extract, transform, load, etc. 4 stages. And reporting an execution result. The data construction Agent reports the execution process and result of the data construction to the data construction core Server, and the execution state of the task is updated by the data construction core Server.

Referring again to fig. 4, a flowchart of a data processing method according to an embodiment of the present application may be implemented by an intelligent device, which may implement the method by running the above-mentioned data construction agent. The intelligent device can be a single server or the like, or can be a part of a management platform. The method comprises the following steps.

S401: and acquiring a data construction task, wherein the data construction task is generated according to the detected data construction event. The data construction task may be sent by a core service module in fig. 2, where the core service module generates the data construction task and sends the data construction task to the intelligent device according to the embodiment of the present application when receiving a data construction event initiated by a user or receiving the data construction event from an API. The data construction task comprises a data selection script, a data recording script, a device identifier of source data environment device, an identifier of target data environment device and the like which are set according to the data construction event. In other embodiments, the data construction task may also include data extraction scripts, data conversion scripts, and the like.

S402: analyzing the data construction task, acquiring a data selection script related to the data construction task, and acquiring source data by executing the data selection script, wherein the data selection script is used for acquiring the source data from source data environment equipment. The data selection script can be directly written by a relevant user, or can be written according to a provided script template in the process of configuring a data construction task or before sending a data construction event. By running the data selection script, the source data of the specified type can be acquired from the specified device. The source data acquired by the data selection script comprises: any one or more of CC type source data, BC type source data, key type source data, CMDB type source data, CKV type source data and DB type source data can be acquired, for example, DB source data stored in source data environment equipment can be acquired by running a data selection script, and all data stored in DB can be acquired. In one embodiment, in the process of running the data selection script in S402, the core service module is used to obtain the required source data from the source data environment device.

S403: and acquiring a data loading script associated with the data construction task, and loading target data into target data environment equipment by executing the data loading script, wherein the target data is obtained according to the source data. The data loading script can be directly written by a relevant user, or can be written according to a provided script template during the process of configuring the data construction task or before sending the data construction event. By running the data loading script, all data of the source data acquired in S402 can be loaded into the data environment of the target data environment device according to the environment requirement of the target data environment device, for example, all data of the DB source data acquired in S403 from the source data environment device shown in fig. 2 can be loaded into the DB database in the target data environment device in fig. 2, to constitute the DB source data in the target data environment device.

S404: and feeding back a notification message about the loading of the target data to the target data environment device to a management platform, wherein the notification message is used for instructing the management platform to respond to the data construction event and process the target data in the target data environment device. After loading is completed, a notification message may be fed back to the management platform of the data construction task, so that the core service module in the management platform (for example, in fig. 2) processes the target data, where the processing of the target data includes directly obtaining the target data contained in the data environment in the target data environment device, and may further send the obtained target data to other systems or platforms through an API (Application Programming Interface, application program interface) interface and other modes according to the actual situation.

Fig. 5a is a schematic flow chart of a data processing method according to an embodiment of the present application, which may be executed by the above-mentioned intelligent device. The method comprises the following steps.

S501: and acquiring a data construction task, wherein the data construction task is generated according to the detected data construction event. The data construction event may be detected after detecting that a user submitted task configuration information on a configuration interface; or the data construction event can also be detected after an external system or platform initiates a data acquisition request through an interface. In a different manner, different processing is performed after the construction of the data is completed. Wherein, the specific expression and application of the configuration interface refer to the description of the embodiments described below.

S502: analyzing the data construction task, acquiring a data selection script related to the data construction task, and acquiring source data by executing the data selection script, wherein the data selection script is used for acquiring the source data from source data environment equipment. In one embodiment, the identification of the source data environment device may also be configured on the configuration interface. The type of source data may be referred to the description in the foregoing embodiments.

S503: and acquiring a data extraction script associated with the data construction task, and extracting intermediate data meeting screening conditions from the obtained source data by executing the data extraction script. The filtering conditions are defined by the user in the data extraction script. The source data selected in S502 may be a table in which a large amount of types of data are recorded, and only a part of types of data are defined in the script.

S504: and acquiring a data conversion script associated with the data construction task, and converting the intermediate data into target data meeting conversion conditions by executing the data conversion script. The data conversion script mainly converts some or all of the intermediate data into data that is meaningful for a system or a platform that reads data from the target data environment device, for example, there are calculation result data and calculation original data in the intermediate data, the calculation result data being calculated based on one or more calculation original data, and if only the calculation result data and the calculation original data are directly loaded into the target data environment, it is possible for the platform or the system that extracts data from the target data environment that the calculation result data and the calculation original data are not meaningful and that the relation between them is unclear, and thus, script contents regarding a calculation process that acquires the calculation result data can be added in the data conversion script, so that the converted target data satisfying the conversion condition includes the calculation result data, the calculation original data, and the calculation rule for the calculation result data and the calculation original data.

S505: and acquiring a data loading script associated with the data construction task, and loading target data into target data environment equipment by executing the data loading script, wherein the target data is obtained according to the source data, namely the target data is obtained by carrying out the data extraction and the data conversion on the source data.

The step of acquiring the corresponding scripts mentioned in S502 to S504 may be reading all scripts of the data construction task from the cache device at one time so as to perform corresponding operation processing, and sequentially acquiring and executing all scripts read and stored in the local. Or sequentially acquiring corresponding scripts from the cache according to the instruction of the data construction task to perform operation processing, namely acquiring and executing the data selection script, acquiring and executing the data extraction script, acquiring and executing the data conversion script, and finally acquiring and executing the data loading script.

S506: and feeding back a notification message about the loading of the target data to the target data environment device to a management platform, wherein the notification message is used for instructing the management platform to respond to the data construction event and process the target data in the target data environment device.

It will be appreciated that the manner in which the data interactions referred to in this application may all be implemented by way of a computer network. Descriptive information of the content and the field format of each source data in the source data environment device can be displayed, so that a user writing the above-mentioned data selection script, data extraction script, data conversion script and data loading script can write corresponding scripts based on the descriptive information.

In one embodiment, the data construction event is generated in response to a detected user commit operation received from the configuration interface, the feeding back a notification message to a management platform that the target data is loaded to a target data environment device, comprising: generating a notification message, wherein the notification message comprises storage indication information of target data loaded into target data environment equipment, and the storage indication information comprises a database type and a storage address; and feeding back the notification message to a management platform, wherein the notification message is used for indicating the management platform to acquire target data from target data environment equipment according to the indication of the storage indication information. That is, if it is a task that the user initiates directly on the configuration interface, the smart device may notify the management platform to directly acquire target data in the target data environment for provision to other systems or platforms. The different databases are obtained in different ways, so that the database type is added in the storage indication information, so that the management platform can accurately initiate a data request to a storage address to obtain target data.

In one embodiment, the data construction event is generated in response to a data construction request received from an interface to which the management platform is connected, the feeding back a notification message to the management platform that the target data is loaded to the target data environment device, comprising: generating a notification message, wherein the notification message comprises storage indication information of target data loaded into target data environment equipment, and the storage indication information comprises a database type and a storage address; and feeding back the notification message to a management platform, wherein the notification message is used for instructing the management platform to forward the storage indication information to an initiating terminal initiating the data construction request so that the initiating terminal can acquire target data from target data environment equipment according to the indication of the storage indication information. That is, if an external system or platform requests data construction through an interface, the management platform may be triggered to forward a notification message to an initiator of the data construction request, so that the initiator obtains data from a corresponding storage address based on the storage indication information to a corresponding database. The different databases are obtained in different ways, so that the database type is added in the storage indication information, so that a corresponding platform or system can accurately initiate a data request to a storage address to obtain target data.

The data construction process divides the four layers so as to process the data construction process according to different stages and layers. When the data volume to be constructed is large, the time spent in the data construction process is also large, so in the embodiment of the application, the data can be partitioned so as to be convenient for concurrent processing. The concurrency processing mainly involves two stages, namely a stage of completing data selection by the data selection script and a stage of completing data conversion by the data conversion script. First, the extracting the intermediate data satisfying the filtering condition from the obtained source data by executing the data extraction script may include: partitioning the source data according to the data relation among the data items in the source data to obtain source partitioned data, and adding the source partitioned data into corresponding queues according to the data relation to obtain a plurality of source partitioned queues; respectively and synchronously extracting intermediate data meeting screening conditions from each source block data in each source block queue by executing the data extraction script to obtain a plurality of intermediate data block queues;

Next, the converting the intermediate data into the target data satisfying the conversion condition by executing the data conversion script may include: and performing data conversion script synchronization to convert the middle data blocks in each middle data block queue to obtain target data meeting conversion conditions respectively, and outputting each target data.

Fig. 5b is a schematic flow chart of partitioning source data to facilitate synchronous data extraction and conversion. For the acquired source data, data blocking is performed based on the relation between the source data, for example, for the acquired 10 pieces of table data, 3 pieces of table data are related to financial data, and can be divided into one source blocking data, 3 pieces of table data are related to sales data, and can be divided into one source blocking data, and 4 pieces of table data are related to cost data, and can be divided into one source blocking data, so that three source blocking data can be obtained, and extraction processing and conversion processing are respectively performed on the three source blocking data at the same time, thereby improving the processing efficiency.

According to the embodiment of the application, the data extraction script, the data conversion script and the data loading script are automatically operated, so that the conversion and construction of the data are fully automatically realized, the automatic requirement of a user on the construction of the data is met, and the requirement of personalized data construction is also met. In the whole process, a user only needs to write part of scripts and configure the scripts to the data construction task, so that the user time is saved, the automation requirement of data construction is met, the blocking processing logic is introduced, the processing concurrency is improved, and the data construction efficiency is further improved.

Referring to fig. 6 again, a flow chart of an interaction method of data processing according to an embodiment of the present application is shown, and the method of the embodiment of the present application includes the following steps.

S601: the management platform triggers data construction. The management platform may receive a data acquisition request initiated by an environment management system such as that in fig. 2, or may detect that a user initiates a confirmation operation on a configuration interface to generate a data acquisition request. After receiving the data acquisition request, the management platform determines that a data construction event is detected, and triggers the core service module to generate a data construction task according to the data acquisition request.

S602: the core service module creates a data construction task and stores relevant scripts corresponding to the data construction task into a memory. The memory may be a dedicated database or a cache, so that the intelligent device with the data construction agent installed can obtain the corresponding script when executing the data creation task.

In one embodiment, the script stored in S602 includes: a data selection script, a data extraction script, a data conversion script, and a data loading script. The memory can feed back the information of successful storage after the related information including the script in the data construction task is successfully stored.

In one embodiment, the core service module is part of the management platform, and the core service module may determine that a data construction event exists directly based on a data acquisition request initiated by the environment management system, i.e., may generate a data construction task according to the data construction event. Or the core service module can determine that the data construction event is detected directly after receiving the confirmation operation of the user on the configuration interface, and generate the data construction task based on the task configuration information received by the user on the configuration interface.

FIG. 7 is a schematic diagram of a configuration interface that may be provided to a user by a core service module, according to an embodiment of the present application. The configuration interface comprises the following configuration items:

the configuration item for configuring the source data environment device identity, namely source environment ID item 701 in FIG. 7, shows in FIG. 7a specific source environment ID number entered by the user: ENV1597908578T7279981;

The configuration items used to configure the target data environment device identification, namely target environment ID item 702 in FIG. 7, show the specific target data environment ID number entered by the user in FIG. 7: ENV1517208544T6275904;

The configuration item for configuring the data type of the data to be constructed, namely the construction data type item 703 in fig. 7, the user can select a CC data type, a BC data type, a key data type, etc., and can also customize "other" data types;

A configuration item of data indication information, such as configuration item 704 in fig. 7, including: the source type and/or the data association information of the data selected by the data selection script are/is needed; as shown in FIG. 7, the source type may refer to "mysql" type, i.e. selecting source data from a database, the source type may refer to "file" type and selecting source data from a file, and the source type may refer to "api" type, i.e. selecting source data from an api interface. The data association information is used for specifying the source type so that the data can be correctly selected when the data selection script is executed, for example, as shown in fig. 7, when the source type selected by the user is "mysql", the corresponding data association information includes: the data association information can also comprise a user name, a password and a library name, when the text is selected to acquire the source data based on the information operation data, the source data environment equipment can be obtained to verify the source data acquisition, and after the authorization is obtained, the intelligent equipment is allowed to acquire the source data corresponding to the corresponding database address.

A configuration item of data storage information, such as configuration item 705 in fig. 7, including: the storage type and/or data storage information of the data loaded by the data loading script is required. As shown in FIG. 7, a storage type may refer to a "mysql" type, i.e., storing source data into a database, a storage type may refer to a "file" type, i.e., storing source data into a file, and a storage type may refer to an "api" type, i.e., storing source data into a device or application associated with an api interface through the api interface. The data storage information is used for specifying a storage type so that data can be loaded correctly when the data loading script is executed, for example, as shown in fig. 7, when the storage type selected by the user is "file", the corresponding data storage information includes: the file path can correctly find the source data required by the corresponding file determination according to the information, the data storage information can also comprise a user name, a password and a file name, when the source data is loaded to a certain file by running a data loading text based on the information, the loading of the source data by the target data environment equipment can be obtained, and after the authorization is obtained, the intelligent equipment is allowed to record the obtained source data into the file corresponding to the file path.

After the user clicks a submit button on the configuration interface, a data construction request can be initiated, or the core service module is triggered to create a data construction task as a data construction event.

In one embodiment, for some data construction requirements, data in the selected source data may also be filtered, and only part of the data is extracted. Thus, the configuration interface may further include: an uploading configuration item of the data extraction script, wherein the uploading configuration item of the data extraction script comprises the following components:

A record box for recording script identification of the uploaded data extraction script; such as record block 706 in fig. 7. An upload-in button for acquiring a user-confirmed data extraction script in response to a user click operation and uploading the acquired data extraction script; such as the "upload script" button in fig. 7. A download input button for displaying a data extraction script template to a user in response to a user download operation, and acquiring a data extraction script generated from the data extraction script template and uploading the generated data extraction script, such as the "script template download" button in fig. 7.

In one embodiment, the data extracted by the data extraction script or the source data selected by the data selection script may be further converted to obtain target data meeting the data construction requirement, so the configuration interface may further include: an uploading configuration item of the data conversion script, wherein the uploading configuration item of the data conversion script comprises the following components:

A record box for recording script identification of the uploaded data conversion script; such as record box 707 in fig. 7. An upload-in button for acquiring a user-confirmed data conversion script in response to a user click operation and uploading the acquired data conversion script; such as the "upload script" button in fig. 7. A download input button for displaying a data conversion script template to a user in response to a user download operation, and acquiring a data conversion script generated from the data conversion script template and uploading the generated data conversion script, such as the "script template download" button in fig. 7.

S603: the core service module receives the feedback message of successful storage.

S604: and the core service module issues the data construction task to the intelligent device. For the intelligent device, a data construction task is received, and the data processing of the embodiment of the application starts to be executed.

S605: the intelligent device parses the data construction task. The intelligent device acquires a data construction task, and reads task configuration information from the data construction task so as to conveniently execute selection of source data, data extraction, data conversion and data recording.

S606: the intelligent device obtains the script. And finding a corresponding script file in a memory according to the identification of the data construction task. In S606, it includes: acquiring a data selection script associated with the data construction task, acquiring a data loading script associated with the data construction task, acquiring a data extraction script associated with the data construction task, and acquiring a data conversion script associated with the data construction task.

S607: the memory returns the required script.

S608: the intelligent device creates a Select process;

S609: and the intelligent equipment obtains source data by executing the data selection script.

S610: the intelligent device receives source data returned by the source data environment device;

s611: the intelligent device creates an Extract process;

S612: and the intelligent equipment extracts the intermediate data meeting the screening conditions from the obtained source data by executing the data extraction script. After the Extract process is created, the source data can be partitioned, and subsequent data processing can be performed on the created concurrent tasks.

S613: the intelligent device creates Tranfrom processes;

s614: and the intelligent equipment converts the intermediate data into target data meeting the conversion conditions by executing the data conversion script. In the process of executing the Extract script, if the source data is segmented in S612, the plurality of segments extracted by the data extraction script are correspondingly transformed in S614, so as to obtain a plurality of target data.

S615: the intelligent device creates a load process;

s616: the intelligent device loads target data into the target data environment device by executing the data loading script;

S617: and the intelligent device receives a load result returned by the target data environment device.

S618: the intelligent device feeds back a notification message to the core service module about the loading of the target data to the target data environment device, wherein the notification message is used for instructing the management platform to respond to the data construction event and process the target data in the target data environment device.

S619: the core service module sends a construction result update notification to the management platform so as to inform the management platform that the corresponding data is constructed successfully.

In the embodiment of the application, a simple and visual configuration interface can be provided for a user, so that the user can conveniently configure information required by data construction, and can also conveniently upload scripts or quickly configure and upload scripts through templates, and the data extraction scripts, the data conversion scripts and the data loading scripts are automatically operated through data selection scripts configured by the user, thereby fully automatically realizing data conversion and construction, meeting the automatic requirement of the user on data construction, and meeting the requirement of personalized data construction. And the blocking processing logic is introduced, so that the processing concurrency is improved, and the data construction efficiency is further improved.

Referring to fig. 8 again, a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention may be provided in an intelligent device such as a server, where the intelligent device may be the intelligent device mentioned in the foregoing embodiment. The device comprises the following modules.

An obtaining module 801, configured to obtain a data construction task, where the data construction task is generated according to a detected data construction event;

The processing module 802 is configured to parse the data construction task, obtain a data selection script associated with the data construction task, and obtain source data by executing the data selection script, where the data selection script is configured to obtain the source data from a source data environment device; acquiring a data loading script associated with the data construction task, and loading target data into target data environment equipment by executing the data loading script, wherein the target data is obtained according to the source data;

A notification module 803, configured to feed back, to a management platform, a notification message about the target data being loaded to the target data environment device, where the notification message is configured to instruct the management platform to process the target data in the target data environment device in response to the data construction event.

In one embodiment, the processing module 802 is further configured to obtain a data extraction script associated with the data construction task after the data selection script is executed to obtain source data and before the data loading script is executed to load target data into a target data environment device, and extract, by executing the data extraction script, intermediate data that satisfies a filtering condition from the obtained source data; and acquiring a data conversion script associated with the data construction task, and converting the intermediate data into target data meeting conversion conditions by executing the data conversion script.

In one embodiment, the processing module 802 is configured to block the source data according to a data relationship between data items in the source data to obtain source block data, and add the source block data to a corresponding queue according to the data relationship to obtain a plurality of source block queues; respectively and synchronously extracting intermediate data meeting screening conditions from each source block data in each source block queue by executing the data extraction script to obtain a plurality of intermediate data block queues; and performing data conversion script synchronization to convert the middle data blocks in each middle data block queue to obtain target data meeting conversion conditions respectively, and outputting each target data.

In one embodiment, the data construction task includes task configuration information, where the task configuration information is set on a configuration interface; the configuration interface includes any one or more of the following:

A configuration item for configuring a source data environment device identification;

A configuration item for configuring a target data environment device identification;

A configuration item for configuring a data type of data to be constructed;

a configuration item of data indication information, the data indication information comprising: the source type and/or the data association information of the data selected by the data selection script are/is needed;

A configuration item of data storage information, the data storage information comprising: the storage type and/or data storage information of the data loaded by the data loading script is required.

In one embodiment, the configuration interface further includes an upload configuration item of the data extraction script, where the upload configuration item of the data extraction script includes:

a record box for recording script identification of the uploaded data extraction script;

An upload-in button for acquiring a user-confirmed data extraction script in response to a user click operation and uploading the acquired data extraction script;

And a download input button for displaying a data extraction script template to a user in response to a user download operation, and acquiring a data extraction script generated according to the data extraction script template and uploading the generated data extraction script.

In one embodiment, the configuration interface further includes an upload configuration item of the data conversion script, where the upload configuration item of the data conversion script includes:

a record box for recording script identification of the uploaded data conversion script;

an upload-in button for acquiring a user-confirmed data conversion script in response to a user click operation and uploading the acquired data conversion script;

and a download input button for displaying the data conversion script template to the user in response to a user download operation, and acquiring a data conversion script generated according to the data conversion script template and uploading the generated data conversion script.

In one embodiment, the notification module 803 is configured to generate a notification message, where the notification message includes storage indication information of the target data loaded into the target data environment device, where the storage indication information includes a database type and a storage address; and feeding back the notification message to a management platform, wherein the notification message is used for indicating the management platform to acquire target data from target data environment equipment according to the indication of the storage indication information.

In one embodiment, the notification module 803 is configured to generate a notification message, where the notification message includes storage indication information of the target data loaded into the target data environment device, where the storage indication information includes a database type and a storage address; and feeding back the notification message to a management platform, wherein the notification message is used for instructing the management platform to forward the storage indication information to an initiating terminal initiating the data construction request so that the initiating terminal can acquire target data from target data environment equipment according to the indication of the storage indication information.

Referring to fig. 9 again, a schematic structural diagram of an intelligent device according to an embodiment of the present invention corresponds to the intelligent device mentioned in the foregoing embodiment, and may be a server. The smart device according to the embodiment of the present invention may include a storage 901 and a processor 902, and may of course further include interfaces for interaction data, such as a network interface 903 and a communication interface 904. An interface module such as a power module, a USB data interface, etc. may also be included.

The network interface 903 may be various interfaces for accessing to a computer network, and the communication interface 904 may be a mobile communication interface 904, which can access to a mobile communication network such as 4G/5G.

The storage 901 may include volatile memory (RAM), such as random-access memory (RAM); the storage device 901 may also include a nonvolatile memory (non-volatile memory), such as a flash memory (flash memory), a solid state disk (solid-state disk) (STATE DRIVE, SSD), etc.; the storage 901 may also include a combination of the above types of memories.

The processor 902 may be a central processing unit 902 (central processing unit, CPU). The processor 902 may further comprise a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (programmable logic device, PLD), or the like. The PLD may be a field-programmable gate array (FPGA) of field-programmable GATE ARRAY, general-purpose array logic (GENERIC ARRAY logic, GAL), or the like.

Optionally, the storage 901 is further configured to store a computer program. The processor 902 may invoke the computer program to implement the relevant methods as mentioned in the previous embodiments of the application.

In one embodiment, the processor 902 invokes a computer program stored in the storage 901 for obtaining a data construction task, wherein the data construction task is generated according to a detected data construction event; analyzing the data construction task, acquiring a data selection script related to the data construction task, and acquiring source data by executing the data selection script, wherein the data selection script is used for acquiring the source data from source data environment equipment; acquiring a data loading script associated with the data construction task, and loading target data into target data environment equipment by executing the data loading script, wherein the target data is obtained according to the source data; and feeding back a notification message about the loading of the target data to the target data environment device to a management platform, wherein the notification message is used for instructing the management platform to respond to the data construction event and process the target data in the target data environment device.

In one embodiment, the processor 902 is further configured to obtain a data extraction script associated with the data construction task after executing the data selection script to obtain source data and before executing the data loading script to load target data into a target data environment device, and extract, by executing the data extraction script, intermediate data that satisfies a filtering condition from the obtained source data; and acquiring a data conversion script associated with the data construction task, and converting the intermediate data into target data meeting conversion conditions by executing the data conversion script.

In one embodiment, the processor 902 is configured to block the source data according to a data relationship between data items in the source data to obtain source block data, and add the source block data to a corresponding queue according to the data relationship to obtain a plurality of source block queues; respectively and synchronously extracting intermediate data meeting screening conditions from each source block data in each source block queue by executing the data extraction script to obtain a plurality of intermediate data block queues; and performing data conversion script synchronization to convert the middle data blocks in each middle data block queue to obtain target data meeting conversion conditions respectively, and outputting each target data.

A configuration item for configuring a data type of data to be constructed;

In one embodiment, the data construction event is generated in response to a detected user submit operation received from the configuration interface, the processor 902 is configured to generate a notification message, where the notification message includes storage indication information of target data loaded into a target data environment device, where the storage indication information includes a database type and a storage address; and feeding back the notification message to a management platform, wherein the notification message is used for indicating the management platform to acquire target data from target data environment equipment according to the indication of the storage indication information.

In one embodiment, the data construction event is generated in response to a data construction request received from an interface of the management platform connection, and the processor 902 is configured to generate a notification message, where the notification message includes storage indication information of the target data loaded into the target data environment device, where the storage indication information includes a database type and a storage address; and feeding back the notification message to a management platform, wherein the notification message is used for instructing the management platform to forward the storage indication information to an initiating terminal initiating the data construction request so that the initiating terminal can acquire target data from target data environment equipment according to the indication of the storage indication information.

Those skilled in the art will appreciate that implementing all or part of the above-described methods in accordance with the embodiments may be accomplished by way of a computer program stored on a computer readable storage medium, which when executed may comprise the steps of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random-access Memory (Random Access Memory, RAM), or the like.

The above disclosure is only a few examples of the present invention, and it is not intended to limit the scope of the present invention, but it is understood by those skilled in the art that all or a part of the above embodiments may be implemented and equivalents thereof may be modified according to the scope of the present invention.

Claims

1. A method of data processing, comprising:

Storing a pre-written data selection script for acquiring the source data of the specified type from the specified source data environment equipment, and storing a pre-written data loading script for loading the source data acquired by the data selection script into the data environment of the target data environment equipment according to the environment requirement of the target data environment equipment;

Generating a data construction event in response to a user submit operation received from the configuration interface; the configuration interface comprises configuration items of data indication information and configuration items of data storage information, wherein the data indication information comprises: the source type and/or the data association information of the data selected by the data selection script are/is needed; the data storage information includes: the storage type and/or data storage information of the data to be loaded by the data loading script;

acquiring a data construction task generated according to the data construction event; the data construction task comprises: selecting a script, a data loading script, a device identifier of source data environment device and an identifier of target data environment device according to the data set by the data construction event;

2. The method of claim 1, wherein after executing the data selection script to obtain source data, and before executing the data loading script to load target data into a target data environment device, the method further comprises:

Acquiring a data extraction script associated with the data construction task, and extracting intermediate data meeting screening conditions from the obtained source data by executing the data extraction script;

and acquiring a data conversion script associated with the data construction task, and converting the intermediate data into target data meeting conversion conditions by executing the data conversion script.

3. The method of claim 2, wherein,

The extracting the intermediate data meeting the screening condition from the obtained source data by executing the data extraction script comprises the following steps:

Partitioning the source data according to the data relation among the data items in the source data to obtain source partitioned data, and adding the source partitioned data into corresponding queues according to the data relation to obtain a plurality of source partitioned queues;

Respectively and synchronously extracting intermediate data meeting screening conditions from each source block data in each source block queue by executing the data extraction script to obtain a plurality of intermediate data block queues;

the converting the intermediate data into target data satisfying a conversion condition by executing the data conversion script includes:

And performing data conversion script synchronization to convert the middle data blocks in each middle data block queue to obtain target data meeting conversion conditions respectively, and outputting each target data.

4. A method according to any one of claims 1-3, wherein the data construction task comprises task configuration information, the task configuration information being provided on a configuration interface; the configuration interface includes any one or more of the following:

a configuration item for configuring a data type of the data to be constructed.

5. The method of claim 4, wherein the configuration interface further comprises an upload configuration item of a data extraction script, and the upload configuration item of the data extraction script comprises:

6. The method of claim 4, wherein the configuration interface further comprises an upload configuration item of a data conversion script, and the upload configuration item of the data conversion script comprises:

7. The method of claim 4, wherein the data build event is generated in response to a detected user submit operation received from the configuration interface, the feeding back a notification message to a management platform that the target data is loaded to a target data environment device, comprising:

Generating a notification message, wherein the notification message comprises storage indication information of target data loaded into target data environment equipment, and the storage indication information comprises a database type and a storage address;

and feeding back the notification message to a management platform, wherein the notification message is used for indicating the management platform to acquire target data from target data environment equipment according to the indication of the storage indication information.

8. The method of claim 4, wherein the data construction event is generated in response to a data construction request received from an interface to which the management platform is connected, the feeding back a notification message to the management platform that the target data is loaded to a target data environment device, comprising:

And feeding back the notification message to a management platform, wherein the notification message is used for instructing the management platform to forward the storage indication information to an initiating terminal initiating the data construction request so that the initiating terminal can acquire target data from target data environment equipment according to the indication of the storage indication information.

9. A data processing apparatus, comprising:

The module is used for storing a pre-written data selection script used for acquiring the source data of the specified type from the specified source data environment equipment, and storing a pre-written data loading script used for loading the source data acquired by the data selection script into the data environment of the target data environment equipment according to the environment requirement of the target data environment equipment;

Means for generating a data construction event in response to a user submit operation received from the configuration interface; the configuration interface comprises configuration items of data indication information and configuration items of data storage information, wherein the data indication information comprises: the source type and/or the data association information of the data selected by the data selection script are/is needed; the data storage information includes: the storage type and/or data storage information of the data to be loaded by the data loading script;

The acquisition module is used for acquiring data construction tasks generated according to the data construction events, and the data construction tasks comprise: selecting a script, a data loading script, a device identifier of source data environment device and an identifier of target data environment device according to the data set by the data construction event;

10. An intelligent device comprising a storage means in which a computer program is stored and a processor which invokes the computer program to implement a data processing method as claimed in any one of claims 1 to 8.

11. A computer storage medium, in which a computer program is stored which, when being executed by a processor, implements a data processing method according to any of claims 1-8.

12. A computer program product comprising computer programs/instructions which, when executed by a processor, implement a data processing method according to any of claims 1-8.