CN112130933A - Method and device for constructing and calling operator set - Google Patents
Method and device for constructing and calling operator set Download PDFInfo
- Publication number
- CN112130933A CN112130933A CN202010773698.6A CN202010773698A CN112130933A CN 112130933 A CN112130933 A CN 112130933A CN 202010773698 A CN202010773698 A CN 202010773698A CN 112130933 A CN112130933 A CN 112130933A
- Authority
- CN
- China
- Prior art keywords
- operator
- module
- parameters
- operator module
- parameter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
- G06F9/4482—Procedural
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The invention provides a method and a device for constructing an operator set, wherein an operator unit receives the operator set, and the operator unit is a processing unit and is used for completing a corresponding processing task; acquiring operator interface parameters, namely acquiring the operator interface parameters according to the operator unit, wherein the operator interface parameters comprise starting parameters and operating parameters; and (5) constructing an operator module, namely constructing the operator unit with the acquired operator interface parameter into the operator module. The interface parameters of the operator unit are unified and comprise a starting parameter and an operating parameter, the specification of the operator interface parameters is unified, and the connection difficulty of multiple languages and algorithms of developers is reduced; for business personnel, the business personnel can directly call the operator module, and the work difficulty is reduced. The invention also provides an operator set calling method and device, and business personnel can select an applicable operator module according to the operation environment and then send the selected operator module to the operation node for operation, so that the working efficiency of the business personnel is improved.
Description
Technical Field
The invention relates to the technical field of big data analysis algorithm library construction, in particular to a method and a device for constructing and calling an operator set.
Background
Currently, a new round of scientific and technological revolution and industrial revolution are germinating, the formation of big data, the innovation of theoretical algorithm, the promotion of computing power and the development of network facility evolution-driven big data analysis technology enter a new stage, and the intellectualization becomes an important direction for the development of technology and industry. In the process of constructing enterprise-level big data analysis application, due to the complexity of the scale and the form of business data, different computing frames, computing resources and computing languages are needed to be relied on to adapt to different business scenes in the processes of developing and operating data processing, feature engineering and machine learning algorithms: when the service data reaches hundred million-level scale and is preprocessed, a distributed computing framework is required to be selected to realize parallel distribution computation of the data, so that the data processing efficiency is improved.
Although developers in the market can select different computing frames, computing resources and computing languages to realize analysis models of different service scenes, in the actual analysis modeling process, the requirements of the operators and the developers for algorithm development and use are different, the operators often have shallow knowledge about the interior of the operators, the operators are difficult to carry out complex debugging on the interior of the operators, much time is spent in the using process of the operators, and the working efficiency is influenced; the operator set of the developer has no uniform specification and definition, and the connection of multiple languages and algorithms is difficult, so that much time is wasted for the developer.
Therefore, there is a need in the art for a method and apparatus for operator set construction and invocation.
Accordingly, the present invention is directed to such a system.
Disclosure of Invention
The invention aims to provide a method and a device for constructing and calling an operator set, so as to solve at least one technical problem.
The invention provides an operator set construction method, which comprises the following steps:
receiving by an operator unit, wherein the operator unit is a processing unit and is used for completing a corresponding processing task;
acquiring operator interface parameters, namely acquiring the operator interface parameters according to the operator unit, wherein the operator interface parameters comprise starting parameters and operating parameters;
and (5) constructing an operator module, namely constructing the operator unit with the acquired operator interface parameter into the operator module.
By adopting the scheme, the interface parameters of the operator units are unified and comprise the starting parameters and the operating parameters, and the specification of the interface parameters of the operators is unified, so that the operator set has unified specification and definition, and the connection difficulty of multiple languages and algorithms of developers is reduced; for business personnel, the specification of the operator interface parameters is unified, so that the business personnel can directly call the operator module without complex debugging, and the working difficulty is reduced.
Further, the operator set construction method further comprises the step of operator set construction, wherein a plurality of operator modules are placed in the first container, and the operator set is constructed.
By adopting the scheme, the operator set can comprise a plurality of different operator modules, and business personnel can directly extract the needed operator modules in the operator set, so that the time spent on searching the needed operators is reduced, and the working efficiency is improved.
Furthermore, the processing unit may be a function, or may be a combination of multiple functions, including a Group function, a Sort function, and the like, and may complete the conversion of data.
Further, the starting parameter is a command line header parameter, and is used for starting the operation of the operator unit.
By adopting the scheme, the operator unit can be started to operate conveniently.
Furthermore, the operation parameters include an entry parameter and an exit parameter, the entry parameter is an input path of the operator module, the exit parameter is an output path of the operator module, and the operator modules can be matched with each other through the matching of the entry parameter and the exit parameter.
By adopting the scheme, business personnel can directly call the operator modules, the operator modules are arranged according to the business sequence, the operators can be matched with each other, the operation parameters of the operators do not need to be adjusted in the calling process, and the convenience of calling the operators by the business personnel is improved.
Further, the operator unit further includes internal parameters, and the internal parameters are used for the operator unit to complete corresponding processing tasks.
Further, the internal parameters include operator operating parameters, which may be iteration times, learning rate, and regular coefficient.
Preferably, the operator module comprises a first channel, and one end of the first channel is connected with the internal parameter.
By adopting the scheme, higher flexibility is provided for business personnel, and the business personnel adjust internal parameters when using the operator module.
Further, the operator module construction further comprises construction of an operator label, and the operator label comprises an operator name.
Preferably, the operator name is derived from a processing function of the operator.
By adopting the scheme, business personnel can know the operator function through the operator label, so that the business personnel can quickly find the needed operator module.
Further, the operator set construction further comprises an operator module directory construction, and the operator module directory construction step comprises:
receiving the operator module;
extracting an operator label in the operator module;
and constructing the operator module directory according to the operator label.
By adopting the scheme, business personnel can directly search in the operator module directory, and the operator module can be conveniently extracted by the business personnel.
Preferably, a new operator module can be added after the construction of the operator set is completed.
By adopting the scheme, the operator set supports expansion, the operator module types in the operator set are increased, and the operator diversity in the operator set is improved.
Preferably, the operator tag further comprises an operator version number.
Preferably, the operator set includes a plurality of operator modules with the same operator name, and the operator modules with the same operator name have different operator version numbers.
By adopting the scheme, multiple choices are provided for the same operator, and the operator selectivity is improved.
The invention provides an operator set calling method, which comprises the following steps:
constructing an operating environment, wherein the operating environment comprises at least one operating node;
receiving the operator set and outputting an operator module selection instruction;
receiving a selected operator module in the operator set;
and sending the operator module to an operation node for operation.
By adopting the scheme, business personnel can select the applicable operator module according to the operation environment and then send the selected operator module to the operation node for operation, so that the work efficiency of the business personnel is improved.
Further, the building of the operating environment may be building of an operating environment in a distributed cluster, the distributed cluster may be a Hadoop distributed cluster, and the operating environment may be a Spark environment; the running node can be a computer or a server and the like.
Further, the internal parameters include operator operating parameters, and the step of receiving a selected operator module in the operator set further includes:
outputting the operator operation parameters to an operator module surface layer through a first channel;
judging whether the operator operation parameters are changed or not;
if not, keeping the operation parameters of the original operator unchanged;
and if so, receiving the changed operator operating parameters, and updating the original operator operating parameters into the changed operator operating parameters.
By adopting the scheme, when the operator module is used by business personnel, the internal parameters can be adjusted, so that the operator module can operate according to the parameters required by the operator module, and the operation efficiency is improved.
Further, the selected operator module may be a plurality of modules, and receiving the selected operator module in the operator set further includes modeling the operator module.
Further, the step of modeling the operator module comprises:
carrying out hierarchical sequencing on a plurality of operator modules;
in adjacent operator modules with upper and lower layer relations, the outlet parameters of the upper operator module are matched with the inlet parameters of the lower operator module, so that the output data of the upper operator module is received by the lower operator module.
By adopting the scheme, the output path of the upper operator is matched with the input path of the lower operator, so that the output data of the upper operator module is received by the lower operator module, the operator module has the specification of unifying the interface parameters of the operators, the operator set has the unified specification and definition, service personnel do not need to spend time on the interface parameters during modeling, and the working efficiency is improved.
Further, the step of sending the operator module to the operation node for operation further includes:
judging whether the number of nodes in the operating environment is greater than 1;
if yes, the sending operator module operates at the operation node;
if not, acquiring working parameters of the nodes;
and sending the operator module to a node with higher working parameters for operation.
Further, the operating parameter may be a remaining operating memory, a remaining hard disk memory, or a remaining graphics card memory.
Preferably, if the node working parameter changes during the transmission of the operator module, which results in that the working parameter of the node receiving the operator module is lower than that of other nodes, the receiving node is replaced with another node with the highest working parameter compared with other nodes.
By adopting the scheme, the operation environment of the operator module is ensured, and the operation speed is increased.
The invention also protects an operator set construction device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the method.
The invention also protects an operator set using device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the method.
In conclusion, the invention has the following beneficial effects:
1. according to the operator set construction method, the interface parameters of the operator units are unified, the opening parameters and the operation parameters are included, and the specification of the operator interface parameters is unified, so that the operator set has unified specification and definition, and the connection difficulty of multiple languages and algorithms of developers is reduced; for business personnel, because the specification of the operator interface parameters is unified, the business personnel can directly call the operator module without complex debugging, and the work difficulty is reduced;
2. according to the operator set construction method, the operator set can comprise various different operator modules, business personnel can directly extract the needed operator modules from the operator set, time spent on searching the needed operators is reduced, and working efficiency is improved;
3. according to the operator set construction method, business personnel can directly search in the operator module catalog, and the operator module is convenient to extract by the business personnel;
4. according to the operator set calling method, the service personnel can select the applicable operator module according to the operation environment, and then send the selected operator module to the operation node for operation, so that the working efficiency of the service personnel is improved;
5. according to the operator set calling method, the output path of the upper-layer operator is matched with the input path of the lower-layer operator, so that the output data of the upper-layer operator module is received by the lower-layer operator module, the operator module has the specification of unifying the interface parameters of the operators, the operator set has the unified specification and definition, service personnel do not need to spend time on the interface parameters during modeling, and the working efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of one embodiment of an algorithm set construction method of the present invention;
FIG. 2 is a flow chart of an embodiment of constructing an operator module directory in the operator set construction method of the present invention;
FIG. 3 is a flow chart of one embodiment of a method for calling an operator set in accordance with the present invention;
FIG. 4 is a flow chart of one embodiment of a method for receiving a selected operator module in an operator set calling method of the present invention;
FIG. 5 is a flow chart of another embodiment of the operator module selected from the received operator set in the operator set calling method according to the present invention.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
As shown in fig. 1, the present invention provides an operator set construction method, which includes the following steps:
s100, receiving by an operator unit, wherein the operator unit is a processing unit and is used for completing a corresponding processing task;
s200, acquiring operator interface parameters, wherein the operator interface parameters are acquired according to the operator unit and comprise starting parameters and operating parameters;
s300, constructing an operator module, namely constructing an operator unit with acquired operator interface parameters into the operator module.
By adopting the scheme, the interface parameters of the operator units are unified and comprise the starting parameters and the operating parameters, and the specification of the interface parameters of the operators is unified, so that the operator set has unified specification and definition, and the connection difficulty of multiple languages and algorithms of developers is reduced; for business personnel, the specification of the operator interface parameters is unified, so that the business personnel can directly call the operator module without complex debugging, and the working difficulty is reduced.
In a specific implementation process, the operator set construction method further includes S400, operator set construction, where a plurality of operator modules are placed in the first container to construct the operator set.
By adopting the scheme, the operator set can comprise a plurality of different operator modules, and business personnel can directly extract the needed operator modules in the operator set, so that the time spent on searching the needed operators is reduced, and the working efficiency is improved.
In a specific implementation process, the first container may be a normal file, a compressed file, or a device with storage capability such as a hard disk and a magnetic disk.
In a specific implementation process, the processing unit may be one function, or may be a combination of multiple functions, including a Group function, a Sort function, and the like, and may complete data conversion.
The Group function is used for distinguishing grouped common lines and aggregated lines; the Sort function can be used in languages such as Java, Python, Scale, R and the like, and sorts all elements in a given interval, wherein the default is ascending Sort or descending Sort.
In a specific implementation process, the starting parameter is a command line header parameter, and is used for starting the operation of the operator unit.
In the specific implementation process, in the Spark framework program, the head of the command line is Spark-submit-class. For the python program, the command line header is python.
By adopting the scheme, the operator unit can be started to operate conveniently.
In a specific implementation, the Spark framework is a fast and general-purpose computing engine designed specifically for large-scale data processing.
In a specific implementation process, the operation parameters include an entry parameter and an exit parameter, the entry parameter is an input path of the operator module, the exit parameter is an output path of the operator module, and the operator modules can be matched with each other through the entry parameter and the exit parameter.
By adopting the scheme, business personnel can directly call the operator modules, the operator modules are arranged according to the business sequence, the operators can be matched with each other, the operation parameters of the operators do not need to be adjusted in the calling process, and the convenience of calling the operators by the business personnel is improved.
In a specific implementation process, the operator unit further includes an internal parameter, and the internal parameter is used for the operator unit to complete a corresponding processing task.
In a specific implementation process, the internal parameters include operator operating parameters, and the operator operating parameters may be parameters such as iteration number, learning rate, and regular coefficient.
In a preferred embodiment of the present invention, the operator module includes a first channel, one end of the first channel is connected to the internal parameter, and the other end of the first channel is connected to the operator module surface layer.
In a specific implementation process, the first channel provides a channel for changing the internal parameters, so that the internal parameters of the code layer are represented in the operator module table layer.
By adopting the scheme, higher flexibility is provided for business personnel, and the business personnel do not need to enter a code layer to adjust, and only need to adjust internal parameters on an operator module table layer.
In a specific implementation process, the operator module construction further comprises construction of an operator label, and the operator label comprises an operator name.
In an optional embodiment of the present invention, the operator name is obtained according to a processing function of an operator, and the operator name may be a stop word, a participle, a word filter, a feature index, or the like.
By adopting the scheme, business personnel can know the operator function through the operator label, so that the business personnel can quickly find the needed operator module.
In a specific implementation process, the operator tag further comprises an operator category, a calculation mode, a special category, a calculation framework and calculation resources; the operator categories include, but are not limited to, operators of categories such as data loading, data conversion, feature engineering, statistical analysis, machine learning, and the like, and the operator modes include a single mode and a distributed mode; the special category includes but is not limited to category operators such as visualization operators, programming operators, ETL operators and the like; the computing frameworks comprise Tensorflow, PyTorch and other computing frameworks; the computing resources comprise CPU resources or GPU resources, and for the visual operators, the output of the visual operators is appointed to a specific visual database to realize visual analysis of data; the programming operator provides four types of online programming operators of Python, Shell, R and SparkSql, and can directly realize flexible access of the operator by online compiling script language codes; the ETL type operator is a data source access operator, and can realize configuration loading and data reading operation of various relational and non-relational data sources.
In a specific implementation process, the operator comprises a common operator and a deep learning operator, and an operator label of the common operator comprises an operator name, an operator category, a calculation mode and a special category; the deep learning operator comprises an operator name, an operator category, a computing frame, a computing mode and computing resources.
As shown in fig. 2, in a specific implementation process, the operator set building further includes building an operator module directory, and the step of building the operator module directory includes:
s401, receiving the operator module;
s402, extracting an operator label in the operator module;
s403, constructing the operator module directory according to the operator labels.
By adopting the scheme, business personnel can directly search in the operator module directory, and the operator module can be conveniently extracted by the business personnel.
In a preferred embodiment of the present invention, the operator module directory includes a multi-layer directory, where the multi-layer directory may be a primary directory and a secondary directory, and the primary directory may be data, load, convert, and the like; the secondary directory is a subdirectory of the primary directory, and the converted subdirectory can be used for text processing, data conversion, data sampling, data cleaning and the like; the secondary directory further comprises sub-units, and the sub-units can be HDFS, Mongo or FTPS and the like.
The HDFS is a highly fault tolerant system suitable for deployment on inexpensive machines. The HDFS can provide data access with high throughput, and is very suitable for application on a large-scale data set; the Mongo is a product between a relational database and a non-relational database, and the non-relational database has the richest functions and is most similar to the relational database. The data structure supported by the method is very loose and is in a json-like bson format, so that more complex data types can be stored.
In a specific implementation process, a new operator module can be added after the operator set is constructed.
By adopting the scheme, the operator set supports expansion, the operator module types in the operator set are increased, and the operator diversity in the operator set is improved.
In a specific implementation process, the addition of the operator module can compress the operator module into a compressed file and send the compressed file to the operator set.
In a specific implementation process, the operator tag further includes an operator version number.
In a specific implementation process, the operator set comprises a plurality of operator modules with the same operator name, and the operator modules with the same operator name have different operator version numbers.
In the specific implementation process, operator module operator interface parameters with the same operator name and different operator version numbers are different.
By adopting the scheme, multiple choices are provided for the same operator, and the operator selectivity is improved.
As shown in fig. 3, the present invention provides an operator set calling method, which includes the following steps:
s500, constructing an operation environment, wherein the operation environment comprises at least one operation node;
s600, receiving the operator set and outputting an operator module selection instruction;
s700, receiving the operator module selected in the operator set;
and S800, sending the operator module to an operation node for operation.
By adopting the scheme, business personnel can select the applicable operator module according to the operation environment and then send the selected operator module to the operation node for operation, so that the work efficiency of the business personnel is improved.
In a specific implementation process, the building of the operating environment may be building of an operating environment in a distributed cluster, the distributed cluster may be a Hadoop distributed cluster, and the operating environment may be a Spark environment; the running node can be a computer or a server and the like.
The Hadoop is a distributed system infrastructure developed by the Apache Foundation. A user can develop a distributed program without knowing the distributed underlying details. The power of the cluster is fully utilized to carry out high-speed operation and storage. Hadoop realizes a Distributed File System (Hadoop Distributed File System), which is called HDFS for short.
As shown in fig. 4, in a specific implementation process, the step of receiving a selected operator module in an operator set further includes:
outputting the operator operation parameters to an operator module surface layer through a first channel;
judging whether the operator operation parameters are changed or not;
if not, keeping the operation parameters of the original operator unchanged;
and if so, receiving the changed operator operating parameters, and updating the original operator operating parameters into the changed operator operating parameters.
By adopting the scheme, when the operator module is used by business personnel, the operator operation parameters can be adjusted, so that the operator module can operate according to the operator operation parameters required by the operator module, and the operation efficiency is improved.
As shown in fig. 5, in a specific implementation process, the step of outputting the operator operating parameter to the operator module surface layer through the first channel further includes:
judging whether the operator operation parameters exist or not;
if yes, judging whether to change the operator operation parameters;
and if the operator operation parameter does not exist, outputting an operator operation parameter filling request, and receiving the filled operator operation parameter.
By adopting the scheme, for some operators, the internal parameters need to be configured according to actual use conditions, and operation errors caused by lack of the internal parameters are avoided.
In a specific implementation, the selected operator module may be a plurality of operator modules, and the receiving the selected operator module in the operator set further includes modeling the operator module.
In a preferred embodiment of the present invention, the operator module can be used by dragging to complete modeling of the operator module, and finally submitted for execution.
In a specific implementation process, the step of modeling the operator module comprises:
carrying out hierarchical sequencing on a plurality of operator modules;
in adjacent operator modules with upper and lower layer relations, the outlet parameters of the upper operator module are matched with the inlet parameters of the lower operator module, so that the output data of the upper operator module is received by the lower operator module.
By adopting the scheme, the output path of the upper operator is matched with the input path of the lower operator, so that the output data of the upper operator module is received by the lower operator module, the operator module has the specification of unifying the interface parameters of the operators, the operator set has the unified specification and definition, service personnel do not need to spend time on the interface parameters during modeling, and the working efficiency is improved.
In a specific implementation process, the step of sending the operator module to the operation node for operation further includes:
judging whether the number of nodes in the operating environment is greater than 1;
if yes, the sending operator module operates at the operation node;
if not, acquiring working parameters of the nodes;
and sending the operator module to a node with higher working parameters for operation.
In a specific implementation process, the operator module can be operated on one node or a plurality of nodes.
In a specific implementation process, the working parameter may be a remaining operating memory, a remaining hard disk memory, or a remaining graphics card memory.
In a specific implementation process, the step of sending the operator module to the operation node for operation further includes selecting a corresponding node according to the internal parameter.
In a specific implementation process, sending the operator module to the operation node for operation further includes:
and judging whether the operator module is a deep learning operator or not, if so, using the calculation resource of the operator module as a GPU resource.
In a specific implementation process, the internal parameters further include operator execution parameters, and the operator execution parameters include a running memory number, an internal core number, a display card number, and the like.
In a specific implementation process, the number of the operating memories can be 4G, 5G or 6G; the number of the internal cores can be 4 cores, 6 cores or 8 cores, and when the number of the operating memories in the internal parameters is 4G and the number of the internal cores is 6 cores, the number of the selected node residual operating memories is larger than 4G, and the number of the internal cores is 6 cores.
In a preferred embodiment of the present invention, if the node operating parameter changes during the transmission of the operator module, which results in that the operating parameter of the node receiving the operator module is lower than that of other nodes, the receiving node is replaced with another node whose operating parameter is the highest than that of other nodes.
By adopting the scheme, the operation environment of the operator module is ensured, and the operation speed is increased.
In a specific implementation process, the nodes may be CPU resource nodes or GPU resource nodes.
The CPU is a Central Processing Unit (CPU) which is used as an operation and control core of the computer system and is a final execution unit for information processing and program operation; the GPU is a Graphics Processing Unit (abbreviated as GPU), also called a display core, a visual processor, and a display chip, and is a microprocessor that is specially used for image and Graphics related operations on a personal computer, a workstation, a game machine, and some mobile devices (such as a tablet computer, a smart phone, etc.).
The invention also protects an operator set construction device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the method.
The invention also protects an operator set using device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the method.
It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the protection scope of the claims of the present invention.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
It should be understood that the technical problems can be solved by combining and combining the features of the embodiments from the claims.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. An operator set construction method is characterized in that: the method comprises the following steps:
receiving by an operator unit, wherein the operator unit is a processing unit and is used for completing a corresponding processing task;
acquiring operator interface parameters, namely acquiring the operator interface parameters according to the operator unit, wherein the operator interface parameters comprise starting parameters and operating parameters;
and (5) constructing an operator module, namely constructing the operator unit with the acquired operator interface parameter into the operator module.
2. The operator set construction method according to claim 1, wherein: the operator set construction method further comprises the step of operator set construction, wherein a plurality of operator modules are placed in the first container, and the operator set is constructed.
3. The operator set construction method according to claim 1 or 2, characterized in that: the operation parameters comprise an inlet parameter and an outlet parameter, the inlet parameter is an input path of the operator module, the outlet parameter is an output path of the operator module, and the operator modules can be matched with each other through the matching of the inlet parameter and the outlet parameter.
4. The operator set construction method according to claim 3, wherein: the operator unit further comprises internal parameters, the internal parameters are used for the operator unit to complete corresponding processing tasks, the operator module comprises a first channel, and one end of the first channel is connected with the internal parameters.
5. The operator set construction method according to claim 4, wherein: the operator module construction further comprises construction of an operator label, wherein the operator label comprises an operator name.
6. The operator set construction method according to claim 5, wherein: the operator set construction further comprises an operator module directory construction, and the operator module directory construction step comprises:
receiving the operator module;
extracting an operator label in the operator module;
and constructing the operator module directory according to the operator label.
7. An operator set calling method, comprising the steps of:
constructing an operating environment, wherein the operating environment comprises at least one operating node;
receiving the operator set according to any of claims 4-6, outputting an operator module selection instruction;
receiving a selected operator module in the operator set;
and sending the operator module to an operation node for operation.
8. The operator set calling method according to claim 7, wherein: the internal parameters include operator operating parameters, and the step of receiving a selected operator module in the operator set further comprises:
outputting the operator operation parameters to an operator module surface layer through a first channel;
judging whether the operator operation parameters are changed or not;
if not, keeping the operation parameters of the original operator unchanged;
and if so, receiving the changed operator operating parameters, and updating the original operator operating parameters into the changed operator operating parameters.
9. The operator set calling method according to claim 8, wherein: the selected operator module may be a plurality of operator modules, and receiving the selected operator module in the operator set further comprises modeling the operator module, wherein the step of modeling the operator module comprises:
carrying out hierarchical sequencing on a plurality of operator modules;
in adjacent operator modules with upper and lower layer relations, the outlet parameters of the upper operator module are matched with the inlet parameters of the lower operator module, so that the output data of the upper operator module is received by the lower operator module.
10. An operator set building apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any of claims 1 to 6 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010773698.6A CN112130933A (en) | 2020-08-04 | 2020-08-04 | Method and device for constructing and calling operator set |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010773698.6A CN112130933A (en) | 2020-08-04 | 2020-08-04 | Method and device for constructing and calling operator set |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112130933A true CN112130933A (en) | 2020-12-25 |
Family
ID=73851482
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010773698.6A Pending CN112130933A (en) | 2020-08-04 | 2020-08-04 | Method and device for constructing and calling operator set |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112130933A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112784959A (en) * | 2021-01-13 | 2021-05-11 | 鹏城实验室 | Deep learning model rapid building system compatible with multiple frames |
| CN114610294A (en) * | 2022-05-09 | 2022-06-10 | 湖南星河云程信息科技有限公司 | Concurrent computation control method and device for performance indexes of simulation experiment and computer equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107169572A (en) * | 2016-12-23 | 2017-09-15 | 福州大学 | A kind of machine learning Service Assembly method based on Mahout |
| CN107621934A (en) * | 2017-07-28 | 2018-01-23 | 中国人民解放军国防信息学院 | Based on modularization, the evaluation index computational methods of graphical operator and device |
| CN107943945A (en) * | 2017-11-24 | 2018-04-20 | 清华大学 | Isomery operator management method in a kind of big data analysis development platform |
-
2020
- 2020-08-04 CN CN202010773698.6A patent/CN112130933A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107169572A (en) * | 2016-12-23 | 2017-09-15 | 福州大学 | A kind of machine learning Service Assembly method based on Mahout |
| CN107621934A (en) * | 2017-07-28 | 2018-01-23 | 中国人民解放军国防信息学院 | Based on modularization, the evaluation index computational methods of graphical operator and device |
| CN107943945A (en) * | 2017-11-24 | 2018-04-20 | 清华大学 | Isomery operator management method in a kind of big data analysis development platform |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112784959A (en) * | 2021-01-13 | 2021-05-11 | 鹏城实验室 | Deep learning model rapid building system compatible with multiple frames |
| CN114610294A (en) * | 2022-05-09 | 2022-06-10 | 湖南星河云程信息科技有限公司 | Concurrent computation control method and device for performance indexes of simulation experiment and computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109189750B (en) | Operation method, data analysis system and the storage medium of data analysis workflow | |
| CN108885545B (en) | Tools and methods for real-time dataflow programming languages | |
| US6243863B1 (en) | Apparatus and method for parallelizing legacy computer code | |
| US7634756B2 (en) | Method and apparatus for dataflow creation and execution | |
| US7614036B2 (en) | Method and system for dataflow creation and execution | |
| US7761281B2 (en) | System and method for performing compound computational experiments | |
| US10338963B2 (en) | System and method of schedule validation and optimization of machine learning flows for cloud computing | |
| US11256755B2 (en) | Tag mapping process and pluggable framework for generating algorithm ensemble | |
| Zhu et al. | A framework-based approach to utility big data analytics | |
| US10140098B2 (en) | Code generation | |
| CN111488211A (en) | Task processing method, device, equipment and medium based on deep learning framework | |
| CN101110029A (en) | Hmi framework for extensible automation system engineering platforms | |
| US11379203B2 (en) | Systems and methods for generating distributed software packages using non-distributed source code | |
| US12001823B2 (en) | Systems and methods for building and deploying machine learning applications | |
| US10747941B2 (en) | Tag mapping process and pluggable framework for generating algorithm ensemble | |
| CN112130933A (en) | Method and device for constructing and calling operator set | |
| CN112132530B (en) | Visual dynamic flow arranging method and system | |
| CN116738081A (en) | Front-end component binding method, device and storage medium | |
| CN110928535B (en) | Derived variable deployment method, device, equipment and readable storage medium | |
| WO2023207630A1 (en) | Task solving method and apparatus therefor | |
| CN112612481A (en) | System architecture of intelligent middle station | |
| CN108073389A (en) | A kind of automotive engine system based on script | |
| Krasanakis et al. | JGNN: Graph Neural Networks on native Java | |
| Moser et al. | Making expert knowledge explicit to facilitate tool support for integrating complex information systems in the atm domain | |
| CN110633077A (en) | Rapid development system and method based on modularization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201225 |