CN117056019A - Cluster processing method and device, electronic equipment and computer readable medium - Google Patents
Cluster processing method and device, electronic equipment and computer readable medium Download PDFInfo
- Publication number
- CN117056019A CN117056019A CN202311014321.2A CN202311014321A CN117056019A CN 117056019 A CN117056019 A CN 117056019A CN 202311014321 A CN202311014321 A CN 202311014321A CN 117056019 A CN117056019 A CN 117056019A
- Authority
- CN
- China
- Prior art keywords
- cluster
- determining
- target node
- identifier
- starting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 27
- 238000000034 method Methods 0.000 claims abstract description 64
- 230000008569 process Effects 0.000 claims abstract description 46
- 238000012545 processing Methods 0.000 claims abstract description 40
- 230000004044 response Effects 0.000 claims abstract description 25
- 238000004590 computer program Methods 0.000 claims description 16
- 230000000977 initiatory effect Effects 0.000 claims description 2
- 238000007781 pre-processing Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 15
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000007689 inspection Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 235000014510 cooky Nutrition 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The application discloses a cluster processing method, a cluster processing device, electronic equipment and a computer readable medium, and relates to the technical field of big data preprocessing, wherein a specific implementation mode comprises the steps of receiving a cluster processing request and acquiring a corresponding role identifier and a cluster identifier; determining a corresponding cluster starting sequence based on the cluster identification; calling a corresponding starting command according to the role identification and the cluster identification, executing the corresponding starting command based on the cluster starting sequence, and checking whether a process corresponding to the corresponding starting command is started or not after a preset time to obtain checking result data; acquiring availability identifiers of all nodes in a cluster corresponding to the cluster identifiers, and determining a target node based on the checking result data and the availability identifiers; and submitting the job task to a target node, determining a cluster shutdown sequence based on the cluster startup sequence in response to successful execution of the job task, and executing a corresponding cluster shutdown command based on the cluster shutdown sequence. The cluster processing efficiency is improved.
Description
Technical Field
The present application relates to the field of big data preprocessing, and in particular, to a cluster processing method, apparatus, electronic device, and computer readable medium.
Background
At present, the parallel transcoding system uses a plurality of servers and a plurality of open source software as a parallel transcoding system of a big data processing platform, directly uses an open source product and a plurality of machines, cannot support the starting of programs and the monitoring of the programs after the starting, needs a great deal of manual operation, and has low cluster processing efficiency.
Disclosure of Invention
In view of the above, embodiments of the present application provide a cluster processing method, apparatus, electronic device, and computer readable medium, which can solve the problem of low efficiency of the existing cluster processing.
To achieve the above object, according to an aspect of an embodiment of the present application, there is provided a cluster processing method, including:
receiving a cluster processing request, and acquiring a corresponding role identifier and a cluster identifier;
determining a corresponding cluster starting sequence based on the cluster identification;
calling a corresponding starting command according to the role identification and the cluster identification, executing the corresponding starting command based on the cluster starting sequence, and checking whether a process corresponding to the corresponding starting command is started or not after a preset time to obtain checking result data;
acquiring availability identifiers of all nodes in a cluster corresponding to the cluster identifiers, and determining a target node based on the checking result data and the availability identifiers;
and submitting the job task to a target node, determining a cluster shutdown sequence based on the cluster startup sequence in response to successful execution of the job task, and executing a corresponding cluster shutdown command based on the cluster shutdown sequence.
Optionally, determining the corresponding cluster start-up sequence includes:
determining the corresponding cluster type according to the cluster identification;
according to the cluster types, determining cluster subordination relationships, and further determining cluster starting sequences according to the cluster subordination relationships.
Optionally, before invoking the corresponding start command, the method further comprises:
determining a corresponding starting command list according to the character identification;
based on the cluster identification, a corresponding start command is determined from the start command list.
Optionally, determining the target node includes:
and in response to the fact that the processes corresponding to the corresponding start commands are started according to the check result data, determining that the availability identification corresponding to the available nodes in each slave node in the cluster is a target node.
Optionally, submitting the job task to the target node includes:
determining the number of job tasks according to the corresponding starting command;
and submitting the job tasks to the target node based on the number of job tasks.
Optionally, after submitting the job task to the target node, the method further comprises:
a check program is invoked to check whether the submitted job task is accepted and available by the target node, and in response to the job task being accepted and available by the target node, to determine that the job task was submitted successfully.
Optionally, determining the cluster shutdown sequence based on the cluster startup sequence includes:
and determining the reverse order of the cluster starting order as the cluster closing order.
In addition, the application also provides a cluster processing device, which comprises:
the receiving unit is configured to receive the cluster processing request and acquire the corresponding role identifier and the cluster identifier;
an order determining unit configured to determine a corresponding cluster start order based on the cluster identification;
the process starting unit is configured to call a corresponding starting command according to the role identifier and the cluster identifier, execute the corresponding starting command based on the cluster starting sequence, and after a preset time, check whether a process corresponding to the corresponding starting command is started or not, and acquire check result data;
the target node determining unit is configured to acquire the availability identifier of each node in the cluster corresponding to the cluster identifier, and determine the target node based on the checking result data and the availability identifier;
and the execution unit is configured to submit the job task to the target node, determine a cluster shutdown sequence based on the cluster startup sequence in response to successful execution of the job task, and execute a corresponding cluster shutdown command based on the cluster shutdown sequence.
Optionally, the order determination unit is further configured to:
determining the corresponding cluster type according to the cluster identification;
according to the cluster types, determining cluster subordination relationships, and further determining cluster starting sequences according to the cluster subordination relationships.
Optionally, the process starting unit is further configured to:
determining a corresponding starting command list according to the character identification;
based on the cluster identification, a corresponding start command is determined from the start command list.
Optionally, the target node determining unit is further configured to:
and in response to the fact that the processes corresponding to the corresponding start commands are started according to the check result data, determining that the availability identification corresponding to the available nodes in each slave node in the cluster is a target node.
Optionally, the execution unit is further configured to:
determining the number of job tasks according to the corresponding starting command;
and submitting the job tasks to the target node based on the number of job tasks.
Optionally, the execution unit is further configured to:
a check program is invoked to check whether the submitted job task is accepted and available by the target node, and in response to the job task being accepted and available by the target node, to determine that the job task was submitted successfully.
Optionally, the execution unit is further configured to:
and determining the reverse order of the cluster starting order as the cluster closing order.
In addition, the application also provides cluster processing electronic equipment, which comprises: one or more processors; and a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the cluster processing method as described above.
In addition, the application also provides a computer readable medium, on which a computer program is stored, which when executed by a processor, implements the cluster processing method as described above.
To achieve the above object, according to still another aspect of an embodiment of the present application, there is provided a computer program product.
The computer program product of the embodiment of the application comprises a computer program, and the cluster processing method provided by the embodiment of the application is realized when the program is executed by a processor.
One embodiment of the above application has the following advantages or benefits: the method comprises the steps of obtaining corresponding role identifiers and cluster identifiers by receiving a cluster processing request; determining a corresponding cluster starting sequence based on the cluster identification; calling a corresponding starting command according to the role identification and the cluster identification, executing the corresponding starting command based on the cluster starting sequence, and checking whether a process corresponding to the corresponding starting command is started or not after a preset time to obtain checking result data; acquiring availability identifiers of all nodes in a cluster corresponding to the cluster identifiers, and determining a target node based on the checking result data and the availability identifiers; and submitting the job task to a target node, determining a cluster shutdown sequence based on the cluster startup sequence in response to successful execution of the job task, and executing a corresponding cluster shutdown command based on the cluster shutdown sequence. Aiming at the start-stop work of the daily parallel transcoding cluster, the manual complex start-stop and check work is reduced through the corresponding start command, and the cluster processing efficiency is improved.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the application and are not to be construed as unduly limiting the application. Wherein:
FIG. 1 is a schematic diagram of the main flow of a cluster processing method according to one embodiment of the application;
FIG. 2 is a schematic diagram of the main flow of a cluster processing method according to one embodiment of the application;
FIG. 3 is a schematic flow diagram of a cluster processing method according to one embodiment of the application;
FIG. 4 is a schematic diagram of the main units of a cluster processing apparatus according to an embodiment of the application;
FIG. 5 is an exemplary system architecture diagram in which embodiments of the present application may be applied;
fig. 6 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the application.
Detailed Description
Exemplary embodiments of the present application will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present application are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness. In the technical scheme of the application, the aspects of acquisition, analysis, use, transmission, storage and the like of the related user personal information all meet the requirements of related laws and regulations, are used for legal and reasonable purposes, are not shared, leaked or sold outside the aspects of legal use and the like, and are subjected to supervision and management of a supervision department. Necessary measures should be taken for the personal information of the user to prevent illegal access to such personal information data, ensure that personnel having access to the personal information data comply with the regulations of the relevant laws and regulations, and ensure the personal information of the user. Once these user personal information data are no longer needed, the risk should be minimized by limiting or even prohibiting the data collection and/or deletion.
User privacy is protected by de-identifying data when used, including in some related applications, such as by removing a particular identifier, controlling the amount or specificity of stored data, controlling how data is stored, and/or other methods.
Fig. 1 is a schematic diagram of main flow of a cluster processing method according to an embodiment of the present application, and as shown in fig. 1, the cluster processing method includes:
step S101, a cluster processing request is received, and a corresponding role identifier and a corresponding cluster identifier are obtained.
In this embodiment, the execution body (for example, may be a server) of the cluster processing method may receive the cluster processing request through a wired connection or a wireless connection. In particular, the cluster processing request may be a request to control cluster start-stop. The embodiment of the application does not limit the content of the cluster processing request in detail. After receiving the cluster processing request, the executing body can acquire the role identifier and the cluster identifier carried in the request. In the context of master-slave mode, role identification may be used to characterize whether the service is provided by the master or slave. Embodiments of the application may relate to zookeeper clusters, hadoop clusters. The cluster identifier may be used to characterize clusters that are used by the cluster processing method of the embodiment of the present application, such as a zookeeper cluster and a hadoop cluster. The zookeeper cluster and the hadoop cluster are clusters which adopt a parallel transcoding master-slave mode to carry out service output. Specifically, the parallel transcoding is to use a "Spark on YARN" mode to transcode the file, and the coding mode is converted from EBCDIC to UTF-8. Wherein Spark on YARN: the calculation engine Spark application program is operated on the YARN cluster, any process service is not required to be started by Spark, and a Spark job task can be submitted to the YARN cluster for operation only by selecting one node to install Spark as a client. Master-slave mode: the master/works architecture is a partial idea embodiment of a computer system. The master machine (master) may be used to maintain cluster metadata and schedule based on such metadata, and the slave machine (woker) may be used to read and write specific data slices (storage systems) or as an execution unit for sub-tasks (computing systems).
Step S102, based on the cluster identification, determining a corresponding cluster starting sequence.
For example, the zookeeper develops as a hadoop sub-item, based on which, if the cluster corresponding to the cluster identifier includes the zookeeper cluster and the hadoop cluster, the corresponding cluster starting sequence may be determined as follows: zookeeper cluster-hadoop cluster. The hadoop cluster needs to be started after the zookeeper cluster is started.
Step S103, according to the role identification and the cluster identification, calling a corresponding starting command, executing the corresponding starting command based on the cluster starting sequence, and after a preset time, checking whether a process corresponding to the corresponding starting command is started or not, and obtaining check result data.
When the role identification corresponds to a slave machine, the cluster identification corresponds to a zookeeper cluster, the corresponding start command may be, for example: app_start.sh cookie can have the meaning: logging in a worker01, a worker02 and a worker03, and starting a zookeeper command to start a zookeeper program; and starting 30s, and checking whether a program of the distributed application program coordination service software zookeeper exists or not and whether the state of the node is normal or not.
When the role identifier corresponds to a master machine and a slave machine, the cluster identifier corresponds to a hadoop cluster, the corresponding start command may be, for example: app_start.sh hadoop, the meaning may be: logging in a master01 machine, and starting a program of an hdfs subsystem of the hadoop cluster; waiting 30 seconds, checking at the master node if there is a metadata node NameNode (metadata node is used to manage the namespaces of the file system, it holds metadata of all files and folders in one file system tree), and the failover controller DFSZKF ai lover controller, checking at the worker01-worker03 if there is a process start as follows: the data node DataNode and the journal memory node JohenalNode, and other worker nodes check whether the data node DataNode process is started or not; and then starting a program of a yarn subsystem of the hadoop cluster, waiting for 30s, checking whether a resource management resource manager process is started on a master node, and checking whether a node agent Nodemanager process is started on a worker node. And check if the worker node is available using the YARN node-list command. YARN is a resource manager, a common resource management system.
And executing the corresponding starting command based on the cluster starting sequence, and after a preset time (for example, 30 seconds), checking whether a process corresponding to the corresponding starting command is started or not, and acquiring check result data.
Step S104, the availability identification of each node in the cluster corresponding to the cluster identification is obtained, and the target node is determined based on the checking result data and the availability identification.
Specifically, determining the target node includes: in response to the start of each process corresponding to the corresponding start command corresponding to the check result data, for example, the start command app_start.sh hadoop corresponding to the metadata node NameNode, the corresponding process of the failover controller dfszkfailovertontroller, the corresponding process of the resource manager, and the corresponding process of the node proxy node manager are all started, the node corresponding to the availability identifier in each slave node in the cluster can be determined as the target node.
By way of example, a check is made as to whether a worker slave node in the corresponding cluster is available for cluster identification. The available woker slave node is determined to be the target node.
Step S105, submitting the job task to the target node, determining a cluster shutdown sequence based on the cluster startup sequence in response to successful execution of the job task, and executing a corresponding cluster shutdown command based on the cluster shutdown sequence.
Specifically, submitting a job task to a target node includes: determining the number of job tasks according to the corresponding starting command; and submitting the job tasks to the target node based on the number of job tasks.
By way of example, the corresponding start command may be: app_start.sh converter N, where N may be the number of job tasks. The corresponding start command may be input by a user, N in the start command may be set by the user, and the number N of tasks in the start command is not specifically limited in the embodiment of the present application, and the meaning of the start command app_start.sh converter N may be: the spark of the transcoding converter (with a cluster management, which is a separate scheduler) is submitted, and N is the number of job tasks. After start-up it is checked whether the job task is accepted and available by hadoop (distributed system infrastructure) clusters (in particular, it can be checked using the yarn application-list command).
The embodiment obtains the corresponding role identifier and the cluster identifier by receiving the cluster processing request; determining a corresponding cluster starting sequence based on the cluster identification; calling a corresponding starting command according to the role identification and the cluster identification, executing the corresponding starting command based on the cluster starting sequence, and checking whether a process corresponding to the corresponding starting command is started or not after a preset time to obtain checking result data; acquiring availability identifiers of all nodes in a cluster corresponding to the cluster identifiers, and determining a target node based on the checking result data and the availability identifiers; and submitting the job task to a target node, determining a cluster shutdown sequence based on the cluster startup sequence in response to successful execution of the job task, and executing a corresponding cluster shutdown command based on the cluster shutdown sequence. Aiming at the start-stop work of the daily parallel transcoding cluster, the manual complex start-stop and check work is reduced through the corresponding start command, and the cluster processing efficiency is improved.
Fig. 2 is a main flow diagram of a cluster processing method according to an embodiment of the present application, and as shown in fig. 2, the cluster processing method includes:
step S201, a cluster processing request is received, and a corresponding role identifier and a cluster identifier are obtained.
The role identification may be used to characterize whether the service is provided by the master or slave. The cluster identifier may be used to characterize clusters that are used by the cluster processing method of the embodiment of the present application, such as a zookeeper cluster and a hadoop cluster.
Step S202, determining a corresponding cluster starting sequence based on the cluster identification.
In the embodiment of the application, a mark representing the cluster starting sequence may be further set in the cluster identifier, for example, the cluster identifier may be a zookeeper-1, hadoop-2, where 1 and 2 may represent the cluster starting sequence, and the corresponding cluster starting sequence may be determined according to the acquired data or letters (e.g. a and b) by acquiring numbers or letters in the cluster identifier.
Step S203, according to the cluster identification, determining the corresponding cluster type.
The cluster identifier may also contain a flag for indicating a cluster type, for example, the cluster type represented by the zookeeper in the cluster identifier zookeeper-1 is a zookeeper cluster, and the cluster type represented by the hadoop in the hadoop-2 is a hadoop cluster.
Step S204, determining cluster affiliation according to cluster types, and further determining cluster starting sequence according to the cluster affiliation.
According to the determined cluster type and the corresponding relation between the preset cluster type and the cluster subordinate relation, the determined cluster subordinate relation can be that the zookeeper cluster belongs to the hadoop cluster, the starting sequence of the cluster with the cluster subordinate relation at the subordinate low position is the front, and in the example, the zookeeper cluster is at the subordinate low position compared with the hadoop cluster, and the starting sequence of the zookeeper cluster is earlier than that of the hadoop cluster, namely, the cluster starting sequence is that: zookeeper cluster-hadoop cluster.
Step S205, according to the role identification and the cluster identification, a corresponding start command is called, the corresponding start command is executed based on the cluster start sequence, after a preset time, whether a process corresponding to the corresponding start command is started or not is checked, and check result data is obtained.
Each character identifier and cluster identifier has a corresponding start command. The execution body may call a correspondence table composed of the role identifications and the cluster identifications and the start commands, so as to determine and call the corresponding start commands from the correspondence table according to the role identifications and the cluster identifications in the embodiment of the present application. And executing the corresponding starting command based on the determined cluster starting sequence, checking whether each process corresponding to the corresponding starting command is started normally or not after a preset time, for example, 30 seconds, and acquiring checking result data of whether each process is started normally or not.
Step S206, availability identifiers of all nodes in the cluster corresponding to the cluster identifiers are obtained, and the target node is determined based on the checking result data and the availability identifiers.
The availability identification may include Y, N, for example, where Y may represent that the node is available and N may represent that the node is unavailable. A node with Y in the availability identity is determined as the target node.
Step S207, submitting the job task to the target node, determining a cluster shutdown sequence based on the cluster startup sequence in response to successful execution of the job task, and executing a corresponding cluster shutdown command based on the cluster shutdown sequence.
The execution body may execute the app_start.sh allcvt N command: and sequentially starting a zookeeper cluster and a hadoop cluster, and submitting a converter job task. Then executing the script jspal.sh: and logging in master, worker nodes of the cluster, and displaying java programs started on the nodes.
Specifically, after submitting the job task to the target node, the method further comprises: a check program is invoked to check whether the submitted job task is accepted and available by the target node, and in response to the job task being accepted and available by the target node, to determine that the job task was submitted successfully.
Illustratively, after submitting the job task to the target node, the script app_stop. Sh is executed: the parameters are similar to app_start.sh, execute the stop command, and check the relevant programs on the machine, and the programs that remain after 30s are killed.
Fig. 3 is a main flow diagram of a cluster processing method according to an embodiment of the present application, and as shown in fig. 3, the cluster processing method includes:
step S301, a cluster processing request is received, and a corresponding role identifier and a cluster identifier are obtained.
Step S302, based on the cluster identification, determining a corresponding cluster starting sequence.
Step S303, determining a corresponding starting command list according to the character identification.
For example, when the role identification corresponds to the slave machine, it may be determined that the list of start commands involved by the slave machine may be: app_start.sh zookeeper, app _start.sh hadoop, app_start.sh converter N, app_start.sh calcvt N, jspal.sh, app_stop.sh.
Step S304, based on the cluster identification, determining a corresponding start command from the start command list.
After determining the start command list app_start.sh zookeeper, app _start.sh hash, app_start.sh converter N, app_start.sh allcvt N, after that, the execution subject may determine that the start command corresponding to the zookeeper is app_start.sh zookeeper according to the cluster identifier, for example, zookeeper and hadoop, determine that the start command related to the hadoop is app_start.sh hash, app_start.sh converter N, app_start.sh allcvt N, and execute script jspal, app_stop.sh after the start command is executed to display the java program started on the node and end the corresponding process by finally executing the stop command.
Step S305, calling a corresponding start command, executing the corresponding start command based on the cluster start sequence, and after a preset time, checking whether a process corresponding to the corresponding start command is started or not, and obtaining check result data.
After determining the startup command corresponding to the role identifier and the cluster identifier, the execution main body may execute the startup command based on the cluster startup sequence, for example, execute the startup command corresponding to the zookeeper cluster first, and execute the startup command corresponding to the hadoop cluster after the zookeeper cluster is started. After a preset time (for example, 30 seconds), the execution body checks whether each process corresponding to the executed start command has been started, and obtains the check result data. The inspection result data may include that each process is started and that there is no process started in each process, and the embodiment of the present application does not specifically limit the inspection result data.
Step S306, availability identifiers of all nodes in the cluster corresponding to the cluster identifiers are obtained, and the target node is determined based on the checking result data and the availability identifiers.
Step S307, submitting the job task to the target node, determining a cluster shutdown sequence based on the cluster startup sequence in response to successful execution of the job task, and executing a corresponding cluster shutdown command based on the cluster shutdown sequence.
Specifically, determining a cluster shutdown sequence based on the cluster startup sequence includes: and determining the reverse order of the cluster starting order as the cluster closing order. For example, when the cluster start sequence is a zookeeper cluster-hadoop cluster-converter application, the cluster shutdown sequence may be a converter application-hadoop cluster-zookeeper cluster. The execution body may execute an app_stop.sh allcvt command to close the controller application, hadoop cluster, and zookeeper cluster in sequence.
The embodiment of the application provides a script for starting and stopping a one-key hadoop application cluster and checking the starting and stopping condition in the starting and stopping process, and simultaneously provides a script for displaying a java program on a cluster machine, thereby being convenient for the operation and maintenance work of a daily parallel transcoding system. Aiming at the startup and shutdown operations of the daily parallel transcoding clusters, the embodiment of the application provides a simple script, reduces the manual complex startup, shutdown and inspection operations, and improves the capability of solving problems when serious problems occur in production environments and test environments (such as large-area shutdown). The cluster processing method of the embodiment of the application can be extended to other business components using hadoop clusters and zookeeper clusters, and improves the service level of automation of other system components.
Fig. 4 is a schematic diagram of main units of a cluster processing apparatus according to an embodiment of the application. As shown in fig. 4, the cluster processing apparatus 400 includes a receiving unit 401, a sequence determining unit 402, a process starting unit 403, a target node determining unit 404, and an executing unit 405.
The receiving unit 401 is configured to receive the cluster processing request, and obtain the corresponding role identifier and the cluster identifier.
An order determination unit 402 is configured to determine a corresponding cluster start-up order based on the cluster identity.
The process starting unit 403 is configured to invoke a corresponding starting command according to the role identifier and the cluster identifier, execute the corresponding starting command based on the cluster starting sequence, and after a preset time, check whether the process corresponding to the corresponding starting command is started, and obtain the check result data.
The target node determining unit 404 is configured to obtain the availability identifier of each node in the cluster corresponding to the cluster identifier, and determine the target node based on the inspection result data and the availability identifier.
An execution unit 405 configured to submit the job task to the target node, determine a cluster shutdown order based on the cluster startup order in response to successful execution of the job task, and execute a corresponding cluster shutdown command based on the cluster shutdown order.
In some embodiments, the order determination unit 402 is further configured to: determining the corresponding cluster type according to the cluster identification; according to the cluster types, determining cluster subordination relationships, and further determining cluster starting sequences according to the cluster subordination relationships.
In some embodiments, process initiation unit 403 is further configured to: determining a corresponding starting command list according to the character identification; based on the cluster identification, a corresponding start command is determined from the start command list.
In some embodiments, the target node determination unit 404 is further configured to: and in response to the fact that the processes corresponding to the corresponding start commands are started according to the check result data, determining that the availability identification corresponding to the available nodes in each slave node in the cluster is a target node.
In some embodiments, the execution unit 405 is further configured to: determining the number of job tasks according to the corresponding starting command; and submitting the job tasks to the target node based on the number of job tasks.
In some embodiments, the execution unit 405 is further configured to: a check program is invoked to check whether the submitted job task is accepted and available by the target node, and in response to the job task being accepted and available by the target node, to determine that the job task was submitted successfully.
In some embodiments, the execution unit 405 is further configured to: and determining the reverse order of the cluster starting order as the cluster closing order.
It should be noted that, the cluster processing method and the cluster processing device of the present application have a corresponding relationship in the implementation content, so the repeated content will not be described.
Fig. 5 illustrates an exemplary system architecture 500 in which a cluster processing method or cluster processing apparatus of an embodiment of the application may be applied.
As shown in fig. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 is used as a medium to provide communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 505 via the network 504 using the terminal devices 501, 502, 503 to receive or send messages or the like. Various communication client applications may be installed on the terminal devices 501, 502, 503, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 501, 502, 503 may be various electronic devices with clustered screens and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 505 may be a server providing various services, such as a background management server (by way of example only) providing support for cluster processing requests submitted by users using the terminal devices 501, 502, 503. The background management server can receive the cluster processing request and acquire the corresponding role identifier and cluster identifier; determining a corresponding cluster starting sequence based on the cluster identification; calling a corresponding starting command according to the role identification and the cluster identification, executing the corresponding starting command based on the cluster starting sequence, and checking whether a process corresponding to the corresponding starting command is started or not after a preset time to obtain checking result data; acquiring availability identifiers of all nodes in a cluster corresponding to the cluster identifiers, and determining a target node based on the checking result data and the availability identifiers; and submitting the job task to a target node, determining a cluster shutdown sequence based on the cluster startup sequence in response to successful execution of the job task, and executing a corresponding cluster shutdown command based on the cluster shutdown sequence. Aiming at the start-stop work of the daily parallel transcoding cluster, the manual complex start-stop and check work is reduced through the corresponding start command, and the cluster processing efficiency is improved.
It should be noted that, in the embodiment of the present application, the cluster processing method is generally executed by the server 505, and accordingly, the cluster processing device is generally disposed in the server 505.
It should be understood that the number of terminal devices, networks and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 6, there is illustrated a schematic diagram of a computer system 600 suitable for use in implementing an embodiment of the present application. The terminal device shown in fig. 6 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present application.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the computer system 600 are also stored. The CPU601, ROM602, and RAM603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a liquid crystal credit authorization query processor (LCD), and the like, and a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present application are performed when the computer program is executed by a Central Processing Unit (CPU) 601.
The computer readable medium shown in the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present application may be implemented in software or in hardware. The described units may also be provided in a processor, for example, described as: a processor includes a receiving unit, a sequence determining unit, a process starting unit, a target node determining unit, and an executing unit. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.
As another aspect, the present application also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs, which when executed by one of the devices, cause the device to receive a cluster processing request, and obtain a corresponding role identifier and a cluster identifier; determining a corresponding cluster starting sequence based on the cluster identification; calling a corresponding starting command according to the role identification and the cluster identification, executing the corresponding starting command based on the cluster starting sequence, and checking whether a process corresponding to the corresponding starting command is started or not after a preset time to obtain checking result data; acquiring availability identifiers of all nodes in a cluster corresponding to the cluster identifiers, and determining a target node based on the checking result data and the availability identifiers; and submitting the job task to a target node, determining a cluster shutdown sequence based on the cluster startup sequence in response to successful execution of the job task, and executing a corresponding cluster shutdown command based on the cluster shutdown sequence.
The computer program product of the present application comprises a computer program which, when executed by a processor, implements the cluster processing method in the embodiment of the present application.
According to the technical scheme provided by the embodiment of the application, aiming at the start-stop work of the daily parallel transcoding cluster, the manual complex start-stop and checking work is reduced through the corresponding start command, and the cluster processing efficiency is improved.
The above embodiments do not limit the scope of the present application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of the present application.
Claims (16)
1. A cluster processing method, comprising:
receiving a cluster processing request, and acquiring a corresponding role identifier and a cluster identifier;
determining a corresponding cluster starting sequence based on the cluster identification;
calling a corresponding starting command according to the role identifier and the cluster identifier, executing the corresponding starting command based on the cluster starting sequence, and checking whether a process corresponding to the corresponding starting command is started or not after a preset time to obtain checking result data;
acquiring availability identifiers of all nodes in the cluster corresponding to the cluster identifiers, and determining a target node based on the checking result data and the availability identifiers;
and submitting a job task to the target node, determining a cluster shutdown sequence based on the cluster startup sequence in response to successful execution of the job task, and executing a corresponding cluster shutdown command based on the cluster shutdown sequence.
2. The method of claim 1, wherein the determining the corresponding cluster start-up sequence comprises:
determining the corresponding cluster type according to the cluster identifier;
and determining a cluster subordinate relation according to the cluster type, and further determining a cluster starting sequence according to the cluster subordinate relation.
3. The method of claim 1, wherein prior to the invoking the corresponding start command, the method further comprises:
determining a corresponding starting command list according to the role identifier;
and determining a corresponding starting command from the starting command list based on the cluster identification.
4. The method of claim 1, wherein the determining the target node comprises:
and in response to the checking result data corresponding to the processes corresponding to the corresponding starting commands are started, determining that the availability identification in each slave node in the cluster corresponds to an available node as a target node.
5. The method of claim 1, wherein the submitting job tasks to the target node comprises:
determining the number of job tasks according to the corresponding starting command;
and submitting the job tasks to the target node based on the job task number.
6. The method of claim 1, wherein after the submitting the job task to the target node, the method further comprises:
and invoking a checking program to check whether the submitted job task is accepted and available by the target node, and determining that the job task is submitted successfully in response to the job task being accepted and available by the target node.
7. The method of claim 1, wherein the determining a cluster shutdown order based on the cluster startup order comprises:
and determining the reverse order of the cluster starting order as a cluster closing order.
8. A cluster processing apparatus, comprising:
the receiving unit is configured to receive the cluster processing request and acquire the corresponding role identifier and the cluster identifier;
an order determining unit configured to determine a corresponding cluster start order based on the cluster identification;
the process starting unit is configured to call a corresponding starting command according to the role identifier and the cluster identifier, execute the corresponding starting command based on the cluster starting sequence, and after a preset time, check whether a process corresponding to the corresponding starting command is started or not, and acquire check result data;
the target node determining unit is configured to acquire the availability identifier of each node in the cluster corresponding to the cluster identifier, and determine the target node based on the checking result data and the availability identifier;
and the execution unit is configured to submit a job task to the target node, determine a cluster shutdown sequence based on the cluster startup sequence in response to successful execution of the job task, and execute a corresponding cluster shutdown command based on the cluster shutdown sequence.
9. The apparatus of claim 8, wherein the order determination unit is further configured to:
determining the corresponding cluster type according to the cluster identifier;
and determining a cluster subordinate relation according to the cluster type, and further determining a cluster starting sequence according to the cluster subordinate relation.
10. The apparatus of claim 8, wherein the process initiation unit is further configured to:
determining a corresponding starting command list according to the role identifier;
and determining a corresponding starting command from the starting command list based on the cluster identification.
11. The apparatus of claim 8, wherein the target node determination unit is further configured to:
and in response to the checking result data corresponding to the processes corresponding to the corresponding starting commands are started, determining that the availability identification in each slave node in the cluster corresponds to an available node as a target node.
12. The apparatus of claim 8, wherein the execution unit is further configured to:
determining the number of job tasks according to the corresponding starting command;
and submitting the job tasks to the target node based on the job task number.
13. The apparatus of claim 8, wherein the execution unit is further configured to:
and invoking a checking program to check whether the submitted job task is accepted and available by the target node, and determining that the job task is submitted successfully in response to the job task being accepted and available by the target node.
14. A cluster processing electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-7.
15. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-7.
16. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311014321.2A CN117056019A (en) | 2023-08-11 | 2023-08-11 | Cluster processing method and device, electronic equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311014321.2A CN117056019A (en) | 2023-08-11 | 2023-08-11 | Cluster processing method and device, electronic equipment and computer readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117056019A true CN117056019A (en) | 2023-11-14 |
Family
ID=88661962
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311014321.2A Pending CN117056019A (en) | 2023-08-11 | 2023-08-11 | Cluster processing method and device, electronic equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117056019A (en) |
-
2023
- 2023-08-11 CN CN202311014321.2A patent/CN117056019A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112445868B (en) | Service message processing method and device | |
| CN116303608A (en) | Data processing method and device for application service | |
| CN111831503A (en) | Monitoring method based on monitoring agent and monitoring agent device | |
| CN113010238A (en) | Permission determination method, device and system for micro application call interface | |
| CN117131516B (en) | Operation and maintenance method and device | |
| CN113282589A (en) | Data acquisition method and device | |
| CN109683942B (en) | Script management method, script management device, script management medium and electronic equipment | |
| CN117056019A (en) | Cluster processing method and device, electronic equipment and computer readable medium | |
| CN116775613A (en) | Data migration method, device, electronic equipment and computer readable medium | |
| CN116737662A (en) | Method, device, electronic equipment and storage medium for processing business data | |
| CN112559001B (en) | Method and device for updating application | |
| CN112241332B (en) | Interface compensation method and device | |
| CN117743264A (en) | File storage mode conversion method and device | |
| CN118656190A (en) | Data scanning method, device, electronic equipment and computer readable medium | |
| CN117082149A (en) | Transaction processing method, device, electronic equipment and computer readable medium | |
| CN118747113A (en) | Task execution method and device, electronic equipment and computer readable medium | |
| CN118295806A (en) | Distributed task processing method and device | |
| CN118673483A (en) | Application security processing method and device | |
| CN118192941A (en) | Data processing method and device | |
| CN116166404A (en) | Task processing method and device | |
| CN116755758A (en) | Data processing method, device, electronic equipment and computer readable medium | |
| CN118606031A (en) | Task processing method and device, electronic equipment and computer readable medium | |
| CN115993981A (en) | Data processing method, device, electronic equipment and computer readable medium | |
| CN118819930A (en) | Data analysis method, device, electronic equipment and computer readable medium | |
| CN117311804A (en) | Application migration method, device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |