CN110245125B - Data migration method, device, computer equipment and storage medium - Google Patents
Data migration method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110245125B CN110245125B CN201910418940.5A CN201910418940A CN110245125B CN 110245125 B CN110245125 B CN 110245125B CN 201910418940 A CN201910418940 A CN 201910418940A CN 110245125 B CN110245125 B CN 110245125B
- Authority
- CN
- China
- Prior art keywords
- data
- task
- migration
- data table
- address
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013508 migration Methods 0.000 title claims abstract description 243
- 230000005012 migration Effects 0.000 title claims abstract description 243
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000012545 processing Methods 0.000 claims abstract description 154
- 230000008569 process Effects 0.000 claims abstract description 29
- 238000004590 computer program Methods 0.000 claims description 20
- 238000012216 screening Methods 0.000 claims description 7
- 238000012790 confirmation Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 239000000284 extract Substances 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data migration method, a data migration device, computer equipment and a storage medium. The method comprises the following steps: scanning data processing tasks in an operating state; inquiring a task data table on which the data processing task depends; creating a temporary data table corresponding to the queried task data table; directing a data access address in the data processing task to the temporary data table; performing data migration on the task data table; and when the task data table is migrated, pointing the data access address in the data processing task to the migrated task data table. By adopting the method, the data processing task can be executed without interruption in the migration process of the task data table, and the data processing efficiency of the data processing task is improved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data migration method, apparatus, computer device, and storage medium.
Background
With the development of computer technology and internet technology, various services are processed through a computer and transmitted through the internet, a large amount of data needs to be stored, the number of databases storing the data is increased and decreased, and data in the databases sometimes needs to be migrated according to service requirements among the databases, so that related technologies of large data migration appear.
However, in the conventional big data migration technology, the data in the database can be migrated only after the access to the data is stopped in the migration process, and the data access can be continued until the data migration in the database is completed, so that the data processing task of the online service is influenced, and the data processing efficiency of the data processing task is reduced.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data migration method, apparatus, computer device, and storage medium capable of improving data processing efficiency of a data processing task.
A method of data migration, the method comprising:
Scanning data processing tasks in an operating state;
inquiring a task data table on which the data processing task depends;
creating a temporary data table corresponding to the queried task data table;
Directing a data access address in the data processing task to the temporary data table;
performing data migration on the task data table;
And when the task data table is migrated, pointing the data access address in the data processing task to the migrated task data table.
A data migration apparatus, the apparatus comprising:
The task scanning module is used for scanning the data processing task in the running state;
the data table query module is used for querying a task data table on which the data processing task depends;
The data table creating module is used for creating a temporary data table corresponding to the inquired task data table;
the pointing adjustment module is used for pointing the data access address in the data processing task to the temporary data table;
The data migration module is used for carrying out data migration on the task data table;
And the migration monitoring module is used for pointing the data access address in the data processing task to the migrated task data table when the task data table is migrated.
A computer device comprising a memory storing a computer program and a processor which when executing the computer program performs the steps of:
Scanning data processing tasks in an operating state;
inquiring a task data table on which the data processing task depends;
creating a temporary data table corresponding to the queried task data table;
Directing a data access address in the data processing task to the temporary data table;
performing data migration on the task data table;
And when the task data table is migrated, pointing the data access address in the data processing task to the migrated task data table.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
Scanning data processing tasks in an operating state;
inquiring a task data table on which the data processing task depends;
creating a temporary data table corresponding to the queried task data table;
Directing a data access address in the data processing task to the temporary data table;
performing data migration on the task data table;
And when the task data table is migrated, pointing the data access address in the data processing task to the migrated task data table.
According to the data migration method, the data migration device, the computer equipment and the storage medium, the task data table on which the data processing task in the running state depends is queried, the temporary data table corresponding to the task data table is created, and the data access address in the data processing task in the running state is pointed to the temporary data table, so that the data processing task can normally run when the task data table is migrated. When the task data table is migrated, the data access address in the data processing task is pointed to the migrated task data table, and meanwhile, when the task data table is migrated, the data processing task can rely on the migrated task data table to continue to run, so that the data processing task does not interrupt execution in the process of migrating the task data table, and the data processing efficiency of the data processing task is improved.
Drawings
FIG. 1 is an application scenario diagram of a data migration method in one embodiment;
FIG. 2 is a flow chart of a method of data migration in one embodiment;
FIG. 3 is a flowchart illustrating a process for validating a memory space in one embodiment;
FIG. 4 is a flow chart illustrating steps for screening data processing tasks in one embodiment;
FIG. 5 is a flowchart illustrating steps for migrating a task data table in one embodiment;
FIG. 6 is a flow diagram that illustrates the steps of generating a data migration report in one embodiment;
FIG. 7 is a flow chart illustrating steps for configuring access rights in one embodiment;
FIG. 8 is a block diagram of a data migration apparatus in one embodiment;
Fig. 9 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
The data migration method provided by the application can be applied to an application environment shown in figure 1. Wherein the terminal 102 communicates with the server 104 via a network, the server 104 communicates with the server 106 via a network, the server 106 communicates with the server 108 via a network, and the server 108 communicates with the server 104 via a network. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smartphones, tablet computers, and portable wearable devices, and the server 104, the server 106, and the server 108 may be implemented as separate servers or a server cluster composed of a plurality of servers. Server 104, server 106, and server 108 may be integrated into the same server.
In one embodiment, as shown in fig. 2, a data migration method is provided, and the method is applied to the server in fig. 1 for illustration, and includes the following steps:
s202, scanning data processing tasks in an operation state.
Specifically, the server receives a data migration request, scans data processing tasks in response to the data migration request, and determines the data processing tasks in an operating state through scanning. The data processing task in the running state is a task that is performing data processing.
In one embodiment, a data migration page is displayed in a terminal, when a click operation of a request trigger button in the data migration page is detected, input data migration parameters in the data migration page are acquired, a data migration request is generated according to the data migration parameters, and the data migration request is sent to a server. And the server receives the data migration request sent by the terminal.
S204, inquiring a task data table on which the data processing task depends.
Specifically, the server extracts task identifiers of the data processing tasks, queries data processing information corresponding to each task identifier, extracts data table identifiers from the data processing information, and the task data table corresponding to the extracted data table identifiers is a task data table on which the data processing tasks depend. The task data table on which the data processing task depends is a task data table that needs to be accessed to execute the data processing task.
S206, creating a temporary data table corresponding to the queried task data table.
Specifically, when the server queries the task data table on which the data processing task depends, the task data table on which each data processing task depends is copied, and the copied data table is used as a temporary data table corresponding to the task data table, so that the temporary data table corresponding to each task data table is obtained.
In one embodiment, after the server queries the data processing task data tables, the server reads the data in each data processing task table, and adds the read data to the pre-created data table to generate a temporary data table corresponding to the task data table.
S208, the data access address in the data processing task is pointed to the temporary data table.
Specifically, when the server creates the temporary data table, the access address of the temporary data table is recorded. The server analyzes the data processing task, determines the data access address in the data processing task through analysis, replaces the determined data access address with the access address of the temporary data table, and changes the data access address in the data processing task so that the changed data access address in the data processing task points to the temporary data table.
In one embodiment, when the server creates the temporary data table, the access address of the temporary data table is stored in correspondence with the data access address of the corresponding task data table. After determining the data access address in the data processing task, the server inquires the access address of the temporary data table corresponding to the determined data access address, and modifies the data access address in the data processing task to the access address corresponding to the inquired temporary data table.
S210, performing data migration on the task data table.
Specifically, the server acquires an migrating database address, establishes connection with the migrating database according to the migrating database address, and sends the task data table to the migrating database through the established connection.
In one embodiment, a server obtains a data table identifier and an migrating database address of a task data table, creates a migrating task according to the obtained data table identifier and the migrating database address, and constructs a migrating task queue according to the created migrating task. The server reads the migration task from the migration task queue, inquires a task data table according to the read data table identification in the migration task queue, and migrates the read task data table into a migration database corresponding to the migration database address in the migration task.
S212, when the task data table is migrated, the data access address in the data processing task is pointed to the migrated task data table.
Specifically, when the server migrates the task data table, the server monitors migration of the task data table, and when the task data table is completely migrated, acquires a migration access address corresponding to the migrated task data table, and modifies the data access address in the data processing task into the migration access address so that the data access address in the data processing task points to the migrated task data table.
In one embodiment, the server monitors the migration task queue, and when the migration task in the migration task queue is monitored to be read, the server monitors that the migration of the task data table is completed.
In this embodiment, a task data table on which a data processing task in an operating state depends is queried, a temporary data table corresponding to the task data table is created, and a data access address in the data processing task in the operating state is pointed to the temporary data table, so that the data processing task can normally operate when the task data table is migrated. When the task data table is migrated, the data access address in the data processing task is pointed to the migrated task data table, and meanwhile, when the task data table is migrated, the data processing task can rely on the migrated task data table to continue to run, so that the data processing task does not interrupt execution in the process of migrating the task data table, and the data processing efficiency of the data processing task is improved.
As shown in fig. 3, in one embodiment, S202 further includes a step of confirming the storage space, where the step specifically includes the following:
S302, acquiring a data migration instruction.
Specifically, the terminal obtains the set migrating database address and the migrating database address, encapsulates the obtained migrating database address and the obtained migrating database address, generates a data migration instruction through encapsulation, and sends the data migration instruction to the server 104 in the big data platform. The big data platform is composed of at least server 104, server 106 and server 106.
S304, extracting the migrating database address and the migrating database address in the data migration instruction.
Specifically, after receiving the data migration instruction, the server 104 parses the data migration instruction, and extracts the migrated database address and the migrated database address from the data migration instruction through parsing. The migrate database address is the address of the migrate database where the task data table is currently located, and the migrate database may be set in the server 106. The migration database address is the address of the migration database to which the task data table is to be migrated, and the migration database may be set in the server 108.
S306, counting the stored space capacity in the migrated database address.
Specifically, server 104 generates a used capacity query request from the migrating database address and sends the used capacity query request to server 106. The server 106 analyzes the used capacity inquiry request, extracts the migration database address in the used capacity inquiry request, counts the stored space capacity in the migration database corresponding to the migration database address, and returns the stored space capacity to the server 104. Server 104 receives the storage space capacity returned by server 106.
S308, determining whether the residual space capacity corresponding to the migrated database address is larger than the stored space capacity.
Specifically, server 104 generates a residual capacity query request from the migration database address, and sends the residual capacity query request to server 108. Server 108 parses the residual capacity query request, extracts the migration database address in the residual capacity query request, counts the residual space capacity in the migration database corresponding to the migration database address, and returns the residual space capacity to server 104. Server 104 receives the remaining space capacity returned by server 106. Server 104 compares the remaining space capacity of the migration database with the stored space capacity of the migration database to determine whether the remaining space capacity is greater than the stored space capacity.
And S310, when the residual space capacity is larger than the stored space capacity, scanning the data processing task in the running state.
Specifically, when the remaining space capacity is greater than the stored space capacity, it indicates that there is enough storage space in the migration database to store the task data table migrated in the migration database; when the residual space capacity is smaller than or equal to the stored space capacity, the method indicates that insufficient storage space exists in the migration database to store the migrated task data table in the migration database. And when the residual space capacity is larger than the stored space capacity, scanning the data processing task in the running state.
In this embodiment, after the data migration instruction is obtained, the stored space capacity in the migration database and the remaining space capacity corresponding to the migration database are counted, and when the remaining space capacity is greater than the stored space capacity, the processing task in the running state is scanned, so that sufficient remaining space capacity in the migration database is ensured, data migration is restarted, and data migration abnormality caused by insufficient storage space in the data migration process is avoided.
As shown in fig. 4, in one embodiment, S202 specifically includes a step of screening data processing tasks, which specifically includes the following:
s402, scanning task identifiers corresponding to data processing tasks.
Specifically, the server 104 scans the data processing task, and determines a task identifier corresponding to the data processing task through scanning. The task identification is an identification for uniquely identifying the data processing task.
S404, determining the working states corresponding to the task identifiers.
Specifically, after the server 104 scans task identifiers corresponding to the data processing tasks, for each task identifier, a state identifier corresponding to each task identifier is extracted, and a working state corresponding to the state identifier corresponding to each task identifier is identified.
S406, screening the data processing task corresponding to the task identifier with the working state being the running state.
Specifically, the server 104 identifies a task identifier whose working state is an operation state, determines a task identifier corresponding to the state identifier of the operation state, and screens a data processing task corresponding to the determined task identifier.
In this embodiment, according to the working state corresponding to the task identifier, the data processing task corresponding to the task identifier in the running state is determined, so as to determine the data processing task that needs to be migrated and is in the running state, and ensure accurate migration of the data processing task in the running state.
As shown in fig. 5, in one embodiment, S210 specifically includes a step of migrating a task data table, which specifically includes the following:
S502, creating a data migration task according to the task data table.
Specifically, the server acquires the data table identifier of the task data table, creates a data migration task corresponding to the task data table according to the acquired data table identifier, and obtains the data migration task corresponding to each task data table. The data migration task is a task for migrating a task data table.
S504, adding the created data migration tasks into a preset number of migration task queues.
Specifically, a preset number of migration task queues are arranged in the server. And the server adds the data migration task corresponding to the created task data table into each migration task queue to obtain each migration task queue comprising the data migration task.
S506, the data migration tasks in each migration task queue are read in parallel.
Specifically, the server processes the data migration tasks in each migration task queue in parallel. The server reads the data migration task in each migration task queue in parallel.
S508, performing data migration on the task data table corresponding to the read data migration task.
Specifically, the server analyzes each read data migration task, analyzes and extracts a task table identifier in the data migration task, queries a task data table corresponding to the task table identifier, acquires an migrating database address, and stores the queried task data table in a database corresponding to the migrating database address.
In this embodiment, a data migration task is created according to a task data table, the created data migration task is added to a preset number of migration task queues, and the data migration task is processed in parallel by reading the data migration task in each migration task queue in parallel, so that network congestion caused by excessive data migration tasks is avoided, and the migration efficiency of the task data table is reduced.
As shown in fig. 6, in one embodiment, S210 further includes a step of generating a data migration report, where the step includes the following steps:
S602, recording migration process data of a task data table.
Specifically, when the server migrates the task data table, the server records migration process data of each task data table. The migration process data includes at least one of a migration state, a migration start time, a migration end time, and a time period spent for migration of the task data table.
S604, generating a data migration report according to the recorded migration process data.
Specifically, after the recorded migration process data, the server correspondingly stores the data table identification of the task data table and the migration process data, and generates a data migration report.
And S606, sending the data migration report to a terminal logged in by an administrator account.
Specifically, after generating a data migration report, the server acquires an administrator account, inquires a terminal address corresponding to the administrator account, and sends the data migration report to a terminal corresponding to the inquired terminal address.
In this embodiment, during the migration process of the task data table, the migration process data of the task data table is recorded, a data migration report is generated according to the migration process data, the migration state during the data migration process is recorded through the data migration report, and the data migration report is sent to the terminal logged in by the administrator account, so that it is ensured that the administrator can monitor the data through the data migration report.
As shown in fig. 7, in one embodiment, S212 further includes a step of configuring access rights, where the step includes the following:
S702, access right information corresponding to the task data table is obtained.
Specifically, the server statically acquires the data table identifiers of the task data tables, generates an access right acquisition request corresponding to the data table identifiers, sends the access right acquisition request to a database corresponding to an migrating database address, and acquires access right information corresponding to the data table identifiers, which is queried by the migrating database according to the access right acquisition request.
S704, determining a migrated task data table corresponding to the task data table.
Specifically, after the task data table is migrated, the server acquires the data table address of the migrated task data table, and correspondingly stores the data table address after the migration and the data table identifier of the task data table. The server inquires the data table address corresponding to the data table identifier, and the inquired data table address is the data table address corresponding to the task data table after migration.
S706, configuring access rights to the migrated data table address according to the access rights information.
Specifically, after migrating the task data table to the migration database corresponding to the migration database address, the server determines the migrated task data table, and configures access rights for the migrated data table address according to the access rights information.
In this embodiment, after the task data table is migrated, the access authority of the migrated task data table is configured according to the access authority information of the task data table, so that access control on the migrated task data table is ensured, and the security of the migrated task data table is ensured.
It should be understood that, although the steps in the flowcharts of fig. 2-7 are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 2-7 may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor does the order in which the sub-steps or stages are performed necessarily occur in sequence, but may be performed alternately or alternately with at least a portion of the sub-steps or stages of other steps or other steps.
In one embodiment, as shown in FIG. 8, there is provided a data migration apparatus 800 comprising: a task scan module 802, a data table query module 804, a data table creation module 806, a direction adjustment module 808, a data migration module 810, and a migration monitor module 812, wherein:
task scanning module 802 is configured to scan data processing tasks in an operational state.
The data table query module 804 is configured to query a task data table on which a data processing task depends.
The data table creating module 806 is configured to create a temporary data table corresponding to the queried task data table.
The direction adjustment module 808 is configured to direct the data access address in the data processing task to the temporary data table.
The data migration module 810 is configured to perform data migration on the task data table.
And the migration monitoring module 812 is configured to direct the data access address in the data processing task to the migrated task data table when the task data table is migrated.
In one embodiment, the data migration apparatus 800 specifically further includes: and a capacity confirmation module.
The capacity confirmation module is used for acquiring a data migration instruction; extracting an migrating database address and an migrating database address in a data migration instruction; counting the stored space capacity corresponding to the migrated database address; determining whether the residual space capacity corresponding to the migrated database address is larger than the stored space capacity;
Task scanning module 802 is also configured to scan data processing tasks in an operational state when the remaining space capacity is greater than the stored space capacity.
In one embodiment, the task scanning module 802 is further configured to scan task identifiers corresponding to each of the plurality of data processing tasks; determining a working state corresponding to the task identifier; and screening the data processing task corresponding to the task identifier with the working state being the running state.
In one embodiment, the data migration module 810 is further configured to create a data migration task from the task data table; adding the created data migration tasks into a preset number of migration task queues; reading data migration tasks in each migration task queue in parallel; and carrying out data migration on the task data table corresponding to the read data migration task.
In one embodiment, the data migration module 810 is further configured to record migration process data of the task data table; generating a data migration report according to the recorded migration process data; and sending the data migration report to a terminal logged in by an administrator account.
In one embodiment, the data migration module 810 is further configured to obtain access rights information corresponding to the task data table; determining the transferred data table address corresponding to the task data table; and configuring access rights to the migrated data table address according to the access rights information.
In this embodiment, a task data table on which a data processing task in an operating state depends is queried, a temporary data table corresponding to the task data table is created, and a data access address in the data processing task in the operating state is pointed to the temporary data table, so that the data processing task can normally operate when the task data table is migrated. When the task data table is migrated, the data access address in the data processing task is pointed to the migrated task data table, and meanwhile, when the task data table is migrated, the data processing task can rely on the migrated task data table to continue to run, so that the data processing task does not interrupt execution in the process of migrating the task data table, and the data processing efficiency of the data processing task is improved.
For specific limitations of the data migration apparatus, reference may be made to the above limitation of the data migration method, and no further description is given here. The various modules in the data migration apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 9. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing a data table. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data migration method.
It will be appreciated by persons skilled in the art that the architecture shown in fig. 9 is merely a block diagram of some of the architecture relevant to the present inventive arrangements and is not limiting as to the computer device to which the present inventive arrangements are applicable, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a computer device is provided comprising a memory storing a computer program and a processor that when executing the computer program performs the steps of: scanning data processing tasks in an operating state; inquiring a task data table on which the data processing task depends; creating a temporary data table corresponding to the queried task data table; directing a data access address in a data processing task to a temporary data table; performing data migration on the task data table; and when the task data table is migrated, the data access address in the data processing task is pointed to the migrated task data table.
In one embodiment, before scanning the data processing task in the running state, the processor when executing the computer program further performs the steps of: acquiring a data migration instruction; extracting an migrating database address and an migrating database address in a data migration instruction; counting the stored space capacity corresponding to the migrated database address; determining whether the residual space capacity corresponding to the migrated database address is larger than the stored space capacity; and when the residual space capacity is larger than the stored space capacity, scanning the data processing task in the running state.
In one embodiment, scanning data processing tasks in an operational state includes: scanning task identifiers corresponding to the data processing tasks respectively; determining a working state corresponding to the task identifier; and screening the data processing task corresponding to the task identifier with the working state being the running state.
In one embodiment, performing data migration on a task data table includes: creating a data migration task according to the task data table; adding the created data migration tasks into a preset number of migration task queues; reading data migration tasks in each migration task queue in parallel; and carrying out data migration on the task data table corresponding to the read data migration task.
In one embodiment, after data migration of the task data table, the processor when executing the computer program further performs the steps of: recording migration process data of a task data table; generating a data migration report according to the recorded migration process data; and sending the data migration report to a terminal logged in by an administrator account.
In one embodiment, after the task data table is migrated, the processor further performs the following steps when executing the computer program, after pointing the data access address in the data processing task to the migrated task data table: obtaining access right information corresponding to a task data table; determining the transferred data table address corresponding to the task data table; and configuring access rights to the migrated data table address according to the access rights information.
In this embodiment, a task data table on which a data processing task in an operating state depends is queried, a temporary data table corresponding to the task data table is created, and a data access address in the data processing task in the operating state is pointed to the temporary data table, so that the data processing task can normally operate when the task data table is migrated. When the task data table is migrated, the data access address in the data processing task is pointed to the migrated task data table, and meanwhile, when the task data table is migrated, the data processing task can rely on the migrated task data table to continue to run, so that the data processing task does not interrupt execution in the process of migrating the task data table, and the data processing efficiency of the data processing task is improved.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of: scanning data processing tasks in an operating state; inquiring a task data table on which the data processing task depends; creating a temporary data table corresponding to the queried task data table; directing a data access address in a data processing task to a temporary data table; performing data migration on the task data table; and when the task data table is migrated, the data access address in the data processing task is pointed to the migrated task data table.
In one embodiment, the computer program, when executed by the processor, further performs the steps of: acquiring a data migration instruction; extracting an migrating database address and an migrating database address in a data migration instruction; counting the stored space capacity corresponding to the migrated database address; determining whether the residual space capacity corresponding to the migrated database address is larger than the stored space capacity; and when the residual space capacity is larger than the stored space capacity, scanning the data processing task in the running state.
In one embodiment, scanning data processing tasks in an operational state includes: scanning task identifiers corresponding to the data processing tasks respectively; determining a working state corresponding to the task identifier; and screening the data processing task corresponding to the task identifier with the working state being the running state.
In one embodiment, performing data migration on a task data table includes: creating a data migration task according to the task data table; adding the created data migration tasks into a preset number of migration task queues; reading data migration tasks in each migration task queue in parallel; and carrying out data migration on the task data table corresponding to the read data migration task.
In one embodiment, after data migration of the task data table, the computer program when executed by the processor further performs the steps of: recording migration process data of a task data table; generating a data migration report according to the recorded migration process data; and sending the data migration report to a terminal logged in by an administrator account.
In one embodiment, after the task data table is migrated, the computer program when executed by the processor further performs the steps of: obtaining access right information corresponding to a task data table; determining the transferred data table address corresponding to the task data table; and configuring access rights to the migrated data table address according to the access rights information.
In this embodiment, a task data table on which a data processing task in an operating state depends is queried, a temporary data table corresponding to the task data table is created, and a data access address in the data processing task in the operating state is pointed to the temporary data table, so that the data processing task can normally operate when the task data table is migrated. When the task data table is migrated, the data access address in the data processing task is pointed to the migrated task data table, and meanwhile, when the task data table is migrated, the data processing task can rely on the migrated task data table to continue to run, so that the data processing task does not interrupt execution in the process of migrating the task data table, and the data processing efficiency of the data processing task is improved.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SYNCHLINK) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (10)
1. A method of data migration, the method comprising:
Acquiring a data migration instruction;
Extracting an migrating database address and an migrating database address in a data migration instruction;
counting the stored space capacity corresponding to the migrated database address;
Determining whether the residual space capacity corresponding to the migrated database address is greater than the stored space capacity;
when the residual space capacity is larger than the stored space capacity, scanning the data processing task in an operation state;
inquiring a task data table on which the data processing task depends;
creating a temporary data table corresponding to the queried task data table;
Storing the data access address of the temporary data table and the data access address of the corresponding task data table correspondingly;
Analyzing the data access address in the data processing task, and inquiring the data access address of the temporary data table corresponding to the data access address in the data processing task;
modifying the data access address in the data processing task into the data access address corresponding to the queried temporary data table;
creating data migration tasks corresponding to the task data tables according to the data table identifiers of the task data tables, and obtaining data migration tasks corresponding to the task data tables;
adding the created data migration tasks to a preset number of migration task queues;
reading data migration tasks in each migration task queue in parallel;
Analyzing the data table identifier in the read data migration task, inquiring the task data table and the migration database address corresponding to the data table identifier in the data migration task, and storing the inquired task data table into the database corresponding to the migration database address;
And when the task data table is migrated, pointing the data access address in the data processing task to the migrated task data table.
2. The method of claim 1, wherein the directing the data access address in the data processing task to the migrated task data table comprises:
Acquiring a migration access address corresponding to the migrated task data table;
And modifying the data access address in the data processing task into the migration access address.
3. The method of claim 1, wherein the scanning data processing tasks in an operational state comprises:
scanning task identifiers corresponding to the data processing tasks respectively;
determining a working state corresponding to the task identifier;
And screening the data processing task corresponding to the task identifier with the working state being the running state.
4. The method of claim 1, wherein after the data migration to the task data table, further comprising:
Recording migration process data of the task data table;
generating a data migration report according to the recorded migration process data;
and sending the data migration report to a terminal logged in by an administrator account.
5. The method of claim 4, wherein the migration process data comprises at least one of a migration status, a migration start time, a migration end time, and a time period spent migrating of the task data table.
6. The method according to any one of claims 1-5, wherein after the data access address in the data processing task is pointed to the migrated task data table when the task data table is migrated, further comprising:
Obtaining access right information corresponding to the task data table;
Determining the data table address after migration corresponding to the task data table;
and configuring access rights to the migrated data table address according to the access rights information.
7. A data migration apparatus, the apparatus comprising:
The capacity confirmation module is used for acquiring a data migration instruction; extracting an migrating database address and an migrating database address in a data migration instruction; counting the stored space capacity corresponding to the migrated database address; determining whether the residual space capacity corresponding to the migrated database address is greater than the stored space capacity;
The task scanning module is used for scanning the data processing task in the running state when the residual space capacity is larger than the stored space capacity;
the data table query module is used for querying a task data table on which the data processing task depends;
The data table creating module is used for creating a temporary data table corresponding to the inquired task data table;
the pointing adjustment module is used for correspondingly storing the data access address of the temporary data table and the data access address of the corresponding task data table; analyzing the data access address in the data processing task, and inquiring the data access address of the temporary data table corresponding to the data access address in the data processing task; modifying the data access address in the data processing task into the data access address corresponding to the queried temporary data table;
The data migration module is used for creating data migration tasks corresponding to the task data tables according to the data table identifiers of the task data tables to obtain data migration tasks corresponding to the task data tables; adding the created data migration tasks to a preset number of migration task queues; reading data migration tasks in each migration task queue in parallel; analyzing the data table identifier in the read data migration task, inquiring the task data table and the migration database address corresponding to the data table identifier in the data migration task, and storing the inquired task data table into the database corresponding to the migration database address;
And the migration monitoring module is used for pointing the data access address in the data processing task to the migrated task data table when the task data table is migrated.
8. The apparatus of claim 7, wherein the migration snoop module is further configured to:
Acquiring a migration access address corresponding to the migrated task data table;
And modifying the data access address in the data processing task into the migration access address.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910418940.5A CN110245125B (en) | 2019-05-20 | 2019-05-20 | Data migration method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910418940.5A CN110245125B (en) | 2019-05-20 | 2019-05-20 | Data migration method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110245125A CN110245125A (en) | 2019-09-17 |
| CN110245125B true CN110245125B (en) | 2024-11-01 |
Family
ID=67884488
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910418940.5A Active CN110245125B (en) | 2019-05-20 | 2019-05-20 | Data migration method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110245125B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113297145B (en) * | 2020-02-24 | 2023-12-22 | 阿里巴巴集团控股有限公司 | Migration report generation method and device, electronic equipment and computer storage medium |
| CN111694517B (en) * | 2020-05-28 | 2024-02-13 | 新浪技术（中国）有限公司 | Distributed data migration method, system and electronic equipment |
| CN111930715B (en) * | 2020-07-16 | 2024-09-13 | 北京金山云网络技术有限公司 | Data migration method, device, computer equipment and storage medium |
| CN112328573A (en) * | 2020-11-03 | 2021-02-05 | 杭州朗澈科技有限公司 | Database migration method and system in kubernets cluster |
| CN114265828A (en) * | 2021-12-20 | 2022-04-01 | 平安证券股份有限公司 | Line migration elimination method and device, computer equipment and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106534226A (en) * | 2015-09-09 | 2017-03-22 | 阿里巴巴集团控股有限公司 | Processing method and apparatus for data migration |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7634512B2 (en) * | 2006-10-20 | 2009-12-15 | Oracle International Corporation | Migrating temporary data of a session |
| CN106909595B (en) * | 2016-06-20 | 2020-12-29 | 创新先进技术有限公司 | Data migration method and device |
| CN108268501B (en) * | 2016-12-30 | 2020-09-18 | 中国移动通信集团北京有限公司 | Service processing method and device in online data migration process |
| CN108509556B (en) * | 2018-03-22 | 2021-03-23 | 上海达梦数据库有限公司 | Data migration method and device, server and storage medium |
-
2019
- 2019-05-20 CN CN201910418940.5A patent/CN110245125B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106534226A (en) * | 2015-09-09 | 2017-03-22 | 阿里巴巴集团控股有限公司 | Processing method and apparatus for data migration |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110245125A (en) | 2019-09-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110245125B (en) | Data migration method, device, computer equipment and storage medium | |
| CN110209652B (en) | Data table migration method, device, computer equipment and storage medium | |
| CN110008118B (en) | Page data testing method and device, computer equipment and storage medium | |
| CN110290212B (en) | Service call recording method, device, computer equipment and storage medium | |
| CN109064345B (en) | Message processing method, system and computer readable storage medium | |
| CN109446068B (en) | Interface test method, device, computer equipment and storage medium | |
| CN110245505B (en) | Data table access method, device, computer equipment and storage medium | |
| CN109474578B (en) | Message checking method, device, computer equipment and storage medium | |
| CN108197200B (en) | Log tracking method and device, computer equipment and storage medium | |
| CN108491450B (en) | Data caching method, device, server and storage medium | |
| CN109814854B (en) | Project framework generation method, device, computer equipment and storage medium | |
| CN110231937B (en) | Script defect scanning method, script defect scanning device, computer equipment and storage medium | |
| CN111078733B (en) | Batch task processing method, device, computer equipment and storage medium | |
| CN109389299B (en) | Workflow process deployment method, workflow process deployment device, computer equipment and storage medium | |
| CN111078559B (en) | Method, device, medium and computer equipment for extracting function call in java code | |
| CN112115167B (en) | Cache system hot spot data access method, device, equipment and storage medium | |
| CN110704095A (en) | Workflow-based configuration processing method and device, computer equipment and storage medium | |
| CN113159737B (en) | RPA service processing method, RPA management platform, device and medium | |
| CN114531477B (en) | Method and device for configuring functional components, computer equipment and storage medium | |
| CN110727698A (en) | Database access method and device, computer equipment and storage medium | |
| CN110795171B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN111190609B (en) | Service processing method, device, equipment and storage medium | |
| CN109542764B (en) | Webpage automatic testing method and device, computer equipment and storage medium | |
| CN113254399A (en) | Log query method and device, electronic equipment and medium | |
| CN111224939B (en) | Task request intercepting method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |