WO2023115935A1 - Data processing method, and related apparatus and device - Google Patents

Abstract

Disclosed in the embodiments of the present application are a data processing method, and a related apparatus and a device, which are used for improving the performance of a system after data recovery. The method in the embodiments of the present application comprises: acquiring a position information set of a row data set, wherein the row data set comprises hot data, the position information set comprises first position information corresponding to the hot data, and the frequency of access to the hot data is higher than the frequency of access to other data in the row data set; acquiring a data recovery instruction, wherein same is used for indicating the recovery of the data in the row data set; and in response to the data recovery instruction, loading the hot data into a memory according to the first position information.

Description

Data processing method, related device and equipment

This application claims priority to a Chinese patent application with application number 202111582635.3 and titled "Data Processing Method, Related Devices and Equipment" filed with the State Intellectual Property Office of China on December 22, 2021, the entire contents of which are hereby incorporated by reference In this application.

technical field

The embodiments of the present application relate to the field of computers, and in particular, to data processing methods, related devices and equipment.

Background technique

An in-memory database is a database that stores all data directly in memory. Compared with a solution that stores data in disk, it can significantly improve access speed. However, due to the volatility of the storage medium itself, the memory data will be lost after power failure or database restart. How to recover data has become an urgent problem to be solved.

In a data processing method, when a checkpoint operation is performed, the location information of row data in the log "copy" is backed up, and when data is restored, the location information of all row data in the database is fully restored based on the log. When the system is running, based on these location information, the row data is loaded at runtime.

In this method, when data is restored, the location information of the row data is restored, and the row data is loaded only when the system is running, resulting in low performance of the system after the data is restored.

Contents of the invention

The embodiment of the present application provides a data processing method, related device and equipment, which defines row data with high access frequency as hot data, and backs up location information of all row data when a checkpoint operation is performed. When data is restored, the hot data is loaded into the memory according to the first location information of the hot data, so that the hot data does not need to be loaded at runtime. Combined with the Zif distribution, it can be seen that most of the business is based on access to a small amount of data (hot data). Therefore, the data restoration method provided by the embodiment of the present application improves the performance of the system after data restoration by ensuring the performance of services accessing hot data.

The first aspect of the embodiment of the present application provides a data processing method, including:

In the memory database, at least one row of data is stored, and the at least one row of data is included in the row data set. When performing a checkpoint operation, the network device obtains the location information set corresponding to the row data set, the location information in the location information set corresponds to the row data in the row data set one by one, and the location information indicates the location of the row data in the log. The row data set includes hot data, the access frequency of the hot data is higher than that of other data in the row data set, and the location information set includes first location information corresponding to the row data. In the case of a fault in the memory database, the first network device can acquire a data recovery instruction, where the data recovery instruction is used to instruct to restore the data in the row data set. The network device will respond to the data recovery instruction, and load the hot data into the memory according to the first location information. After the data is restored, the hot data can be directly accessed in the memory.

It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

Define row data with high access frequency as hot data, and back up the location information of all row data when performing a checkpoint operation. When data is restored, the hot data is loaded into the memory according to the first location information of the hot data, so that the hot data does not need to be loaded at runtime. Combined with the Zipf distribution, it can be seen that most of the business is based on accessing a small amount of data (hot data). Therefore, the data restoration method provided by the embodiment of the present application improves the performance of the system after data restoration by ensuring the performance of services accessing hot data.

In some optional embodiments of the first aspect, before acquiring the location information set of the row data set, the method further includes: recording the popularity information of each row data in the row data set, where the popularity information is used to indicate that each row Data access frequency.

In the embodiment of the present application, the access frequency of the row data is reflected by recording the heat information, which provides a basis for determining the hot data in the row data set, and improves the feasibility of the technical solution.

In some optional embodiments of the first aspect, the popularity information includes the number of visits, recording the popularity information of each row data in the row data set, including: obtaining the first data access request, the first data access request indicating access to the row data The first target row of data in the collection. Then, according to the first data access request, the number of accesses corresponding to the first target row data is increased. In the embodiment of the present application, there are many ways to record the number of visits. You can set a frequency counter, an expiration time, and/or a timer to record the number of visits, which is not limited here.

In the embodiment of the present application, multiple ways of obtaining popularity information are provided, and in practical applications, they can be flexibly selected, which improves the flexibility and practicability of the technical solution of the present application.

In some optional embodiments of the first aspect, the method further includes: the network device may further provide a temperature threshold configuration interface, and obtain the temperature threshold input by the user through the temperature threshold configuration interface. The popularity threshold may indicate the proportion of hot data in the row data set, or indicate the minimum value of access times corresponding to the hot data, which is not limited here. In some optional embodiments, the temperature threshold can also be set according to the type of application, or set by the R&D/testing personnel, which is not specifically limited here. The network device can determine hot data from the row data set according to the hotness information and hotness threshold.

In the embodiment of the present application, the popularity threshold can be set by the user, which enhances the user experience. In addition, there are many ways to define the heat threshold, which can be selected according to the needs of practical applications, which further improves the flexibility of the technical solution of the present application.

In some optional embodiments of the first aspect, the other data includes cold data, and the location information set further includes second location information corresponding to the cold data. After the data is restored, that is, after the hot data is loaded into the memory, the present application can also perform a data access process. Specifically, the network device may obtain a second data access request from the terminal device, and the second data access request indicates to access the second target row data in the row data set. According to different data types corresponding to the second target row data, the second target row data is accessed based on different methods. If the second target row data is hot data, access the hot data in the memory; if the second target row data is cold data, load the cold data according to the second location information. Wherein, accessing the second target row data may include performing other operations on the second target row data in addition to reading or writing the second target row data, for example, modifying the content of the second target row data, etc. , not limited here.

In the embodiment of this application, the row data set in the memory database is divided into hot data and cold data, and different data are processed differently during the data recovery process. Hot data can be directly accessed in memory, ensuring good performance of the system. Cold data with low access frequency is backed up as location information. When accessing cold data, it is loaded at runtime according to the location information, which reduces the overall data recovery time of the memory database.

The second aspect of the embodiment of the present application provides a data processing method, the method is applied to a first network device, the first network device is included in a data processing system, and the data processing system further includes a second network device, the method includes:

Acquiring hot checkpoint information, the hot checkpoint information corresponds to the hot data in the row data set, and the access frequency of the hot data is higher than the access frequency of other data in the row data set; and sending the hot checkpoint information to the second network device to After the first network device fails, hot data is accessed based on the second network device.

In the embodiment of the present application, the first network device synchronizes the hot checkpoint information with the second network device instead of log information, so that the synchronization speed of the first network device and the second network device match, which is more in line with the needs of practical applications. It also improves the practicability of the technical solution of the present application.

In some optional embodiments of the second aspect, the hot checkpoint information includes hot data, so that the second network device stores the hot data in memory after receiving the hot data.

In some optional embodiments of the second aspect, the hot checkpoint information includes location information corresponding to the hot data, and the location information is used to indicate the location of the hot data in the log, so that after the second network device receives the location information , the hot data will be loaded into the memory according to the location information.

In the embodiment of the present application, there are various types of hot checkpoint information, which can be determined according to actual application requirements, which improves the flexibility of the technical solution.

In some optional embodiments of the second aspect, if the first network device is not faulty, the first network device may further execute the method of the foregoing first aspect. Relevant beneficial effects are similar to those of the first aspect, see the first aspect for details, and will not be repeated here.

The third aspect of the embodiment of the present application provides a data processing method, the method is applied to a data processing system, the data processing system includes a first network device and a second network device, the method includes:

The first network device acquires hot checkpoint information, where the hot checkpoint information corresponds to hot data in the row data set, and the access frequency of the hot data is higher than that of other data in the row data set. After acquiring the hot checkpoint information, the first network device sends the hot checkpoint information to the second network device. The second network device receives the checkpoint information, and loads hot data in the memory according to the hot checkpoint information. In case of failure of the first network device, the second network device will receive a data access request, and the data access request indicates access to hot data; and the device responds to the data access request, and accesses the hot data in memory.

In some optional embodiments of the third aspect, the hot checkpoint information includes hot data or location information corresponding to the hot data, where the location information is used to indicate a location of the hot data in the log. If the hot checkpoint information is hot data, the second network device will store the hot data in memory after receiving the hot data; if the hot checkpoint information is the location information corresponding to the hot data, the second network device will receive the location information The hot data is then loaded into memory based on this location information.

In some optional embodiments of the third aspect, if the first network device is not faulty, the first network device may further execute the method of the foregoing first aspect. In the case of a failure of the first network device, the second network device may also execute the method in the first aspect described above, which will not be repeated here. Relevant beneficial effects are similar to those of the first aspect, see the first aspect for details, and will not be repeated here.

In the embodiment of the present application, the first network device synchronizes the hot checkpoint information with the second network device instead of log information, so that the synchronization speed of the first network device and the second network device match, which is more in line with the needs of practical applications. It also improves the practicability of the technical solution of the present application.

The fourth aspect of the embodiment of the present application provides a network device, including:

An acquisition unit, configured to acquire a location information set of a row data set, the row data set includes hot data, the location information set includes first location information corresponding to the hot data, and the access frequency of the hot data is higher than that of other data in the row data set .

The obtaining unit is also used to obtain a data restoration instruction, and the data restoration instruction is used to instruct to restore the data in the row data set.

The processing unit is configured to load the hot data into the memory according to the first location information in response to the data recovery instruction.

The network device is configured to execute the method in the first aspect, and the beneficial effect shown in this aspect is similar to the beneficial effect in the first aspect, which will not be repeated here.

The fifth aspect of the embodiment of the present application provides a first network device. The first network device is included in a data processing system. The data processing system further includes a second network device. The first network device includes:

The acquiring unit is configured to acquire hot checkpoint information, the hot checkpoint information corresponds to the hot data in the row data set, and the access frequency of the hot data is higher than that of other data in the row data set.

The sending unit is configured to send hot checkpoint information to the second network device, so that the hot data can be accessed based on the second network device after the first network device fails.

The first network device is configured to execute the method in the second aspect, and the beneficial effect shown in this aspect is similar to the beneficial effect in the second aspect, which will not be repeated here.

A sixth aspect of the embodiments of the present application provides a data processing system, including a first network device and a second network device.

The first network device is used to obtain hot checkpoint information, the hot checkpoint information corresponds to the hot data in the row data set, and the access frequency of the hot data is higher than the access frequency of other data in the row data set; send to the second network device Hot checkpoint information.

The second network device is configured to load hot data in the memory according to the hot checkpoint information; if the first network device fails, the second network device receives a data access request, and the data access request indicates access to hot data; in response to the data access request, the Access hot data in memory.

The data processing system is used to execute the method of the third aspect, and the beneficial effect shown in this aspect is similar to the beneficial effect of the third aspect, which will not be repeated here.

The seventh aspect of the embodiment of the present application provides a network device, including a processor, a memory and a communication interface, the processor, the memory and the communication interface are connected, and the processor is used to execute the method in any one of the first to third aspects. .

The eighth aspect of the embodiment of the present application provides a computer-readable storage medium, in which a program is stored, and when the computer executes the program, any one of the aforementioned first to third aspects is executed Methods.

A ninth aspect of the embodiment of the present application provides a computer program product, which is characterized in that, when the computer program product is executed on a computer, the computer executes the method of any one of the foregoing first aspect to the third aspect.

The beneficial effects shown in the sixth aspect to the ninth aspect are similar to those shown in the first aspect to the third aspect, and will not be repeated here.

Description of drawings

Figure 1 is a schematic diagram of an in-memory database;

Fig. 2 is a schematic diagram of an application architecture of the data processing method;

FIG. 3 is a schematic flow chart of a data processing method provided in an embodiment of the present application;

Fig. 4a is another schematic flowchart of the data processing method provided by the embodiment of the present application;

Figure 4b is a schematic diagram of the data form provided by the embodiment of the present application;

FIG. 5 is another schematic flowchart of the data processing method provided by the embodiment of the present application;

FIG. 6 is another schematic flowchart of the data processing method provided in the embodiment of the present application;

Fig. 7a is a schematic diagram of the threshold setting interface provided by the embodiment of the present application;

Fig. 7b is another schematic diagram of the threshold setting interface provided by the embodiment of the present application;

FIG. 8 is another schematic flowchart of the data processing method provided in the embodiment of the present application;

FIG. 9 is a schematic structural diagram of a data processing system provided by an embodiment of the present application;

FIG. 10 is another schematic flowchart of the data processing method provided by the embodiment of the present application;

FIG. 11 is another schematic flowchart of the data processing method provided by the embodiment of the present application;

FIG. 12 is another schematic flowchart of the data processing method provided by the embodiment of the present application;

FIG. 13 is a schematic structural diagram of a network device provided by an embodiment of the present application;

FIG. 14 is a schematic structural diagram of a first network device provided by an embodiment of the present application;

FIG. 15 is another schematic structural diagram of the data processing system provided by the embodiment of the present application;

FIG. 16 is another schematic structural diagram of a network device provided by an embodiment of the present application.

Detailed ways

The embodiment of the present application provides a data processing method, related devices and equipment, which defines row data with high access frequency as hot data, and backs up location information of all row data when a checkpoint operation is performed. When data is restored, the hot data is loaded into the memory according to the first location information of the hot data, so that the hot data does not need to be loaded at runtime. Combined with the Zif distribution, it can be seen that most of the business is realized based on accessing a small amount of data (hot data). Therefore, the data restoration method provided by the embodiment of the present application improves the performance of the system after data restoration by ensuring the performance of services accessing hot data.

Embodiments of the present application are described below in conjunction with the accompanying drawings. Those of ordinary skill in the art know that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

The terms "first", "second" and the like in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is merely a description of the manner in which objects with the same attribute are described in the embodiments of the present application. Furthermore, the terms "comprising" and "having", and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product, or apparatus comprising a series of elements is not necessarily limited to those elements, but may include without other elements explicitly listed or inherent to the process, method, product, or apparatus. In addition, "at least one" means one or more, and "plurality" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B, which can mean: A exists alone, A and B exist at the same time, and B exists alone, where A, B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one item (piece) of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be single or multiple .

First, related concepts that may be involved in the embodiments of the present application are explained.

1. In-memory database.

An in-memory database is a database that stores data in memory and directly operates it. The significant advantage of an in-memory database is that the access speed of data in memory is greatly improved compared with that on disk, which can improve application performance. The data structure of the in-memory database will be described below in conjunction with FIG. 1 . Please refer to FIG. 1 , which is a schematic diagram of the in-memory database.

Generally, the data organization of an in-memory database is shown in Figure 1, which mainly includes row data and indexes.

The index can be used to quickly locate the row ID (row ID) where the row data is located, and the index can adopt tree structures such as adaptive radix tree (ART), MassTree, and bwTree. Among them, the ART tree may also be called a prefix tree. It should be noted that in practical applications, the index may also be based on other types of tree structures, or not adopt tree structures, which are not specifically limited here. In the embodiment shown in Figure 1, the index includes a primary index (primary index) and a secondary index (secondary index), the primary index corresponds to the unique identifier in the row data, and can accurately index the unique row data; the secondary index can It is understood as an auxiliary identifier for rough retrieval.

Each row of data is assigned a unique row ID, and the row ID is indexed by the row map. In order to support high-concurrency data processing capabilities, multi-version concurrency control (MVCC) is usually used to access data, so that different versions of a row of data can be accessed, and each row of data can be accessed at the current moment. version of. After data modification and other operations, the same row of data will include multiple record (record) versions, and multiple versions of records form a version chain (version chain), and each record in the version chain is generally arranged in order from new to old .

The storage layer (storage) stores logs in the log store, which is the unit for collecting, storing and querying log data in the log service. The storage layer also includes checkpoint information. In the embodiment of the present application, the checkpoint information includes the location information set corresponding to the row data set. The location information in the location information set corresponds to the row data in the row data set. The location information Reflects the position of row data in the log. When the memory database fails, data recovery can be performed through the checkpoint information.

2. Zif distribution.

The Ziff distribution is also known as the 80/20 rule. Simply put, in general, the access data of the application complies with the 80/20 rule, and 80% of the business is realized by accessing 20% of the business. Therefore, in the embodiment of the present application, the hot data with high access frequency is recorded in the memory in advance, which can ensure the performance requirements of most access requests after the data is restored.

Next, please refer to FIG. 2 . FIG. 2 is a schematic diagram of an application architecture of the data processing method provided by the embodiment of the present application. As shown in FIG. 2 , a communication connection is established between a network device 201 and a terminal device 202 , and a memory database exists in the network device 201 . The terminal device 202 may send a data access request to the network device 201, where the data access request indicates to access data in the memory database. Due to the volatility of the memory medium itself, after the memory database fails (such as power failure or database restart), the data in the memory database will be completely lost, and it is necessary to restore the data in time to reduce adverse effects.

It should be noted that FIG. 1 is only an example of the application architecture. In practical applications, besides the server shown in FIG. 1, the network device 201 may also be other media with storage functions, such as a host. There is no limit. The terminal device 202 involved in the embodiment of the present application may include various handheld devices with communication functions, vehicle-mounted devices, wearable devices, computing devices or other processing devices connected to a wireless modem. The terminal device can also be called a terminal, and the terminal device can also be a subscriber unit, a cellular phone, a smart phone, a wireless data card, a personal digital assistant (PDA) ) computer, tablet computer, wireless modem (modem), handheld device (handset), laptop computer (laptop computer), machine type communication (machine type communication, MTC) terminal, etc., are not limited here.

In this embodiment of the application, the network device 201 can also provide services to more terminal devices 202, and the number and types of terminal devices are determined according to actual needs, which are not specifically limited here.

Next, please refer to FIG. 3. FIG. 3 is a schematic flow chart of the data processing method provided by the embodiment of the present application, including the following steps:

301. Acquire the location information set of the row data set, the row data set includes hot data, the location information set includes first location information corresponding to the hot data, and the access frequency of the hot data is higher than that of other data in the row data set.

In the memory database, a row data set is stored, and the row data set includes at least one row data. When the network device performs a checkpoint operation, it scans the location information set corresponding to the row data set, and the location information set indicates the location information of the row data set in the log. Specifically, the position information in the position information set is in one-to-one correspondence with the row data in the row data set.

The process of performing the checkpoint operation in the embodiment of the present application will be described below in conjunction with FIG. 4a. FIG. 4a is a schematic flow chart of the data processing method provided in the embodiment of the present application, including the following steps:

401. A checkpoint thread triggers a checkpoint operation.

During the operation of the memory database, the background will periodically trigger checkpoint requests according to certain policies. The data backup corresponding to the checkpoint operation can be divided into full backup and incremental backup. In this embodiment of the application, full backup refers to backing up the positions of all data in the memory data in the log, and incremental backup refers to backing up the positions of the changed data in the memory database in the log. The strategy for triggering a checkpoint request may include not only the cycle time of the checkpoint operation interval, but also the occurrence frequency of full backup and incremental backup, which is not limited here. For example, the policy may stipulate that four incremental backups are performed after one full backup, and the checkpoint operation is performed reciprocally at a fixed time interval (such as 20 minutes) between each backup. It should be noted that the specific content of the policy that triggers the checkpoint request is determined according to the actual application, and is not limited here.

402. The in-memory database obtains the timestamp corresponding to the checkpoint.

Because there are multiple versions of data stored in the memory database, when performing a checkpoint operation, it is necessary to determine the version of the data corresponding to the current moment to avoid problems caused by repeated backups or inaccurate backups. Therefore, the in-memory database gets the timestamp corresponding to the checkpoint.

403. The memory database scans the snapshot data based on the timestamp, and saves the location information corresponding to the snapshot data.

The in-memory database scans the snapshot version of the data based on the timestamp, that is, scans the snapshot data. The so-called snapshot data refers to a copy of the data at a certain moment. At the same time, the memory database will also store the corresponding location information of the snapshot data in the disk, and the location information indicates the location of the row data in the log. Among them, the disk can be a local disk of the network device, or other media with functions such as data reading and writing, such as a disk on a cloud network (virtual machine). The type and location of the disk are selected according to the needs of the actual application. There is no limit here.

404. The memory database saves the heat information corresponding to the row data to disk.

The in-memory database also stores the heat information corresponding to the row data to the disk, and the heat information is used to indicate the access frequency of the row data and to determine the hot data. Specifically, the row data set includes hot data, and the access frequency of the hot data is higher than other data in the row data set. The location information set includes first location information corresponding to the thermal data, and the first location information indicates the location of the thermal data in the log.

When saving to disk, they can be placed in descending order of access frequency, or they can be saved in other order, for example, by line number, which is not limited here.

In some optional embodiments, after step 404, the network device also executes step 405, that is, the memory database feedbacks that the checkpoint operation is completed.

In the embodiment of the present application, after the checkpoint operation, there are many possible forms of data stored in the disk, which will be described in conjunction with FIG. 4b below. Please refer to FIG. 4b. FIG. 4b is a schematic diagram of the data form provided by the embodiment of the present application.

As shown in Figure 4b, when a network device saves to disk, it can be saved in the form of data form 1, and heat information is attached to the checkpoint file. The checkpoint file includes the location information set corresponding to the row data, and the heat information reflects the row data set. The access frequency of each row of data in is used to determine hot data. The network device can also save to disk according to data form 2, storing the first location information corresponding to the hot data in the hot data file, and storing the second location information corresponding to the cold data in the cold data file. Wherein, how the network device determines hot data from the row data set and determines other data as cold data is described in detail in the embodiments shown in FIG. 5 to FIG. 7b below, see this part for details.

302. Obtain a data restoration instruction, where the data restoration instruction is used to instruct to restore the data in the row data set;

After the in-memory database fails, the in-memory database will receive a data recovery instruction, which instructs to restore data in the row data set.

303. In response to the data recovery instruction, load the hot data into the memory according to the first location information.

The network device will respond to the data recovery instruction, and preload the hot data into the memory according to the first location information, so that when accessing the hot data, it can be directly accessed from the memory.

Step 302 and step 303 will be described in detail below in conjunction with FIG. 5. Please refer to FIG. 5. FIG. 5 is a schematic flow chart of the data processing method provided by the embodiment of the present application, including the following steps:

501. The data recovery thread triggers a data recovery operation.

After the in-memory database fails, an alarm can be issued to prompt technicians to perform data recovery, and the technicians control the data recovery thread to trigger data recovery operations. In practical applications, there are other ways to trigger the data recovery operation, for example, start the timer when the memory database fails, and automatically trigger the data recovery operation when the timer expires and no data recovery command from the user is received. The situation that triggers the data recovery operation is determined according to the actual application, which is not limited here.

502. The memory database loads the checkpoint data.

After the memory database receives the data recovery instruction, it will load the checkpoint data. Specifically, the in-memory database will read the checkpoint data of the full backup and the checkpoint data of the incremental backup after the full backup from the storage layer at the closest time point to the time when the failure occurred. In the embodiment of the present application, since the location information corresponding to the row data is backed up when the checkpoint operation is performed, the loaded checkpoint data is the location information corresponding to the row data.

503. The memory database restores the corresponding offset position of the row data.

After the checkpoint data is loaded, the in-memory database will completely restore the location information of all row data in the in-memory database based on the log library, that is, restore the offset position of the row data in the log to the memory.

504. The memory database sets the global clock.

The data processing process involved in the embodiment of the present application depends on the time stamp. In machine language, the time stamp can be run as a counter. Therefore, in order to prevent errors in the recovered data, a global clock will be set so that the recovered data is located in the correct Location.

505. The memory database preloads hot data.

The in-memory database will preload the hot data in the row data set into the memory, so that the restored in-memory database has better performance.

In some optional embodiments, after step 505, the network device may also execute step 506, that is, return that the data recovery operation is completed.

In the embodiment of the present application, row data with a high access frequency is defined as hot data, and position information of all row data is backed up when a checkpoint operation is performed. When data is restored, the hot data is loaded into the memory according to the first location information of the hot data, so that the hot data does not need to be loaded at runtime. Combined with the Zif distribution, it can be seen that most of the business is based on access to a small amount of data (hot data). Therefore, the data restoration method provided by the embodiment of the present application improves the performance of the system after data restoration by ensuring the performance of services accessing hot data.

In some optional embodiments, before step 301, the network device also records popularity information of each row data in the row data set, where the popularity information is used to indicate the access frequency of each row data.

In the embodiment of the present application, the access frequency of the row data is reflected by recording the heat information, which provides a basis for determining the hot data in the row data set, and improves the feasibility of the technical solution.

In some optional embodiments, the popularity information includes access times, and when the row data is accessed, the network device will modify the popularity information corresponding to the row data. That is to say, recording the popularity information of each row data in the row data set includes: obtaining a first data access request, where the first data access request indicates to access the first target row data in the row data set. Then, according to the first data access request, the number of accesses corresponding to the first target row data is increased.

This process will be described below in conjunction with FIG. 6. Please refer to FIG. 6. FIG. 6 is a schematic flow chart of the data processing method provided by the embodiment of the present application, including the following steps:

601. The network device acquires a first data access request from the terminal device.

The network device will obtain a first data access request from the terminal device, where the first data access request indicates to access the first target row data in the row data set. Wherein, the first data access request may carry the row number corresponding to the first row of data, or the unique index corresponding to the first target row data, so that the network device can determine whether the accessed row data is The first target row data. Wherein, the unique index is the main index in the embodiment shown in FIG. 1. Exemplarily, it is assumed that the memory database stores information about at least one product, including product name, unit price, inventory and other information. The product names of different products are different. The same, then the product name of each product can be used as a unique index. Accessing the first target row data may include performing other operations on the first target row data in addition to reading or writing the first target row data, for example, modifying the content of the first target row data, etc., specifically There is no limit here.

602. The network device acquires a time stamp corresponding to the first data access request.

The first data access request can also carry the timestamp corresponding to the request, and the timestamp is used to indicate the row data at which time the first target row data is accessed, so as to avoid multiple versions of the row data in the memory database. resulting in access errors.

603. Read the first target row data based on the concurrency control protocol.

In general, access to data in the in-memory database is based on the concurrency control protocol, and the type of the concurrency control protocol is determined according to the type of the in-memory database, which is not limited here.

604. Modify the popularity information corresponding to the first target row data.

When the first target row data is accessed, the network device will modify the popularity information corresponding to the first target row data, specifically, it can increase the number of visits to the first target row data. There are many possible situations how to increase the number of visits, and the possible situations are described below.

1) Record the number of visits based on a frequency counter.

In some optional embodiments, the network device may separately maintain a frequency counter accessCount for each row of data, and its initial value is 0. When the data of the first target row is accessed once, the corresponding accessCount is added with n, that is, the accessCount++ operation is performed once. Wherein, n is any positive number.

Optionally, the accessCount can be changed periodically according to the time interval of executing the checkpoint operation. For example, assuming that the checkpoint operation is performed every 20 minutes, then at the beginning of each cycle, the initial value of accessCount is 0, and the accessCount++ operation is executed according to the number of accesses within the cycle.

Optionally, accessCount may not change periodically. For example, when the row data set is stored in the memory database for the first time, the initial value of accessCount corresponding to each row data is 0, and which row of data is accessed, the accessCount++ operation is performed on the accessCount corresponding to the row data.

It can be understood that, in practical applications, there are more possibilities for counting the accessCount, as long as it can reflect the number of visits to the travel data, which is not limited here.

2) Record the number of visits according to the expiration time.

In some optional embodiments, the network device may also set an expiration time for each row of data, and the duration of the expiration time may be shorter than the interval between checkpoint operations. Before the expiration time corresponding to a row of data expires, if the row of data is accessed, the number of accesses to the row of data will be increased; if the row of data is not accessed within the corresponding expiration time of a row of data, the row will be considered The number of visits corresponding to the data remains unchanged.

3) Record the number of visits according to the frequency counter and expiration time.

In some optional embodiments, the network device may also record the number of visits in combination with the frequency counter and the expiration time. Specifically, the network device sets an expiration time and a frequency counter for each row of data. If the row of data is not accessed within the expiration time, the value of the frequency counter corresponding to the row of data remains unchanged and is refreshed. The invalid expiration time is used to continue counting; if the row data is accessed before the expiration time expires, then the frequency counter corresponding to the row data executes an accessCount++ operation, and the expiration time is refreshed when the row data is accessed. Among them, the meaning of the accessCount++ operation has been explained above and will not be explained here.

Exemplarily, assuming that the expiration time corresponding to a row of data is 30 seconds (s), and the row of data is not accessed within 30s, the value of accessCount corresponding to the row of data remains unchanged, and the expiration time is refreshed. 31s to enter a new round of expiration time. If the row of data is accessed at the 15th second, the frequency counter corresponding to the row of data executes an accessCount++ operation, and refreshes the expiration time, and enters a new round of expiration time from the 16th second.

Optionally, similar to "1) Recording the number of accesses based on the frequency counter", the accessCount can be changed periodically according to the time interval of executing the checkpoint operation, or not. It has been described in detail above, The details will not be repeated here.

It should be noted that the above three methods are only some possibilities for recording the number of visits. In practical applications, the number of visits can also be recorded based on similar principles. For example, setting a timer, if within the time set by the timer, the first target If the row data is accessed, it is determined that the first target row data is hot data; if the first target row data is not accessed within the time set by the timer, then it is determined that the first target row data is cold data. The specific process of recording the number of visits is determined according to the needs of the actual application, and is not specifically limited here.

In some optional embodiments, after step 604, step 605 may also be executed by the network device to report the completion of the data access, where there are multiple ways for the network device to report the completion of the data access: if the first data access request indicates the read the first target row data, then the network device can send the first target row data to the terminal device to feedback the completion of data access. If the first data access request indicates to modify the first target row data, the network device may send a modification completion message to the terminal device to feed back the completion of the data access. In practical applications, there may be other manners, which are not specifically limited here.

In the embodiment of the present application, multiple ways of obtaining popularity information are provided, and in practical applications, they can be flexibly selected, which improves the flexibility and practicability of the technical solution of the present application.

In some optional embodiments, the network device may provide a temperature threshold configuration interface, where the temperature threshold configuration interface is used to obtain a temperature threshold input by a user. The network device determines the hot data from the row data set according to the heat information and the heat threshold. The popularity threshold may indicate the proportion of hot data in the row data set, or indicate the minimum value of access times corresponding to the hot data, which is not limited here. The possible situations are described below.

A) The hotness threshold input by the user indicates the proportion of hot data in the row data set.

In some optional embodiments, the heat threshold configuration interface may be displayed as the threshold setting interface shown in FIG. 7a. If the heat threshold set by the user is 20%, the user only needs to input 20. In practical applications, the threshold setting interface may also have multiple forms, and the values input by the user may also be expressed in different orders of magnitude, which is not specifically limited here.

Assume that the network device follows the above "1) record the number of accesses based on the frequency counter" or the above "3) record the number of accesses based on the frequency counter and expiration time", and in step 404 shown in Figure 4a, the popularity information saved to the disk is in accordance with the accessCount Descending order is performed, and the saved data includes <accessCount, row number>, then the row data corresponding to the location information stored in the first 20% (that is, the 20% with the highest access frequency) is determined as hot data.

It is understandable that in practical applications, there may be cases where the access frequency of data outside the top 20% is the same as that of data within the top 20%. The accessCount value corresponding to the row data is the highest among the 5 row data, all of which are 3, so there are multiple processing situations. If the network device follows the above "1) record the number of visits based on the frequency counter", then the network can determine that any one of the two data is hot data. If the network device records the number of visits according to the above "3) according to the frequency counter and expiration time", then the network device can further compare the number of failures corresponding to the expiration time of the two rows of data, and select the row data with fewer failure times as hot data . If the number of failures is the same, it can be determined that any one of the two data is hot data.

B) The heat threshold value input by the user indicates the minimum value of the number of visits corresponding to the heat data.

In some optional embodiments, the thermal threshold configuration interface may be displayed as the threshold setting interface shown in FIG. 7b. As shown in Figure 7b, the heat threshold set by the user is 100 times. In practical applications, the threshold setting interface may also have multiple representations, and the value input by the user may also be other values, which are not specifically limited here.

In the embodiment of the present application, the popularity threshold can be set by the user, which enhances the user experience. In addition, there are many ways to define the heat threshold, which can be selected according to the needs of practical applications, which further improves the flexibility of the technical solution of the present application.

In some optional embodiments, the temperature threshold can also be set according to the type of application, or set by the R&D/testing personnel, which is not specifically limited here.

It can be understood that, based on the above method, after the network device determines the hot data from the row data set, it can determine other data in the row data set as cold data, and determine the first location corresponding to the hot data from the location information set The information corresponds to the second location information of the cold data.

In the embodiment of the present application, after the data recovery process is executed, the data in the memory database returns to normal, and the network device can receive the data access request and execute the corresponding access operation. That is to say, after step 303 in the embodiment shown in FIG. 3 , the network device may obtain a second data access request from the terminal device, and the second data access request indicates to access the second target row data in the row data set. If the second target row data is hot data, access the hot data in the memory; if the second target row data is cold data, load the cold data according to the second location information.

The following describes the data access process after the data recovery of the memory database in conjunction with FIG. 8. Please refer to FIG. 8. FIG. 8 is a schematic flow chart of the data processing method provided by the embodiment of the present application, including the following steps:

801. The network device acquires the second data access request from the terminal device.

The terminal device may send a second data access request to the network device, where the second data access request indicates to access the second target row data in the row database. Wherein, the second data access request may carry the row number corresponding to the second row data, or the unique index corresponding to the second target row data, so that the network device can determine whether the row data being accessed is Second target row data. Accessing the second target row data may include performing other operations on the second target row data in addition to reading or writing the second target row data, for example, modifying the content of the second target row data, etc., specifically There is no limit here.

802. The network device determines whether the second target row data to be accessed indicated by the second data access request is hot data, if yes, perform step 803, and if not, perform step 804.

In the embodiment of the present application, since the row data in the row data set is divided into hot data and cold data, different processing is performed on the hot data and cold data respectively during the data recovery process. Therefore, the network device needs to determine whether the second target row data to be accessed indicated by the second data access request is hot data, and perform corresponding operations.

803. A network device accesses hot data in memory.

If the second target row data is hot data, the network device can directly access the second target row data in memory.

804. The network device loads cold data according to the second location information.

If the second target row data is cold data, the network device needs to load the cold data according to the second location information, and then access the loaded cold data. Wherein, the second location information indicates the location of the cold data in the log.

In some optional embodiments, after step 804, the network device may also execute step 805, where the network device feeds back that the data access is completed. Similar to step 605 in the embodiment shown in FIG. 6 , in step 805 , there are multiple ways for the network device to report the completion of data access, which will not be repeated here. The difference is that in step 605, the data access completion information fed back by the network device corresponds to the first target row data; in step 805, the data access completion information fed back by the network device corresponds to the second target row data.

In the embodiment of this application, the row data set in the memory database is divided into hot data and cold data, and different data are processed differently during the data recovery process. Hot data can be directly accessed in memory, ensuring good performance of the system. Cold data with low access frequency is backed up as location information. When accessing cold data, it is loaded at runtime according to the location information, which reduces the overall data recovery time of the memory database.

It can be understood that, in the embodiment of the present application, during the process of data access, it is also necessary to record the popularity information of each row data in the row data set, so as to provide a basis for the next checkpoint operation.

In the above introduction, the data processing process of a single computer is described. The data processing method provided by the embodiment of the present application can also be applied in the scenario where multiple network devices interact. This situation will be described below.

Please refer to FIG. 9 . FIG. 9 is a schematic structural diagram of a data processing system provided by an embodiment of the present application.

As shown in Figure 9, the data processing system includes a first network device and a second network device, the first network device may be called a main network device, and the first network device has a main memory database; the second network device may be called a backup network device, there is a backup in-memory database on the second network device, or a secondary in-memory database. The design scheme of the master-slave memory database can achieve high availability.

Synchronization technology is adopted between the main memory database and the slave memory data path. When the main memory database fails, the slave memory database can quickly take over. In this embodiment of the present application, the first network device will synchronize hot checkpoint information with the second network device.

In some optional embodiments, the data processing system further includes a management device, the management device is used to determine the status of the first network device, and when the first network device fails, send a takeover instruction to the second network device, so that the second network device The network device can quickly take over the work of the first network device.

Please refer to FIG. 10. FIG. 10 is a schematic flow chart of the data processing method provided by the embodiment of the present application, including the following steps:

1001. The first network device acquires hot checkpoint information, the hot checkpoint information corresponds to hot data in the row data set, and the access frequency of the hot data is higher than that of other data in the row data set.

In some optional embodiments, the hot checkpoint information includes hot data, and the way for the first network device to determine the hot data has been described in detail above, and will not be repeated here.

In some optional embodiments, the hot checkpoint information includes location information corresponding to the hot data, and the location information is used to indicate the location of the hot data in the log. It can be understood that the location information corresponding to the hot data shown here is the first location information introduced above, and how the network device obtains the first location information has been described in detail above and will not be repeated here.

In the embodiment of the present application, there are various types of hot checkpoint information, which can be determined according to actual application requirements, which improves the flexibility of the technical solution.

1002. The first network device sends hot checkpoint information to the second network device.

The first network device sends hot checkpoint information to the second network device asynchronously, so that after the first network device fails, the base terminal device can access hot data based on the second network device.

1003. If the first network device fails, the second network device takes over the work of the first network device.

In the case of failure of the first network device, the second network device will receive a takeover signal, and take over the work of the first network device according to the takeover instruction. Wherein, the work of the first network device includes accessing data in the memory database. The following description will be made in conjunction with FIG. 11. Please refer to FIG. 11. FIG. 11 is a schematic flow chart of the data processing method provided by the embodiment of the present application, including the following steps:

1101. The management device sends a takeover signal to the second network device.

After the first network device fails, the management device will send a takeover signal to the second network device to instruct the second network device to take over the work of the first network device.

1102. The second network device reports that the takeover is complete.

After the takeover is completed, the second network device may report the completion of the takeover to the management device. Optionally, if the second network device successfully receives the hot checkpoint information transmitted by the first network device before the failure, it may be considered that the second network device takes over successfully.

1103. The terminal device sends a hot data access request to the second network device.

Due to the failure of the first network device, the second network device takes over the work of the first network device, therefore, the data access request is received by the second network device. The terminal device may send a hot data access request to the second network device, where the request is used to instruct access to the hot data in the row data set. Accessing hot data may include, in addition to reading or writing the hot data, other operations on the hot data, for example, modifying the content of the hot data, etc., which is not limited here.

1104. The second network device directly accesses the hot data in memory.

Since the hot data access request indicates that the hot data is accessed, when the hot checkpoint information received by the second network device is hot data, the second network device will store the hot data in the memory. In this case, the second network Devices can access hot data directly in memory.

Optionally, if the hot checkpoint information received by the second network device is the location information corresponding to the hot data, then the second network device will load the hot data into the memory according to the location information after receiving the location information, It is convenient for the second network device to access the hot data in the memory in the subsequent process.

1105. The second network device feeds back that hot data access is complete.

In some optional embodiments, after step 1104, the network device may also perform step 1105, and the second network device feeds back that the data access is completed. Similar to step 605 in the embodiment shown in FIG. 6 , in step 1105 , the second network device can feedback the completion of hot data access in many ways, which will not be repeated here. The difference is that in step 605, the execution subject is a network device, and the data access completion information fed back by the network device corresponds to the first target row data; in step 1105, the execution subject is the second network device, and the second network The data access completion information fed back by the device corresponds to hot data.

In some optional embodiments, the row data set includes not only hot data but also cold data, and the second network device can also implement access to cold data, that is, perform steps 1106 to 1108 . It can be understood that FIG. 11 is only an example of the technical solution of the present application, and does not constitute a limitation on steps, and there is no necessary sequence between step 1106 and step 1103 .

1106. The terminal device sends a cold data access request to the second network device.

The terminal device may send a cold data access request to the second network device, where the request is used to instruct to access the cold data in the row data set. In addition to reading or writing cold data, accessing cold data may also include performing other operations on cold data, for example, modifying the content of cold data, etc., which is not limited here.

1107. The second network device loads the cold data during operation according to the location information of the cold data.

The first network device and the second network device share a disk, and the first network device may perform the operation shown in FIG. 4a to save the location information set corresponding to the row data set to disk. The second network device may obtain the location information corresponding to the cold data through the disk, and load the cold data during operation based on the location information, so as to complete the access to the cold data.

1108. The second network device reports back that the cold data access is completed.

In some optional embodiments, after step 1104, the network device may also perform step 1108, and the second network device feeds back that the data access is completed. Similar to step 605 in the embodiment shown in FIG. 6 , in step 1108 , the second network device may feedback the completion of the cold data access in various ways, which will not be repeated here. The difference is that in step 605, the execution subject is a network device, and the data access completion information fed back by the network device corresponds to the first target row data; in step 1108, the execution subject is a second network device, and the second network The data access completion information fed back by the device corresponds to cold data.

In the embodiment of the present application, the first network device synchronizes the hot checkpoint information with the second network device instead of log information, so that the synchronization speed of the first network device and the second network device match, which is more in line with the needs of practical applications. It also improves the practicability of the technical solution of the present application.

It should be noted that in this embodiment of the application, if the first network device is not faulty, the first network device may also perform the operations performed by the network device in the embodiments shown in FIGS. 1 to 8; In the case of a network device failure, the second network device may also perform the operations performed by the network device in the foregoing embodiments shown in FIG. 1 to FIG. 8 , which will not be repeated here.

The embodiment of the present application also provides a data processing method, the method is applied to a data processing system, and the data processing system includes a first network device and a second network device.

This method is described below in conjunction with FIG. 12. Please refer to FIG. 12. FIG. 12 is a flow chart of the data processing method provided by the embodiment of the present application, including the following steps:

1201. The first network device acquires hot checkpoint information, the hot checkpoint information corresponds to hot data in the row data set, and the access frequency of the hot data is higher than that of other data in the row data set.

1202. The first network device sends hot checkpoint information to the second network device.

Step 1201 to step 1202 are similar to step 1001 and step 1002 in the aforementioned embodiment shown in FIG. 10 , please refer to the description of step 1001 and step 1002 for details, and details are not repeated here.

1203. The second network device loads hot data into the memory according to the hot checkpoint information.

After the second network device obtains the hot checkpoint information, it loads hot data into the memory. Specifically, if the hot checkpoint information is hot data, the second network device will store the hot data in the memory; if the hot checkpoint information is the location information corresponding to the hot data, the second network device will store the Location information to load hot data into memory.

1204. If the first network device fails, the second network device receives a data access request, and the data access request indicates access to hot data.

When the first network device fails, the data access operation is performed by the second network device. The second network device may receive a data access request indicating access to hot data.

1205. The second network device responds to the data access request, and accesses the hot data in the memory.

In some optional embodiments, the second network device may also perform the operations shown in steps 1106 to 1108 in the embodiment shown in FIG. 11 , which will not be detailed here.

In the embodiment of the present application, the first network device synchronizes the hot checkpoint information with the second network device instead of log information, so that the synchronization speed of the first network device and the second network device match, which is more in line with the needs of practical applications. It also improves the practicability of the technical solution of the present application.

It should be noted that in this embodiment of the application, if the first network device is not faulty, the first network device may also perform the operations performed by the network device in the embodiments shown in FIGS. 1 to 8; In the case of a network device failure, the second network device may also perform the operations performed by the network device in the foregoing embodiments shown in FIG. 1 to FIG. 8 , which will not be repeated here.

Next, related devices and equipment provided in the embodiments of the present application will be described.

Please refer to FIG. 13. FIG. 13 is a schematic structural diagram of a network device provided by an embodiment of the present application. The network device 1300 includes:

The acquiring unit 1301 is configured to acquire the location information set of the row data set, the row data set includes hot data, the location information set includes the first location information corresponding to the hot data, and the access frequency of the hot data is higher than that of other data in the row data set frequency.

The acquiring unit 1301 is further configured to acquire a data restoration instruction, and the data restoration instruction is used to instruct to restore data in the row data set.

The processing unit 1302 is configured to load the hot data into the memory according to the first location information in response to the data restoration instruction.

In some optional embodiments, the processing unit 1302 is further configured to record popularity information of each row of data in the row data set, where the popularity information is used to indicate the access frequency of each row of data.

In some optional embodiments, the popularity information includes the number of visits, and the processing unit 1302 is specifically configured to: obtain a first data access request, where the first data access request indicates access to the first target row data in the row data set; A data access request, increasing the number of visits corresponding to the first target row data.

In some optional embodiments, the processing unit 1302 is further configured to: provide a popularity threshold configuration interface, and the popularity threshold configuration interface is used to obtain the popularity threshold input by the user; determine the hotness threshold from the row data set according to the popularity information and the popularity threshold data.

In some optional embodiments, the other data includes cold data, and the location information set further includes second location information corresponding to the cold data.

The acquiring unit 1301 is further configured to acquire a second data access request from the client, where the second data access request indicates to access the second target row data in the row data set;

The processing unit 1302 is further configured to: if the second target row data is hot data, access the hot data in memory; if the second target row data is cold data, load the cold data according to the second location information.

The network device 1300 is configured to execute the operations performed by the network device in the foregoing embodiments shown in FIGS. 1 to 8 , and details are not repeated here.

Please refer to FIG. 14. FIG. 14 is a schematic structural diagram of the first network device provided by the embodiment of the present application. The first network device 1400 is included in the data processing system, and the data processing system also includes a second network device, the first network device 1400 include:

The obtaining unit 1401 is configured to obtain hot checkpoint information, the hot checkpoint information corresponds to the hot data in the row data set, and the access frequency of the hot data is higher than that of other data in the row data set;

The sending unit 1402 is configured to send hot checkpoint information to the second network device, so that the hot data can be accessed based on the second network device after the first network device fails.

In some optional embodiments, the hot checkpoint information includes hot data.

In some optional embodiments, the hot checkpoint information includes location information corresponding to the hot data, and the location information is used to indicate the location of the hot data in the log.

The first network device 1400 is configured to perform the operations performed by the first network device in the foregoing embodiments shown in FIGS. 9 to 11 , and details are not repeated here.

Please refer to FIG. 15 . FIG. 15 is a schematic structural diagram of a data processing system provided by an embodiment of the present application. The data processing system 1500 includes a first network device 1501 and a second network device 1502 .

The first network device 1501 is configured to: acquire hot checkpoint information, the hot checkpoint information corresponds to hot data in the row data set, and the access frequency of the hot data is higher than that of other data in the row data set. Send hot checkpoint information to the second network device.

The second network device 1502 is configured to: load hot data in memory according to hot checkpoint information. If the first network device fails, the second network device receives a data access request, and the data access request indicates access to hot data. Access hot data in memory in response to data access requests.

The data processing system 1500 is used to implement the functions implemented by the data processing system in the foregoing embodiments shown in FIGS. 9 to 12 , and details are not described here again.

Next, the network device provided by the embodiment of the present application will be described, please refer to FIG. 16 , which is a schematic structural diagram of the network device provided by the embodiment of the present application.

The network device 1600 includes: a processor 1601 and a memory 1602, and the memory 1602 stores one or more application programs or data.

Wherein, the storage 1602 may be a volatile storage or a persistent storage. The program stored in the memory 1602 may include one or more modules, and each module may be used to perform a series of operations performed by the network device 1600 . Furthermore, the processor 1601 may communicate with the memory 1602 , and execute a series of instruction operations in the memory 1602 on the network device 1600 . The processor 1601 may be a central processing unit (central processing units, CPU), or a single-core processor, or other types of processors, such as a dual-core processor, which are not specifically limited here.

The network device 1600 may also include one or more communication interfaces 1603, and one or more operating systems, such as Windows Server ^TM , Mac OS X ^TM , Unix ^TM , Linux ^TM , FreeBSD ^TM and so on.

The network device 1600 can perform the operations performed by the network device in the foregoing embodiments shown in FIGS. 1 to 8 , and the operations performed by the first network device in the foregoing embodiments shown in FIGS. 9 to 11 , which will not be repeated here. .

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

Claims (21)

A data processing method, characterized in that, comprising: Acquiring a location information set of a row data set, the row data set includes hot data, the location information set includes first location information corresponding to the hot data, and the access frequency of the hot data is higher than that in the row data set Frequency of access to other data; Obtaining a data restoration instruction, where the data restoration instruction is used to instruct to restore the data in the row data set; In response to the data recovery instruction, load the hot data into memory according to the first location information. The method according to claim 1, wherein, before the acquisition of the location information set of the row data set, the method further comprises: Recording popularity information of each row of data in the row data set, where the popularity information is used to indicate the access frequency of each row of data. The method according to claim 2, wherein the popularity information includes the number of visits, and the recording of the popularity information of each row data in the row data set includes: Obtain a first data access request, where the first data access request indicates access to first target row data in the row data set; According to the first data access request, increase the access times corresponding to the first target row data. The method according to claim 2 or 3, characterized in that the method further comprises: Provide a temperature threshold configuration interface, the temperature threshold configuration interface is used to obtain the temperature threshold input by the user; The heat data is determined from the row data set according to the heat information and the heat threshold. The method according to any one of claims 1 to 4, wherein the other data includes cold data, and the location information set further includes second location information corresponding to the cold data; After said loading said hot data into memory, said method also includes: Acquiring a second data access request from the terminal device, the second data access request indicating access to second target row data in the row data set; If the second target row data is the hot data, access the hot data in the memory; If the second target row data is the cold data, load the cold data according to the second location information. A data processing method, wherein the method is applied to a first network device, the first network device is included in a data processing system, and the data processing system further includes a second network device, and the method includes: Acquiring hot checkpoint information, the hot checkpoint information corresponds to hot data in the row data set, and the access frequency of the hot data is higher than the access frequency of other data in the row data set; The hot checkpoint information is sent to the second network device, so that the hot data can be accessed based on the second network device after the first network device fails. The method according to claim 6, wherein the hot checkpoint information includes the hot data. The method according to claim 6, wherein the hot checkpoint information includes location information corresponding to the hot data, and the location information is used to indicate the location of the hot data in the log. A data processing method, wherein the method is applied to a data processing system, the data processing system includes a first network device and a second network device, and the method includes: The first network device acquires hot checkpoint information, the hot checkpoint information corresponds to hot data in the row data set, and the access frequency of the hot data is higher than the access frequency of other data in the row data set; The first network device sends the hot checkpoint information to the second network device; The second network device loads the hot data in memory according to the hot checkpoint information; If the first network device fails, the second network device receives a data access request, and the data access request indicates access to the hot data; The second network device accesses the hot data in memory in response to the data access request. A network device, characterized in that it includes: An acquisition unit, configured to acquire a location information set of a row data set, the row data set includes hot data, the location information set includes first location information corresponding to the hot data, and the access frequency of the hot data is higher than the set The frequency of access to other data in the data set; The acquiring unit is further configured to acquire a data restoration instruction, and the data restoration instruction is used to instruct to restore the data in the row data set; A processing unit, configured to load the hot data into the memory according to the first location information in response to the data recovery instruction. The network device according to claim 10, wherein the processing unit is further configured to record the popularity information of each row data in the row data set, and the popularity information is used to indicate that each row data frequency of visits. The network device according to claim 11, wherein the popularity information includes the number of visits, and the processing unit is specifically used for: Obtain a first data access request, where the first data access request indicates access to first target row data in the row data set; According to the first data access request, increase the access times corresponding to the first target row data. The network device according to claim 11 or 12, wherein the processing unit is further configured to: Provide a temperature threshold configuration interface, the temperature threshold configuration interface is used to obtain the temperature threshold input by the user; The heat data is determined from the row data set according to the heat information and the heat threshold. The network device according to any one of claims 10 to 13, wherein the other data includes cold data, and the location information set further includes second location information corresponding to the cold data; The acquiring unit is further configured to acquire a second data access request from the client, the second data access request indicating to access the second target row data in the row data set; The processing unit is also used for: If the second target row data is the hot data, access the hot data in the memory; If the second target row data is the cold data, load the cold data according to the second location information. A first network device, characterized in that the first network device is included in a data processing system, and the data processing system further includes a second network device, and the first network device includes: An acquisition unit, configured to acquire hot checkpoint information, the hot checkpoint information corresponds to hot data in the row data set, and the access frequency of the hot data is higher than that of other data in the row data set; A sending unit, configured to send the hot checkpoint information to the second network device, so that the hot data can be accessed based on the second network device after the first network device fails. The first network device according to claim 15, wherein the hot checkpoint information includes the hot data. The first network device according to claim 15, wherein the hot checkpoint information includes location information corresponding to the hot data, and the location information is used to indicate the location of the hot data in the log. A data processing system, characterized by comprising a first network device and a second network device; The first network device is configured to: Acquiring hot checkpoint information, the hot checkpoint information corresponds to hot data in the row data set, and the access frequency of the hot data is higher than the access frequency of other data in the row data set; sending the hot checkpoint information to the second network device; The second network device is configured to: Loading the hot data in memory according to the hot checkpoint information; If the first network device fails, the second network device receives a data access request, and the data access request indicates access to the hot data; In response to the data access request, access the hot data in memory. A network device, characterized in that it includes: a processor, a memory, and a communication interface; The processor and the memory are connected to the communication interface; The processor is configured to execute the method of any one of claims 1-9. A computer-readable storage medium, characterized in that the computer-readable storage medium stores instructions, and when the instructions are run on a computer, the computer executes the method described in any one of claims 1 to 9. method. A computer program product, characterized in that, when the computer program product is executed on a computer, the computer executes the method according to any one of claims 1 to 9.

Images