CN118394731A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN118394731A CN118394731A CN202410353679.6A CN202410353679A CN118394731A CN 118394731 A CN118394731 A CN 118394731A CN 202410353679 A CN202410353679 A CN 202410353679A CN 118394731 A CN118394731 A CN 118394731A
- Authority
- CN
- China
- Prior art keywords
- data
- access request
- storage medium
- storage
- address
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 11
- 238000000034 method Methods 0.000 claims abstract description 52
- 230000004044 response Effects 0.000 claims abstract description 10
- 238000012545 processing Methods 0.000 claims description 25
- 230000008859 change Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 12
- 238000004891 communication Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000008602 contraction Effects 0.000 description 5
- 230000005012 migration Effects 0.000 description 5
- 238000013508 migration Methods 0.000 description 5
- 230000000903 blocking effect Effects 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000010276 construction Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000001419 dependent effect Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method and device, and relates to the technical field of computers. One embodiment of the method comprises the following steps: in response to receiving a data access request, acquiring a data identifier from the data access request; acquiring a storage address corresponding to the data identifier from an address book, wherein the storage address indicates a storage medium; request data corresponding to the data access request is acquired from the storage medium. According to the embodiment, after the data access request is received, the request data is acquired in an address book addressing mode, so that the addressing calculation amount and complexity can be reduced, the addressing efficiency is improved, and the data query efficiency is further improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for data processing.
Background
In the distributed storage system, a hash (Controlled Replication Under Scalable Hashing, controlled replication under scalable hash) algorithm is widely adopted for data storage, data routing is realized, and hash values of data are mapped to a tree structure of a socket to obtain storage positions of the data. However, in this data routing manner, when the data movement or the expansion and contraction of the storage device is performed, the mapping relationship between the data and the storage location needs to be recalculated, which may cause traffic blocking, and the mapping relationship and the storage location need to be frequently calculated by the credit algorithm, which increases the calculation load and delay of the system.
Disclosure of Invention
In view of this, the embodiments of the present invention provide a method and an apparatus for processing data, which can obtain request data by addressing an address book after receiving a data access request, so as to reduce the amount and complexity of addressing calculation, improve the addressing efficiency, and further improve the data query efficiency.
To achieve the above object, according to one aspect of an embodiment of the present invention, there is provided a method of data processing, including:
In response to receiving a data access request, acquiring a data identifier from the data access request;
Acquiring a storage address corresponding to the data identifier from an address book, wherein the storage address indicates a storage medium;
Request data corresponding to the data access request is acquired from the storage medium.
Optionally, before acquiring the storage address corresponding to the data identifier from the address book, the method further includes:
acquiring data attributes of the storage medium;
and storing the data identification and the storage address in an address book corresponding to the data attribute.
Optionally, before storing the data identifier and the storage address in the address book corresponding to the data attribute, the method further includes:
and establishing an address book corresponding to the data attribute.
Optionally, acquiring request data corresponding to the data access request from the storage medium includes:
Sending a request to a connector according to the storage address;
Determining a pipe corresponding to the storage medium;
the connector is connected to the pipe to obtain the requested data from the storage medium via the pipe.
Optionally, the method further comprises:
In response to receiving a data change notification, obtaining a target data identifier and a target storage address from the data change notification; the target storage address indicating a target storage medium;
Determining a target address book corresponding to the target storage medium;
And updating the target address book according to the target data identification and the target storage address.
Optionally, the method further comprises:
After receiving the data change notification, constructing a loader;
acquiring a primary storage address corresponding to the target data identifier, wherein the primary storage address indicates a primary storage medium;
And hanging the loader on a pipeline corresponding to the original storage medium, and storing the target data identification and the target storage address into the pipeline by using the loader.
Optionally, before acquiring the data identifier from the data access request, the method further includes:
determining that the data access request meets a preset flow limiting strategy; the preset current limiting strategy is realized through a hot plug current limiting strategy plug-in;
And rejecting the data access request under the condition that the data access request does not meet a preset flow limiting strategy.
According to still another aspect of an embodiment of the present invention, there is provided an apparatus for data processing, including:
The receiving module is used for responding to the received data access request and acquiring a data identifier from the data access request;
the first acquisition module acquires a storage address corresponding to the data identifier from an address book, wherein the storage address indicates a storage medium;
and the second acquisition module acquires request data corresponding to the data access request from the storage medium.
According to another aspect of an embodiment of the present invention, there is provided an electronic apparatus including:
one or more processors;
storage means for storing one or more programs,
The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the methods of data processing provided by the present invention.
According to a further aspect of an embodiment of the present invention, there is provided a computer readable medium having stored thereon a computer program which when executed by a processor implements the method of data processing provided by the present invention.
One embodiment of the above invention has the following advantages or benefits: according to the data processing method, after a data access request is received, a data identifier is acquired from the data access request, then a corresponding storage address is acquired from an address book according to the data identifier, and then request data corresponding to the data access request is acquired from a corresponding storage medium according to the storage address. The method provides a general high-performance OSD hierarchical storage data routing method, and after receiving a data access request, request data is acquired in an address book addressing mode, so that the addressing calculation amount and complexity can be reduced, the addressing efficiency is improved, and the data query efficiency is further improved.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of the main flow of a method of data processing according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the main flow of another method of data processing according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the main flow of a method of further data processing according to an embodiment of the invention;
FIG. 4is a schematic diagram of the main modules of an apparatus for data processing according to an embodiment of the present invention;
FIG. 5 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
Fig. 6 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
FIG. 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention, and as shown in FIG. 1, the data processing method includes the following steps:
step S101: in response to receiving the data access request, obtaining a data identifier from the data access request;
step S102: acquiring a storage address corresponding to the data identifier from an address book, wherein the storage address indicates a storage medium;
Step S103: request data corresponding to the data access request is obtained from the storage medium.
In the embodiment of the invention, the data processing method is suitable for data storage and data routing management in a distributed storage system. The data access request may be sent by the client or the client may be sent by a proxy server, such as proxy server. After receiving the data access request, the data access request is parsed, and a data identifier, which may be an OSD (Object-based-storage) Key, may be obtained from the data access request.
In the embodiment of the invention, the address book comprises the corresponding relation between the data identifier of the data and the storage position of the data, according to the data identifier obtained from the data access request, the storage address corresponding to the data identifier can be obtained from the address book, and the storage address indicates the storage medium, so that the request data corresponding to the data access request can be obtained from the storage medium according to the storage address, and the request data can be returned. The storage medium may be a solid state disk (ssd) or hhd hard disk.
In the embodiment of the invention, before the storage address corresponding to the data identifier is acquired from the address book, the method further comprises the following steps:
Acquiring data attributes of a storage medium;
the data identification and the storage address are stored in an address book corresponding to the data attributes.
In the embodiment of the invention, the data identifier and the storage address corresponding to the data identifier are stored in the address book, and the data identifier and the corresponding storage address need to be stored in the address book before the storage address corresponding to the data identifier is acquired from the address book. The data attribute of the storage medium is the attribute of the data stored in the storage medium, the data attribute can be cold or warm, and different data attributes correspond to different address books. By acquiring the data attribute of the storage medium, the data identifier and the storage address are stored in the address book corresponding to the data attribute, so that the differential hierarchical storage of the data object is realized, and the waste of storage resources is reduced. That is, the same socket (the basic storage space of the object storage service) can contain a plurality of storage media with different data attributes, so as to realize OSD hierarchical storage.
In the embodiment of the invention, before the data identification and the storage address are stored in the address book corresponding to the data attribute, the method further comprises the following steps: an address book corresponding to the data attributes is established. That is, address books corresponding to each data attribute are built in advance, so that data identifiers and storage addresses of data objects of storage media with different data attributes are stored on the corresponding address books, namely, the data identifiers and the storage addresses of data on the storage media with the data attributes being heat labels are stored on the address books corresponding to the heat labels, and automatic identification of cold and hot data is realized by building the address books with different data attributes, so that data migration and routing on high-speed and low-speed storage media are realized. Moreover, the method of adopting the address book can reduce the calculation complexity and support the dynamic expansion and contraction capacity.
In the embodiment of the invention, the data identifier in the address book is generated by an md5 (Message-Digest Algorithm) Algorithm according to the full path of the data object on the storage medium, and the data identifier is irrelevant to the storage medium or the storage address where the data object is located, and because the full paths of the data object on different storage media are the same, the data identifier is unchanged. full path includes service identification (bf), application identification (af), socket name, user-defined identification (udf), object name, for example, zhangsanidcard.
Bf=party, af= jdfinance, socket= idcards, udf = CREDITCARDPAY, object name=zhangsanidcard
The OSD key takes the value: MD5 (bf/af/bucket/udf/date/objName) =ae 9537a5ad2007b1a 276fc4bbe99a6.
In an embodiment of the present invention, as shown in fig. 2, obtaining request data corresponding to a data access request from a storage medium includes:
Step S201: sending a request to the connector according to the storage address;
Step S202: determining a pipe corresponding to the storage medium;
Step S203: the connector is connected to the pipe construction to obtain the requested data from the storage medium through the pipe.
In the embodiment of the invention, after receiving a data access request, the addressing request can be sent to the addressing route through the front load, the addressing route determines a storage address corresponding to the data identifier according to the address book, the storage address indicates a storage medium, the storage medium is provided with a corresponding connector and a pipeline, the addressing route sends a connection request to the connector according to the storage address, the connector is connected with the corresponding pipeline after receiving the request, and the data access request is sent to the storage medium through the pipeline to acquire the request data. Wherein, the connector can be used as a front buffer for addressing, which can reduce the perception of address variation to users. The pipes are data-carrying in the connectors and the storage medium. The connectors are in binding relation with the pipelines, one connector can correspond to a plurality of pipelines, and the pipelines and the storage medium are in one-to-one correspondence.
In an embodiment of the present invention, as shown in fig. 3, the method for processing data may further include:
Step S301: in response to receiving the data change notification, obtaining a target data identifier and a target storage address from the data change notification; the target storage address indicates a target storage medium;
step S302: determining a target address book corresponding to the target storage medium;
Step S303: and updating the target address book according to the target data identification and the target storage address.
In the embodiment of the invention, the data processing method can be executed by the controller, the monitoring equipment monitors the data on the storage medium, and when the data on the storage medium is monitored to be changed, a data change notification is sent to the controller. The data change notification may be generated during data migration or expansion and contraction of the storage medium. After receiving the data change notification, resolving a target data identifier and a target storage address from the data change notification, wherein the target storage address indicates a target storage medium, determining a corresponding target address book according to the data attribute of the target storage medium, and updating the target address book according to the target data identifier and the target storage address. When updating the target address book, judging whether the target data identifier exists in the target address book, if so, updating the target storage address into the target address book, and if not, adding the target data identifier into the target address book, and correspondingly storing the target storage address and the target data identifier.
In the embodiment of the present invention, the method for processing data may further include:
After receiving the data change notification, constructing a loader;
Acquiring a primary storage address corresponding to the target data identifier, wherein the primary storage address indicates a primary storage medium;
And hanging the loader on a pipeline corresponding to the original storage medium, and storing the target data identification and the target storage address into the pipeline by using the loader.
In the embodiment of the invention, when the data object is subjected to cross-layer transfer or storage medium expansion and contraction, namely the data object is transferred on storage media with different data attributes of the same socket, or the storage media corresponding to the socket are subjected to expansion or contraction, the original storage address is invalid, and a loader is constructed after receiving a data change notification, so that high-efficiency addressing is realized; the method comprises the steps of obtaining an original storage address corresponding to a target data identifier, determining a pipeline corresponding to an original storage medium corresponding to the original storage address, hanging a loader on the pipeline, initializing the target data identifier and the target storage address into a map structure, storing the map structure in a local cache of the pipeline, updating an address book in the local cache of the pipeline, broadcasting a migration completion message by the loader after updating, and automatically destroying the loader. After receiving the data change notification, the construction load device is hung on the pipeline, so that the addressing efficiency is improved, and service blocking cannot be caused.
In the embodiment of the invention, after the data object is migrated from the original storage address to the target storage address, the consistency of the data before and after the data migration can be verified through md 5.
In the embodiment of the invention, before the data identifier is acquired from the data access request, the method further comprises the following steps:
Determining that the data access request meets a preset flow limiting strategy; the preset current limiting strategy is realized by a hot plug current limiting strategy plug-in;
and rejecting the data access request under the condition that the data access request does not meet the preset flow limiting strategy.
In the embodiment of the invention, after the data access request is received, whether the data access request meets a preset flow limiting strategy is judged, if yes, the data access request is analyzed to obtain a corresponding data identifier, and if not, the access request is refused. The preset flow limiting strategy can be that the number of the data access requests in the preset time does not reach a preset threshold, if the number of the data access requests in the preset data exceeds the preset threshold, the data access requests are refused, and the access frequency can be limited to passively defend the availability of the contained service and prevent malicious attacks when the data access requests are accessed in high-frequency concurrency through flow control, so that the orderly proceeding of the data access is ensured, and the stability of the route is improved. The preset current limiting strategy is realized through a hot plug current limiting strategy plug-in, and the hot plug current limiting strategy plug-in can complete the access and deletion of the strategy plug-in under the condition of not interrupting the data adding and deleting and modifying service, so that the limitation on the data query is realized.
In an embodiment of the present invention, the method may further include: and carrying out encryption transmission on the request data by utilizing a data encryption transmission strategy and returning the request data, wherein the data encryption transmission strategy is realized by a data ciphertext transmission plug-in. By encrypting the request data, the data access is securely reinforced, and the risk of data leakage is reduced.
In the embodiment of the invention, hot plug-in connectors with various strategies can be adopted to realize the current limitation of data inquiry, such as a load balancing plug-in connector and a data ciphertext transmission plug-in connector. The plugins are stored in jar packets. The load balancing plug-in may enable migration to one node when another node is unavailable in the case of a multi-instance deployment. And the dependent scanner periodically scans the dependent library, and when the newly added plugin in the dependent library is identified, the jar packet is reflected and instantiated to acquire the effective data content of the plugin, so that the strategy of the plugin is realized. For example, after identifying the newly added plugin, the execution method of the newly added plugin is called when the data is routed, for example, an encrypted transmission plugin is newly added, and the execution method of the plugin is called when the data is routed to return to the inquirer to encrypt the data. When the number of the plugins in the dependency library is reduced, the plugins are deleted in the plugin list according to the plugin names, so that the execution method of the plugins is not called when data is returned.
In the embodiment of the invention, the data processing method can be realized by an OSD data routing device, the device comprises an interaction layer, a routing layer and a storage medium layer, the interaction layer comprises a router such as an addressing route, and the router is used for acquiring a storage address according to a data identifier after receiving a data access request and sending the request to a connector; the routing layer comprises a data access load, a connector, a pipeline and an addressing load, wherein the access load is used for receiving a data access request, sending the request to the addressing route after buffering, the connector is used for receiving the request sent by the addressing route, connecting the connector with the pipeline, and sending the request to a storage medium through the addressing load after receiving the request to acquire request data; the storage medium layer includes ssd, hhd, etc. The device is controlled by a controller, and the controller is used for initializing the device and realizing the list of the connector, the pipeline and the storage medium.
According to the data processing method, after a data access request is received, a data identifier is acquired from the data access request, then a corresponding storage address is acquired from an address book according to the data identifier, and then request data corresponding to the data access request is acquired from a corresponding storage medium according to the storage address. The method provides a general high-performance OSD hierarchical storage data routing method, which can reduce the addressing calculation amount and complexity in an address book addressing mode, improve the addressing efficiency and further improve the data query efficiency; the method can realize the route switching during the dynamic expansion capacity and data jump of the OSD layered storage without causing service blocking, and realize high-efficiency addressing; and the hot plug current limiting strategy plug-in performs current limiting control on the data access request, so that the routing stability is improved.
According to yet another aspect of an embodiment of the present invention, as shown in fig. 4, there is provided an apparatus 400 for data processing, including:
A receiving module 401, responsive to receiving a data access request, for obtaining a data identifier from the data access request;
A first obtaining module 402, configured to obtain a storage address corresponding to the data identifier from the address book, where the storage address indicates a storage medium;
the second acquisition module 403 acquires request data corresponding to the data access request from the storage medium.
In the embodiment of the present invention, the first obtaining module 402 is further configured to: before a storage address corresponding to the data identifier is acquired from an address book, acquiring the data attribute of a storage medium; the data identification and the storage address are stored in an address book corresponding to the data attributes.
In the embodiment of the present invention, the first obtaining module 402 is further configured to: an address book corresponding to the data attribute is established before the data identification and the storage address are stored in the address book corresponding to the data attribute.
In an embodiment of the present invention, the second obtaining module 403 is further configured to: sending a request to the connector according to the storage address; determining a pipe corresponding to the storage medium; the connector is connected to the pipe construction to obtain the requested data from the storage medium through the pipe.
In an embodiment of the present invention, the data processing apparatus further includes an update module configured to: in response to receiving the data change notification, obtaining a target data identifier and a target storage address from the data change notification; the target storage address indicates a target storage medium; determining a target address book corresponding to the target storage medium; and updating the target address book according to the target data identification and the target storage address.
In an embodiment of the present invention, the data processing apparatus further includes a route switching module, configured to: after receiving the data change notification, constructing a loader; acquiring a primary storage address corresponding to the target data identifier, wherein the primary storage address indicates a primary storage medium; and hanging the loader on a pipeline corresponding to the original storage medium, and storing the target data identification and the target storage address into the pipeline by using the loader.
In the embodiment of the present invention, the receiving module 401 is further configured to: before acquiring a data identifier from a data access request, determining that the data access request meets a preset flow limiting strategy; the preset current limiting strategy is realized by a hot plug current limiting strategy plug-in; and rejecting the data access request under the condition that the data access request does not meet the preset flow limiting strategy.
According to another aspect of an embodiment of the present invention, there is provided an electronic apparatus including: one or more processors; and a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the method for data processing provided by the present invention.
According to a further aspect of an embodiment of the present invention, there is provided a computer readable medium having stored thereon a computer program which when executed by a processor implements the method of data processing provided by the present invention.
Fig. 5 illustrates an exemplary system architecture 500 of a data processing method or apparatus to which embodiments of the present invention may be applied.
As shown in fig. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 is used as a medium to provide communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 505 via the network 504 using the terminal devices 501, 502, 503 to receive or send messages or the like. Various communication client applications may be installed on the terminal devices 501, 502, 503, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 501, 502, 503 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 505 may be a server providing various services, such as a background management server (by way of example only) providing support for shopping-type websites browsed by users using the terminal devices 501, 502, 503. The background management server may analyze and process the received data such as the product information query request, and feedback the processing result (e.g., the target push information, the product information—only an example) to the terminal device.
It should be noted that, the method for processing data provided by the embodiment of the present invention is generally performed by the server 505, and accordingly, the device for processing data is generally disposed in the server 505.
It should be understood that the number of terminal devices, networks and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 6, there is illustrated a schematic diagram of a computer system 600 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 6 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 601.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor includes a receiving module, a first acquisition module, and a second acquisition module. The names of these modules do not in some way limit the module itself, for example, the receiving module may also be described as "a module that obtains a data identification from a data access request in response to receiving the data access request".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to include: in response to receiving the data access request, obtaining a data identifier from the data access request; acquiring a storage address corresponding to the data identifier from an address book, wherein the storage address indicates a storage medium; request data corresponding to the data access request is obtained from the storage medium.
According to the technical scheme of the embodiment of the invention, after the data access request is received, the data identification is acquired from the data access request, then the corresponding storage address is acquired from the address book according to the data identification, and then the request data corresponding to the data access request is acquired from the corresponding storage medium according to the storage address. The method provides a general high-performance OSD hierarchical storage data routing method, which can reduce the addressing calculation amount and complexity in an address book addressing mode, improve the addressing efficiency and further improve the data query efficiency; the method can realize the route switching during the dynamic expansion capacity and data jump of the OSD layered storage without causing service blocking, and realize high-efficiency addressing; and the hot plug current limiting strategy plug-in performs current limiting control on the data access request, so that the routing stability is improved.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method of data processing, comprising:
In response to receiving a data access request, acquiring a data identifier from the data access request;
Acquiring a storage address corresponding to the data identifier from an address book, wherein the storage address indicates a storage medium;
Request data corresponding to the data access request is acquired from the storage medium.
2. The method of claim 1, further comprising, prior to retrieving the memory address corresponding to the data identification from an address book:
acquiring data attributes of the storage medium;
and storing the data identification and the storage address in an address book corresponding to the data attribute.
3. The method of claim 2, wherein prior to storing the data identification and storage address in an address book corresponding to the data attribute, further comprising:
and establishing an address book corresponding to the data attribute.
4. The method of claim 1, wherein obtaining request data corresponding to the data access request from the storage medium comprises:
Sending a request to a connector according to the storage address;
Determining a pipe corresponding to the storage medium;
the connector is connected to the pipe to obtain the requested data from the storage medium via the pipe.
5. The method according to claim 1, wherein the method further comprises:
In response to receiving a data change notification, obtaining a target data identifier and a target storage address from the data change notification; the target storage address indicating a target storage medium;
Determining a target address book corresponding to the target storage medium;
And updating the target address book according to the target data identification and the target storage address.
6. The method of claim 5, wherein the method further comprises:
After receiving the data change notification, constructing a loader;
acquiring a primary storage address corresponding to the target data identifier, wherein the primary storage address indicates a primary storage medium;
And hanging the loader on a pipeline corresponding to the original storage medium, and storing the target data identification and the target storage address into the pipeline by using the loader.
7. The method of claim 1, further comprising, prior to obtaining the data identifier from the data access request:
determining that the data access request meets a preset flow limiting strategy; the preset current limiting strategy is realized through a hot plug current limiting strategy plug-in;
And rejecting the data access request under the condition that the data access request does not meet a preset flow limiting strategy.
8. An apparatus for data processing, comprising:
The receiving module is used for responding to the received data access request and acquiring a data identifier from the data access request;
the first acquisition module acquires a storage address corresponding to the data identifier from an address book, wherein the storage address indicates a storage medium;
and the second acquisition module acquires request data corresponding to the data access request from the storage medium.
9. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
When executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-7.
10. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410353679.6A CN118394731A (en) | 2024-03-26 | 2024-03-26 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410353679.6A CN118394731A (en) | 2024-03-26 | 2024-03-26 | Data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118394731A true CN118394731A (en) | 2024-07-26 |
Family
ID=91990756
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410353679.6A Pending CN118394731A (en) | 2024-03-26 | 2024-03-26 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118394731A (en) |
-
2024
- 2024-03-26 CN CN202410353679.6A patent/CN118394731A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12015666B2 (en) | Systems and methods for distributing partial data to subnetworks | |
| US8756297B2 (en) | Energy-efficient content caching with custodian-based routing in content-centric networks | |
| CN110572422B (en) | Data downloading method, device, equipment and medium | |
| US20160182626A1 (en) | System and method for discovering a lan synchronization candidate for a synchronized content management system | |
| CN113517985B (en) | File data processing method and device, electronic equipment and computer readable medium | |
| JP2018049653A (en) | Cache management | |
| US10574703B1 (en) | Content delivery employing multiple security levels | |
| CN116303608A (en) | Data processing method and device for application service | |
| CN110505074B (en) | Application modularization integration method and device | |
| CN112929414A (en) | Upgrade package downloading method and device, electronic equipment and storage medium | |
| US11853806B2 (en) | Cloud computing platform that executes third-party code in a distributed cloud computing network and uses a distributed data store | |
| CN113595927A (en) | Method and device for processing mirror flow in bypass mode | |
| CN111339206B (en) | Block chain-based data sharing method and device | |
| CN110798495B (en) | Method and server for end-to-end message push in cluster architecture mode | |
| US20220012110A1 (en) | Networking-related system call interception and modification | |
| CN110909030A (en) | Information processing method and server cluster | |
| JP2016099969A (en) | Information processing apparatus, data storage system, and data storage method | |
| CN118394731A (en) | Data processing method and device | |
| CN110247847B (en) | Method and device for back source routing between nodes | |
| CN109213815B (en) | Method, device, server terminal and readable medium for controlling execution times | |
| US10778660B1 (en) | Managing multiple producer consumer—systems with non-identical idempotency keys | |
| US20240106889A1 (en) | Data resource storage method and apparatus, data resource query method and apparatus, and electronic device | |
| CN104601707A (en) | Network disk data processing method and device | |
| CN109600452B (en) | Server cluster, message pushing method and related server | |
| KR20150011087A (en) | Distributed caching management method for contents delivery network service and apparatus therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |