CN110737536B - Message storage method and message reading method based on shared memory - Google Patents
Message storage method and message reading method based on shared memory Download PDFInfo
- Publication number
- CN110737536B CN110737536B CN201910885530.1A CN201910885530A CN110737536B CN 110737536 B CN110737536 B CN 110737536B CN 201910885530 A CN201910885530 A CN 201910885530A CN 110737536 B CN110737536 B CN 110737536B
- Authority
- CN
- China
- Prior art keywords
- message
- storage
- queue
- storage sub
- read
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 133
- 230000015654 memory Effects 0.000 title claims abstract description 48
- 238000004891 communication Methods 0.000 claims abstract description 5
- 230000003993 interaction Effects 0.000 abstract description 19
- 238000012423 maintenance Methods 0.000 abstract description 6
- 230000005540 biological transmission Effects 0.000 abstract description 4
- 238000005516 engineering process Methods 0.000 abstract 1
- 230000008569 process Effects 0.000 description 58
- 238000004364 calculation method Methods 0.000 description 19
- 238000012545 processing Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000009825 accumulation Methods 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- FFBHFFJDDLITSX-UHFFFAOYSA-N benzyl N-[2-hydroxy-4-(3-oxomorpholin-4-yl)phenyl]carbamate Chemical compound OC1=C(NC(=O)OCC2=CC=CC=C2)C=CC(=C1)N1CCOCC1=O FFBHFFJDDLITSX-UHFFFAOYSA-N 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A message storage method and a message reading method based on a shared memory are provided, and a storage space, a message storage method and a message reading method are provided. A message storage method is used for storing a message to be stored in a storage space, the storage space comprises a message control area and a message storage area, the message storage area is used for storing data, the message storage area comprises a plurality of storage sub-queues, and the message control area is used for storing an array for recording read-write information of the storage sub-queues. A message reading method is used for reading the message stored in a storage space according to the message storage method, and the message storage and reading realize the high-speed interaction of the message in a memory copy mode, so that the problems that the message transmission efficiency is low, the real-time property cannot be met under the high concurrency low delay requirement in the existing communication interaction technology, the configuration is inflexible, the learning and maintenance cost is high and the like are solved.
Description
Technical Field
The invention relates to the field of support in the telecommunication industry, in particular to a message storage method and a message reading method based on a shared memory.
Background
Telecommunication is a main form of communication nowadays, as competition among domestic telecommunication industries is increasingly strong, telecommunication operators are increasingly focusing on the operation condition of a service support system, and a telecommunication charging system is used as a core of the telecommunication service support system, and accuracy of the telecommunication charging system is directly related to customer service quality and income of the telecommunication operators. At present, the multi-process interaction mode in the charging system of the telecommunication industry mainly carries out interaction through files, databases and message middleware. The file interaction mode is that data is written into a file server through a former process and is stored in a physical disk, the latter process reads the file from the file server and fetches the data from the physical disk, the interaction process involves the file opening and closing of a read-write end, and finally the read-write of the physical disk is operated. The database interaction mode is that the former process writes data into the database server, the latter process reads data from the database server, and the different processes exchange data through the same table connected with the same database server. The message middleware interaction mode is that data of front and back processes are interacted through middleware of a message queue such as kafka, rabbitMQ, a writing process is used as a generator to generate data, a reading process is used as a consumer to process the data, and data exchange is carried out between different processes through a message server. The message generator sends the message to the message server, which deposits the message in several queues, and at the appropriate time, the message server pushes the message to the message generator.
However, a common file server is needed in the file interaction process, and the file server usually has risks, because the file may be tampered, deleted, or leaked, etc., under the condition of large data volume, the file transmission can be performed, the interaction mode is simple, the scheme is not overtime, the network bandwidth is not occupied, but if the single file has a large data volume, the file writing process needs to be completed after waiting for writing a certain amount, and the file reading process needs to wait for a longer time, so that the mode is not suitable for real-time service; compared with the file interaction mode, the interaction is simpler because the same database is used in the database interaction mode, however, the database relates to the deployment and maintenance of a server, the maintenance cost is high, the database interaction modes basically have a lock mechanism, and the database interaction modes are remotely accessed through a network, so that the method is not suitable for high-concurrency low-delay message data interaction; the message middleware interaction mode has strong functions and high efficiency, but has insufficient flexibility, and the development and maintenance of the message middleware are required to be familiar with the characteristics and requirements of the message middleware to be used better based on clustered deployment, so that a developer is required to learn relevant basic knowledge, the specific configuration and implementation details of the message middleware are required to learn, the learning cost is high, and under the condition of large data volume, the message can be backlogged, so that the message is delayed, lost and even the message middleware is crashed.
Disclosure of Invention
In view of this, the invention aims to solve the problems that the existing process interaction mode cannot meet the requirement of 'real-time', the configuration is not flexible enough, the use is inconvenient, and the learning and maintenance costs are high under the high concurrency low delay requirement, thereby providing a message storage method and a message reading method based on a shared memory.
According to a first aspect, an embodiment of the present invention provides a storage space, including a message control area and a message storage area, where the message storage area is used to store data, and the message storage area includes a plurality of storage sub-queues; the message control area is used for storing and recording an array of read-write information of the storage sub-queue.
With reference to the first aspect, in a first implementation manner of the first aspect, the plurality of storage sub-queues are distributed according to a two-dimensional array manner.
With reference to the first aspect, in a second implementation manner of the first aspect, the message is stored in each storage sub-queue according to a first-in first-out principle.
According to a second aspect, an embodiment of the present invention provides a message storing method, configured to store a message to be stored in a storage space, where the storage space includes the message control area and a message storage area, where the message storage area is used to store data, and where the message storage area includes a plurality of storage sub-queues; the message control area is used for storing and recording an array of read-write information of the storage sub-queue, and comprises: acquiring the length of a message to be stored; acquiring a starting address of the storage sub-queue corresponding to the message to be stored and a message storage position number of the storage sub-queue according to the array of the message control area; determining the initial address offset of the storage sub-queue according to the message storage position number and the length of the message to be stored; calculating the storage address of the message according to the starting address of the storage sub-queue and the starting address offset of the storage sub-queue; and storing the message to be stored according to the storage address.
With reference to the second aspect, in a first implementation manner of the second aspect, the obtaining, according to the array of the message control area, a start address of the storage sub-queue corresponding to the message to be stored includes: acquiring the lengths of all the storage sub-queues before the storage sub-queues of the message to be stored; and obtaining the starting address of the storage sub-queue of the message to be stored according to the length.
With reference to the second aspect, in a second implementation manner of the second aspect, obtaining a message storage location number of the storage sub-queue includes: acquiring the message number of the storage sub-queue of the message to be stored; acquiring the capacity of the storage sub-queue; and obtaining the message storage position number of the storage sub-queue according to the message number of the storage sub-queue of the message to be stored and the capacity of the storage sub-queue.
With reference to the second aspect, in a third implementation manner of the second aspect, the calculating a starting address offset of the storage sub-queue according to the message storage location number includes: acquiring the length of a single message; and determining the initial address offset of the storage sub-queue according to the single message length of the message storage position number.
According to a third aspect, an embodiment of the present invention provides a message reading method, configured to read a message stored in a message storage method according to the second aspect or any embodiment of the second aspect, including: acquiring the length of a message to be read; acquiring a start address of the storage sub-queue corresponding to the information to be read and a position number of the information to be read according to the array of the information control area; determining the initial address offset of the storage sub-queue corresponding to the information to be read according to the position number of the information to be read and the length of the information to be read; determining a reading address of the message according to the initial address of the storage sub-queue corresponding to the information to be read and the initial address offset of the storage sub-queue corresponding to the information to be read; and reading the information to be read according to the reading address.
With reference to the third aspect, in a first implementation manner of the third aspect, the obtaining, according to the array of the message control area, a start address of the storage sub-queue corresponding to the to-be-read information includes: acquiring the actual concurrency of the information to be read; determining the lengths of all the storage sub-queues before the storage sub-queues corresponding to the information to be read according to the actual concurrency; and determining the starting address of the storage sub-queue corresponding to the information to be read according to the lengths of all the storage sub-queues before the storage sub-queue corresponding to the information to be read.
With reference to the third aspect, in a second implementation manner of the third aspect, acquiring the location number of the message to be read includes: acquiring the number of the information to be read of the storage sub-queue corresponding to the information to be read; acquiring the capacity of the storage sub-queue; and obtaining the reading position number in the storage sub-queue corresponding to the information to be read according to the information to be read of the storage sub-queue corresponding to the information to be read and the capacity of the storage sub-queue.
With reference to the third aspect, in a third implementation manner of the third aspect, determining, according to the position number of the message to be read and the length of the message to be read, a start address offset of the storage sub-queue corresponding to the message to be read, includes: acquiring the length of a single message; and determining the initial address offset of the storage sub-queue in which the information to be read is located according to the storage position number of the information to be read in the storage sub-queue and the length of the single message.
According to a third aspect, an embodiment of the present invention provides a computer device, comprising: the memory and the processor are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions, thereby executing the memory space in the first aspect or any implementation manner of the first aspect, executing the message storage method in the second aspect or any implementation manner of the second aspect, and executing the message reading method in the third aspect or any implementation manner of the third aspect.
According to a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having stored thereon computer instructions for causing the computer to execute the storage space of the first aspect or any implementation of the first aspect, to execute the message storing method of the second aspect or any implementation of the second aspect, and to execute the message reading method of the third aspect or any implementation of the third aspect.
The technical scheme of the invention has the following advantages:
1. the invention provides a message storage method and a message reading method based on a shared memory, which are characterized in that a message to be stored is stored in a shared memory space, wherein the shared memory space comprises a message control area and a message storage area, and the message storage area is used for storing the message according to the message type and the concurrency and is virtualized into a plurality of storage sub-queues; the message control area is used for storing and recording an array of read-write information of the storage sub-queues, controls the read-write positions of messages between different processes/threads by calculating the message storage addresses and the message reading addresses, does not need a locking mechanism, realizes high-speed transmission of message interaction, satisfies the real-time performance under the requirement of high concurrency and low delay, divides the shared memory space into a plurality of sub-message queues, satisfies the requirements of different addresses of the message storage, has flexible configuration, can realize interactive transmission of messages between multiple processes and multiple threads, realizes high-efficiency calculation of access addresses, and improves the efficiency of rapid processing of the same type of messages.
2. The invention provides a message storage method and a message reading method based on a shared memory, which realize message storage based on a two-dimensional storage sub-queue by calculating address offset. Based on the message storage of the two-dimensional storage sub-queues, the plurality of writing processes/threads can be writing processes/threads of different types and writing processes/threads of different range segments of the same type of messages, if the same message is generated by the plurality of writing processes/threads, the message can be corresponding to different message storage sub-queues by increasing the message source number, the message generation time and the message route based on the source number, the message storage of the message storage sub-queues meets the requirements of different addresses, complex route calculation is not carried in, and the method is convenient to use and low in learning and maintenance cost.
3. The invention provides a message storage method and a message reading method based on a shared memory, when reading a message based on the accumulation condition of a message queue, if the number of reading processes/threads is increased or reduced, the message can be correctly read according to the rule of calculating the read message address and the method of polling and reading a plurality of message storage sub-queues as long as the number is within the maximum concurrency of the message reading. The sequence reading or parallel reading is carried out among all the processes/threads of the reading storage sub-queue, the sequence number of the reading message sub-queue and the complete address of the current reading message are confirmed through the current message type, the message sequence number and the actual concurrency of the reading processes/threads, lock control and complex route calculation are not needed, and the reading efficiency of the message queue is improved.
4. The invention provides a message storage method and a message reading method based on a shared memory, which greatly expand the application range of a lock-free message queue based on the shared memory in practical application, read and write messages are all through a memory copying method, the reading and writing of the messages do not involve the reading of a plurality of blocks of memories, the lock-free control of the reading and writing is realized through the internal routing calculation of the positions of read and write data, the reading of a plurality of processes of single message data is supported, and the high-speed message storage and reading are realized.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a shared memory according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a storage sub-queue in a storage space according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a storage principle of a storage sub-queue according to an embodiment of the present invention;
FIG. 4 is a flowchart of a specific example of a message storage method in an embodiment of the present invention;
FIG. 5 is a flowchart of a specific example of the message storing method corresponding to S22 in the embodiment of the present invention;
FIG. 6 is a flowchart of another specific example of the message storing method corresponding to S22 in the embodiment of the present invention;
FIG. 7 is a flowchart of a specific example of the message storing method corresponding to S13 in the embodiment of the present invention;
FIG. 8 is a flowchart of a specific example of a message reading method in an embodiment of the present invention;
fig. 9 is a flowchart of a specific example of the message reading method corresponding to S22 in the embodiment of the present invention;
fig. 10 is a flowchart of another specific example of the message reading method corresponding to S22 in the embodiment of the present invention;
fig. 11 is a flowchart of a specific example of the message reading method corresponding to S23 in the embodiment of the present invention;
fig. 12 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In addition, the technical features of the different embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
Example 1
The embodiment provides a storage space, firstly, the storage space is divided into a message control area and a message storage area by applying for the storage space, the message storage area is used for storing data and comprises a plurality of storage sub-queues, the message control area is used for storing and recording an array of read-write information of the storage sub-queues, and the storage sub-queues comprise information such as storage sub-queue capacity, writing process/thread writing message sequences, reading process/thread reading message sequences, sub-message sizes, starting addresses of the storage sub-queues and the like. As shown in fig. 1, assuming that there are now process 1 and process 2, the two processes are mapped to the user-level page table through the virtual address space and then to the same block of memory area of the physical memory through the user-level page table, and inter-process communication is achieved through the shared memory area.
The plurality of storage sub-queues are stored in the storage space and are stored in a two-dimensional array mode, each storage sub-queue corresponds to one piece of control information, the two-dimensional storage sub-queues are in one-to-one correspondence, the storage distribution is shown in fig. 2, the first dimension corresponds to the message type, and the second dimension corresponds to the maximum reading concurrency. Setting storage sub-queues to store M parts (class) of messages in fragments, storing each part (class) of messages in fragments according to the maximum concurrence thread T, storing M times T storage sub-queues corresponding to M times T sub-Queue information by a two-dimensional array cycle_queue [ M ] [ T ], wherein each cycle_queue comprises the capacity value_capacity of the sub-message Queue, a producer writes a message sequence Wr_sequnce_No, and a consumer reads information such as the message sequence Rd_sequnce_No, the sub-message Size Size, the Start address start_addr of the sub-message Queue and the like. If M is 1 and T is 1, storing a single annular message queue, storing a message, and reading and writing by a process/thread; if M is 1 and T is greater than 1, the message is a type (share) message, and at most T processes/threads can be read; if M is greater than 1, T is 1, namely M types (parts) of messages, each type of message is read and written by a process/thread, and the storage address and the read address of message data are calculated through routing according to the storage position of the routing control message.
The storage principle of the storage sub-queue is shown in fig. 3, the storage sub-queue is a first-in first-out circular storage area, the messages are sent and received in sequence, and once the messages are processed by a reading process/thread, the messages are deleted from the queue. Assuming that there is one producer (writing messages to the store sub-Queue) and one consumer (reading messages from the store sub-Queue), the store sub-Queue capacity value is set to N (i.e., the store sub-Queue may store N messages), each message has a Size of Size, the producer writes a message sequence number wr_sequnce_no, the consumer reads a message sequence number rd_sequnce_no, a write message number wr_msg_count, a read message number rd_msg_count, and wr_sequnce_no=wr_msg_count-1, and rd_sequnce_no=msg_count Rd-1, where the sequence number is message number-1, i.e., the message number counts from 1 and the sequence number counts from 0. Wr_sequnce_No needs to be equal to or greater than Rd_sequnce_No according to the first-in first-out (first in first out, FIFO) principle. The Wr_sequnce_No and Rd_sequnce_No are obtained by taking the remainder of the queue_capability, and the Queue position sequence number queue_index of the message is obtained.
The queue_index value loops from 0 to N-1, and the address where the message can be stored is calculated: offset+queue_index Size;
the producer's request_index based on Wr_sequnce_No+1 indicates the next message store location number, and if the request_index is the same as the request_index of Rd_sequnce_No, it indicates that the message store sub-queue is full.
The consumer obtains the queue_index based on Rd_sequnce_No+1 to indicate that the next message position sequence number is read, and if the queue_index is larger than the queue_index obtained by Wr_sequnce_No, the message storage sub-queue is read.
The storage space, the message storage area and the message control area provided by the embodiment store data, the message control area stores an array for recording read-write information of the storage sub-queues, the message queues are realized based on the shared memory, and the method has the characteristics of quick message read-write, flexible distribution rule, quick and simple use, and the read-write information is copied by the memory, so that the method does not involve reading of a plurality of memories, and the processing efficiency is high.
Example 2
The embodiment provides a message storage method, which can be used in a billing system in the telecommunications industry and needs to realize high concurrency low-delay real-time billing requirements, and realize that a message is stored in a storage space described in embodiment 1, firstly, the message to be stored is stored in the storage space, then, by obtaining the length of the message to be stored, the starting address of a storage sub-queue corresponding to the stored message and the message storage position number of the storage sub-queue, the starting address offset of the storage sub-queue is determined, and then, the storage address of the message is determined, and the message to be stored is stored, as shown in fig. 4, the message storage method comprises the following steps:
s21: the length of the message to be stored is obtained.
S22: and acquiring the starting address of the storage sub-queue corresponding to the message to be stored and the message storage position number of the storage sub-queue according to the array of the message control area. The initial address is the memory location corresponding to the storage sub-queue for starting to store the message, the message storage is started from the corresponding memory location, and the message storage location number is the block number of the storage sub-queue and is used for the storage location of the corresponding message in the storage sub-queue.
S23: and determining the initial address offset of the storage sub-queue according to the message storage position number and the length of the message to be stored. And multiplying the message storage position number by the length of the message to be stored to obtain the initial address offset of the storage sub-queue.
S24: and calculating the storage address of the message according to the starting address of the storage sub-queue and the starting address offset of the storage sub-queue. The sum of the start address of the storage sub-queue and the start address offset of the storage sub-queue is the storage address of the message.
S25: and storing the message to be stored according to the storage address.
According to fig. 2, assuming that the capacities of each sub-queue are the same and are all N, the single message length Size, the applied storage space Size is mapped to be from the applied storage Start address ofset, the maximum number of concurrent threads of each type (share) of message is T, the type number of the currently stored message is X (i.e. the first dimension value of the two-dimensional queue is X), the message is required to be written to the Y-th storage sub-queue (i.e. the second dimension value of the two-dimensional queue is Y) within the T concurrency, the Start address of the storage sub-queue is stored in the start_addr under the information of cycle_queue [ X-1] [ Y-1], and the stored calculated value is: offset+ (X-1) T N size+ (Y-1) N Size, and the calculation of the start address is completed when the message queue is initialized.
The starting address of the store sub-queue is: ofSet+ (X-1) T N Size+ (Y-1) N Size, wherein the X value range is 1-M, the Y value range is 1-T, when a certain message is actually stored, X is transmitted through a writing process/thread, the X value indicates that the process/thread needs to confirm which type of message is written, the Y value is obtained according to the 'Key' of the written message, for example, the message sequence Number is serial_Number, the serial_Number is left for T or the total message Number Wr_msg_ALC of a certain type of message is left for T, the value is obtained by +1, or the value is obtained by polling according to the range of 1-T, and the start_addr value of the cycle_queue [ X-1] [ Y-1] is the starting address of the sub-queue of the message to be written;
the message serial number written in the current queue (written message number-1) is surplus according to the queue capacity, and a message storage position number queue_index is obtained;
the starting address offset of the store sub-queue is: queue_index Size;
determining the storage address of the message according to the starting address of the storage sub-queue and the starting address offset of the storage sub-queue: offset+ (X-1) T N size+ (Y-1) N Size + queue_index Size, namely, cycle_queue [ X-1] [ Y-1] start_addr+queue_index_size.
According to the message storage method provided by the embodiment, the storage sub-queues are stored by calculating the address offset, and the plurality of writing processes can be messages with different types or messages with the same type and different range segments. If the same message is written by a plurality of processes, the message can be written into the storage sub-queues without lock by increasing the source number of the message and the generation time of the message, and the message is routed based on the source number and corresponds to different message storage sub-queues, so that the storage speed of the message is improved.
Optionally, in some embodiments of the present invention, as shown in fig. 5, S22 includes the steps of:
s221: and acquiring the lengths of all the storage sub-queues before the storage sub-queues of the message to be stored, wherein the lengths are calculated according to the number of writing processes/threads of the storage sub-queues corresponding to the message to be stored and the message to be stored of different types of messages, namely (X-1) TxNSize+ (Y-1) NSize.
S222: and obtaining the starting address of the storage sub-queue of the message to be stored according to the length. The sum of the start address of the applied storage space and the length of all the storage sub-queues before the storage sub-queue of the message to be stored is the start address of the storage sub-queue of the message to be stored, namely offset+ (X-1) tsize+ (Y-1) tsize.
The initial address of the storage sub-queue is obtained by calculating the lengths of all the storage sub-queues before the storage sub-queues where the information to be stored is located are obtained, complex route calculation is not carried out, flexibility and universality are facilitated, and efficient calculation of the storage address is realized.
Optionally, in some embodiments of the present invention, as shown in fig. 6, S22 further includes the following steps:
s223: and acquiring the message number of the storage sub-queue of the message to be stored. The number of messages refers to the total number of messages that are currently to be stored.
S224: and acquiring the capacity of the storage sub-queue.
S225: and obtaining the message storage position number of the storage sub-queue according to the message number of the storage sub-queue of the message to be stored and the capacity of the storage sub-queue. The storage location number is obtained by taking a remainder of the capacity of the storage sub-queue according to the write message sequence number (message number-1).
The message storage position number of the storage sub-queue is obtained according to the write message sequence number of the storage sub-queue of the message to be stored and the capacity surplus of the storage sub-queue, so that the calculation is simple, flexible and universal, and the efficient calculation of the storage address is realized.
Alternatively, in some embodiments of the present invention, as shown in fig. 7, S23 includes the steps of:
s231: a single message length is obtained.
S232: and determining the initial address offset of the storage sub-queue according to the message storage position number and the single message length. The product of the message storage position number and the length of a single message is the initial address offset of the storage sub-queue.
And determining the initial address offset of the storage sub-queue according to the message storage position number of the storage sub-queue and the length of a single message, so that complex route calculation is not involved, and the calculation speed of the storage address is improved.
Example 3
The present embodiment provides a message reading method for reading a message stored by using the method in embodiment 2, as shown in fig. 8, including:
s31: the length of the message to be read is obtained. And further acquiring the starting position of the storage sub-queue of the message to be read in the memory space according to the length of the message to be read.
S32: and acquiring the starting address of the storage sub-queue corresponding to the information to be read and the position number of the information to be read according to the array of the information control area. The initial address is the memory location corresponding to the storage sub-queue where the message is located, and is used for determining the storage address corresponding to the message to be read, the location number of the message to be read is the location number in the storage sub-queue where the message to be read is located, and is used for corresponding to the storage address of the message in the storage sub-queue, and the message is read from the storage address.
S33: and determining the initial address offset of the storage sub-queue corresponding to the information to be read according to the position number of the information to be read and the length of the information to be read. The position number of the message to be read is multiplied by the length of the message to be read to obtain the initial address offset of the storage sub-queue corresponding to the message to be read.
S34: and determining the reading address of the message according to the starting address of the storage sub-queue corresponding to the information to be read and the starting address offset of the storage sub-queue corresponding to the information to be read. And the sum of the initial address of the storage sub-queue corresponding to the message to be read and the initial address offset of the storage sub-queue corresponding to the message to be read is the read address of the message.
S35: and reading the information to be read according to the reading address.
According to fig. 2, assuming that the capacity of each storage sub-queue is N, the length of a single message is Size, the Size of the applied storage space is mapped to a calculated value of offset+ (X-1) t×size plus (Y-1) n×size plus (Y-1) X (X is the first dimension of the two-dimensional queue), the type value of the read message is X (P < =t), the message of the Y-th storage sub-queue (the second dimension of the two-dimensional queue) is read in a certain type of message X, and the Start address of the storage sub-queue, start_addr, cycle_queue [ X-1] [ Y-1], is a calculated value of offset+ (X-1) t×n×size plus (Y-1) n×size plus (P < =t), and the number of the current read process/thread is Z (1-P).
The starting address of the store sub-queue is: ofset+ (X-1) tsize+ (Y-1) nsize, wherein X is in the range of 1-M, Y is in the range of 1-T, when a message is actually read, X is input through the reading process/thread, the X value indicates that the process/thread needs to confirm which type of message is read, the Y value is calculated according to the maximum concurrency T, the actual concurrency P, the current reading process/thread number Z, the Y value indicates that when the actual concurrency is P, the reading process/thread of the Z number reads the value of the storage sub-queue, and the start_addr value of the cycle_queue [ X-1] [ Y-1] is the starting address of the sub-queue of the message to be read according to the obtained X value and Y value, except that when reading, Y is not necessarily a single value, but may be several values within a range. When the writing process/thread stores the information, the address writing is calculated according to the maximum concurrency degree T, the reading process/thread calculates the address reading according to the actual concurrency degree P, each reading process/thread needs to acquire the information from T/P queues, the sequence number range of the read information storage sub-queue is [ (Z-1) & ltT/P) & gt, Z is & ltT/P) -1], namely the range of Y-1 values, and the read information can be correctly read by polling according to the sequence number range of the queue. For example, when T is 10, P is 5, Z is 2, and the calculated value is [2,3], namely, the process/thread reading message with the number 2 needs to read two storage sub-queues with the number 2 and the number 3 (the number starts from 0), and when the number 2 or the number 3 of the storage sub-queue is read, the starting address of the number 2 or the number 3 storage sub-queue is calculated first; t is 10, P is 1, Z is 1, i.e. number 1 process/thread needs to read 1-10 messages of the store sub-queues (store sub-queues with sequence numbers 0-9); t is 10, p is 10, and z is 1, then this indicates reading from 1 store sub-queue (store sub-queue with sequence number 0) for the same class of messages. When each storage sub-queue reads the message, the offset address of the storage sub-queue needs to be firstly obtained, and then the message reading address is obtained according to the number of the messages of the current queue read in each storage sub-queue. For simple calculation, usually T and P are in a multiple relationship, if the T and P are not integer multiple, the queue interval values calculated by front and back consumers are ensured to be continuous and are not coincident;
the message sequence number (the number of the read messages-1) read by the current queue is surplus according to the queue capacity, and the position number queue_index of the message to be read is obtained;
the initial address offset of the storage sub-queue corresponding to the message to be read is as follows: queue_index Size;
determining the reading address of the message according to the starting address of the storage sub-queue corresponding to the reading message and the starting address offset: offset+ (X-1) T N size+ (Y-1) N size+queue_index Size.
According to the message reading method, through determining the reading address corresponding to the message, when a plurality of reading processes/threads read the message, the reading addresses of the message read by each reading process/thread are different, and the message reading efficiency is improved.
Optionally, in some embodiments of the present invention, as shown in fig. 9, S32 includes the steps of:
s321: and acquiring the actual concurrency of the message to be read, wherein the actual concurrency refers to the concurrency process/thread number of the message to be read.
S322: and determining the lengths of all the storage sub-queues before the storage sub-queues corresponding to the information to be read according to the actual concurrency, wherein the lengths are calculated according to the concurrency process/thread number of the information to be read and the different types of information stored in the storage sub-queues corresponding to the information to be read.
S323: and determining the starting address of the storage sub-queue corresponding to the information to be read according to the lengths of all the storage sub-queues before the storage sub-queue corresponding to the information to be read.
Under the condition of message storage sub-queue message accumulation, if the number of reading processes/threads is increased or reduced, the messages can be correctly read according to the calculation rule of the message reading address and the method of polling a plurality of message storage sub-queues, the message reading can be sequential reading or parallel reading, the sequential reading refers to one sub-queue, and a plurality of reading programs sequentially process the same message: the process/thread 1 is read out, the process/thread 2 is read continuously, the process/thread 2 is read out, the process/thread 3 is read out again, and the sequential reading is controlled by reading the position number of the message; parallel reading is to read messages from different storage sub-queues, namely a plurality of sub-queues of concurrency T.
The initial address of the message storage sub-queue corresponding to the message to be read is calculated according to the lengths of all the storage sub-queues before the storage sub-queues of the message to be read are obtained, complex route calculation is not carried out, flexibility and universality are facilitated, and efficient calculation of the read address is realized.
Optionally, in some embodiments of the present invention, as shown in fig. 10, S32 further includes the following steps:
s324: and acquiring the number of the information to be read of the storage sub-queue corresponding to the information to be read, wherein the number of the information to be read refers to the total number of the information to be read currently.
S325: and acquiring the capacity of the storage sub-queue.
S326: and obtaining the reading position number in the storage sub-queue corresponding to the information to be read according to the information to be read of the storage sub-queue corresponding to the information to be read and the capacity of the storage sub-queue. The read position number is obtained by taking the remainder of the capacity of the storage sub-queue according to the read message sequence number (the number of messages to be read-1).
The reading message position number of the storage sub-queue is obtained according to the reading message sequence number of the storage sub-queue corresponding to the information to be read and the capacity of the storage sub-queue, and the method is simple in calculation, flexible and universal, and achieves efficient calculation of the message reading address.
Alternatively, in some embodiments of the present invention, as shown in fig. 11, S33 includes:
s331: a single message length is obtained.
S332: and determining the initial address offset of the storage sub-queue corresponding to the information to be read according to the storage position number in the storage sub-queue corresponding to the information to be read and the length of the single message. The product of the storage position number in the storage sub-queue corresponding to the message to be read and the length of the single message is the initial address offset of the storage sub-queue corresponding to the message to be read.
And determining the initial address offset of the storage sub-queue corresponding to the message to be read according to the message storage position number of the storage sub-queue corresponding to the message to be read and the length of the single message, so that complex route calculation is not involved, and the calculation speed of the message reading address is improved.
Example 4
The embodiment of the present invention further provides a computer device, as shown in fig. 12, which may include a processor 41 and a memory 42, where the processor 41 and the memory 42 may be connected by a bus or other means, and in fig. 12, the connection is exemplified by a bus.
The processor 41 may be a central processing unit (Central Processing Unit, CPU). The processor 41 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above.
The memory 42 is used as a non-transitory computer readable storage medium for storing non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions corresponding to the message storing method and the message reading method in the embodiments of the present invention. The processor 41 executes various functional applications of the processor and data processing by running non-transitory software programs and instructions stored in the memory 42, i.e., implements the shared memory-based message storage method and the message reading method in the above-described method embodiments.
Memory 42 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created by the processor 41, etc. In addition, memory 42 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 42 may optionally include memory located remotely from processor 41, which may be connected to processor 41 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more modules are stored in the memory 42, which when executed by the processor 41, perform the shared memory based message storage method and the message reading method of the embodiments shown in fig. 1-11.
The details of the above-mentioned computer device may be understood correspondingly with respect to the corresponding relevant descriptions and effects in the embodiments shown in fig. 1 to 11, and will not be repeated here.
It will be appreciated by those skilled in the art that implementing all or part of the above-described embodiment method may be implemented by a computer program to instruct related hardware, where the program may be stored in a computer readable storage medium, and the program may include the above-described embodiment method when executed. Wherein the storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a Flash Memory (Flash Memory), a Hard Disk (HDD), or a Solid State Drive (SSD); the storage medium may also comprise a combination of memories of the kind described above.
Although embodiments of the present invention have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations are within the scope of the invention as defined by the appended claims.
Claims (9)
1. A message storage method, configured to store a message to be stored in a storage space, where the storage space includes a message control area and a message storage area, where the message storage area is used to store data, and the message storage area includes a plurality of storage sub-queues; the message control area is used for storing and recording an array of read-write information of the storage sub-queues, wherein the storage sub-queues are distributed in a two-dimensional array mode; the message is stored in each storage sub-queue according to the first-in first-out principle, which is characterized by comprising the following steps:
acquiring the length of a message to be stored;
acquiring a starting address of the storage sub-queue corresponding to the message to be stored and a message storage position number of the storage sub-queue according to the array of the message control area;
determining the initial address offset of the storage sub-queue according to the message storage position number and the length of the message to be stored;
calculating the storage address of the message according to the starting address of the storage sub-queue and the starting address offset of the storage sub-queue;
storing the message to be stored according to the storage address;
the obtaining the message storage position number of the storage sub-queue includes: acquiring the message number of the storage sub-queue of the message to be stored; acquiring the capacity of the storage sub-queue; and obtaining the message storage position number of the storage sub-queue according to the message number of the storage sub-queue of the message to be stored and the capacity of the storage sub-queue.
2. The method for storing messages according to claim 1, wherein said obtaining, according to the array of the message control area, the start address of the storage sub-queue corresponding to the message to be stored includes:
acquiring the lengths of all the storage sub-queues before the storage sub-queues of the message to be stored;
and obtaining the starting address of the storage sub-queue of the message to be stored according to the length.
3. The message storage method according to claim 1 or 2, wherein the calculating the start address offset of the storage sub-queue according to the message storage location number includes:
acquiring the length of a single message;
and determining the initial address offset of the storage sub-queue according to the single message length of the message storage position number.
4. A message reading method for reading a message stored according to the message storing method of any one of claims 1 to 3, comprising:
acquiring the length of a message to be read;
acquiring a start address of the storage sub-queue corresponding to the information to be read and a position number of the information to be read according to the array of the information control area;
determining the initial address offset of the storage sub-queue corresponding to the information to be read according to the position number of the information to be read and the length of the information to be read;
determining a reading address of the message according to the initial address of the storage sub-queue corresponding to the information to be read and the initial address offset of the storage sub-queue corresponding to the information to be read;
and reading the information to be read according to the reading address.
5. The method for reading a message according to claim 4, wherein the obtaining, according to the array of the message control area, the start address of the storage sub-queue corresponding to the message to be read includes:
acquiring the actual concurrency of the information to be read;
determining the lengths of all the storage sub-queues before the storage sub-queues corresponding to the information to be read according to the actual concurrency;
and determining the starting address of the storage sub-queue corresponding to the information to be read according to the lengths of all the storage sub-queues before the storage sub-queue corresponding to the information to be read.
6. The message reading method according to claim 4 or 5, wherein obtaining the location number of the message to be read comprises:
acquiring the number of the information to be read of the storage sub-queue corresponding to the information to be read;
acquiring the capacity of the storage sub-queue;
and obtaining the reading position number in the storage sub-queue corresponding to the information to be read according to the information to be read of the storage sub-queue corresponding to the information to be read and the capacity of the storage sub-queue.
7. The method for reading a message according to claim 4, wherein determining the start address offset of the storage sub-queue corresponding to the message to be read according to the position number of the message to be read and the length of the message to be read comprises:
acquiring the length of a single message;
and determining the initial address offset of the storage sub-queue in which the information to be read is located according to the storage position number of the information to be read in the storage sub-queue and the length of the single message.
8. A computer device, comprising:
a memory and a processor in communication with each other, the memory having stored therein computer instructions which, upon execution, perform the message storing method of any one of claims 1-3 and the message reading method of any one of claims 4-7.
9. A computer-readable storage medium storing computer instructions for causing the computer to perform the message storing method according to any one of claims 1 to 3 and the message reading method according to any one of claims 4 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910885530.1A CN110737536B (en) | 2019-09-19 | 2019-09-19 | Message storage method and message reading method based on shared memory |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910885530.1A CN110737536B (en) | 2019-09-19 | 2019-09-19 | Message storage method and message reading method based on shared memory |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110737536A CN110737536A (en) | 2020-01-31 |
| CN110737536B true CN110737536B (en) | 2024-03-12 |
Family
ID=69269345
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910885530.1A Active CN110737536B (en) | 2019-09-19 | 2019-09-19 | Message storage method and message reading method based on shared memory |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110737536B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111416858B (en) * | 2020-03-16 | 2023-04-07 | 广州市百果园信息技术有限公司 | Media resource processing platform, method, device and server |
| CN111416825A (en) * | 2020-03-24 | 2020-07-14 | 河南信大网御科技有限公司 | Inter-thread lock-free log management method and system, terminal and storage medium |
| CN111427859B (en) * | 2020-03-25 | 2024-04-05 | 京东科技控股股份有限公司 | Message processing method and device, electronic equipment and storage medium |
| CN113778914B (en) * | 2020-06-09 | 2024-06-18 | 华为技术有限公司 | Apparatus, method, and computing device for performing data processing |
| CN111787107B (en) * | 2020-06-30 | 2023-03-24 | 北京字节跳动网络技术有限公司 | Message processing method and device, readable medium and electronic equipment |
| CN111984198B (en) * | 2020-08-25 | 2024-02-27 | 北京一起教育信息咨询有限责任公司 | Message queue implementation method and device and electronic equipment |
| CN113672400A (en) * | 2021-08-26 | 2021-11-19 | 深信服科技股份有限公司 | Data processing method, device and equipment and readable storage medium |
| CN113778948A (en) * | 2021-09-01 | 2021-12-10 | 中国人民银行清算总中心 | Message persistent storage method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1859325A (en) * | 2006-02-14 | 2006-11-08 | 华为技术有限公司 | News transfer method based on chained list process |
| CN104346229A (en) * | 2014-11-14 | 2015-02-11 | 国家电网公司 | Processing method for optimization of inter-process communication of embedded operating system |
| CN110069438A (en) * | 2018-01-22 | 2019-07-30 | 普天信息技术有限公司 | A kind of method of isomery device Memory communication |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060048162A1 (en) * | 2004-08-26 | 2006-03-02 | Bull Hn Information Systems Inc. | Method for implementing a multiprocessor message queue without use of mutex gate objects |
-
2019
- 2019-09-19 CN CN201910885530.1A patent/CN110737536B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1859325A (en) * | 2006-02-14 | 2006-11-08 | 华为技术有限公司 | News transfer method based on chained list process |
| CN104346229A (en) * | 2014-11-14 | 2015-02-11 | 国家电网公司 | Processing method for optimization of inter-process communication of embedded operating system |
| CN110069438A (en) * | 2018-01-22 | 2019-07-30 | 普天信息技术有限公司 | A kind of method of isomery device Memory communication |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110737536A (en) | 2020-01-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110737536B (en) | Message storage method and message reading method based on shared memory | |
| US8732360B2 (en) | System and method for accessing memory | |
| CN111427859B (en) | Message processing method and device, electronic equipment and storage medium | |
| CN105450618A (en) | Operation method and operation system of big data process through API (Application Programming Interface) server | |
| CN111708738B (en) | Method and system for realizing interaction of hadoop file system hdfs and object storage s3 data | |
| CN102437929A (en) | Method and device for de-queuing data in queue manager | |
| CN111475759A (en) | Message pushing platform, method, device, server and storage medium | |
| CN109564502B (en) | Processing method and device applied to access request in storage device | |
| CN111400212B (en) | Transmission method and device based on remote direct data access | |
| WO2020156797A1 (en) | Handling an input/output store instruction | |
| CN115374046B (en) | Multiprocessor data interaction method, device, equipment and storage medium | |
| US20200371827A1 (en) | Method, Apparatus, Device and Medium for Processing Data | |
| US11467836B2 (en) | Executing cross-core copy instructions in an accelerator to temporarily store an operand that cannot be accommodated by on-chip memory of a primary core into a secondary core | |
| CN117707991B (en) | Data reading and writing method, system, equipment and storage medium | |
| KR102454695B1 (en) | Caching device, cache, system, method and apparatus for processing data, and medium | |
| CN111984198A (en) | Message queue implementation method and device and electronic equipment | |
| CN112052104A (en) | Message queue management method based on multi-computer-room realization and electronic equipment | |
| CN110955461A (en) | Processing method, device and system of computing task, server and storage medium | |
| CN112486413B (en) | Data reading method, device, medium and computing equipment | |
| CN110968538A (en) | Data buffering method and device | |
| CN114063923A (en) | Data reading method and device, processor and electronic equipment | |
| CN102984599B (en) | Based on video acquisition transmitting device and the method for RapidIO protocol network | |
| JP2009199384A (en) | Data processing apparatus | |
| CN118069570B (en) | Doorbell type chip access system, device and method | |
| CN113094328B (en) | Multi-channel parallel computing system for real-time imaging of synthetic aperture radar |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |