CN110737536A - shared memory based message storage method and message reading method - Google Patents
shared memory based message storage method and message reading method Download PDFInfo
- Publication number
- CN110737536A CN110737536A CN201910885530.1A CN201910885530A CN110737536A CN 110737536 A CN110737536 A CN 110737536A CN 201910885530 A CN201910885530 A CN 201910885530A CN 110737536 A CN110737536 A CN 110737536A
- Authority
- CN
- China
- Prior art keywords
- message
- storage
- queue
- read
- storage sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 121
- 230000015654 memory Effects 0.000 title claims abstract description 48
- 238000003491 array Methods 0.000 claims description 2
- 230000003993 interaction Effects 0.000 abstract description 11
- 238000012423 maintenance Methods 0.000 abstract description 2
- 238000004891 communication Methods 0.000 abstract 1
- 238000005516 engineering process Methods 0.000 abstract 1
- 230000008569 process Effects 0.000 description 46
- 238000004364 calculation method Methods 0.000 description 18
- 238000010586 diagram Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000013467 fragmentation Methods 0.000 description 2
- 238000006062 fragmentation reaction Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011002 quantification Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
message storage methods and message reading methods based on shared memory, providing storage spaces, message storage methods and message reading methods, message storage methods, used to store the messages to be stored in the storage space, the storage space including the message control area and the message storage area, the message storage area for storing data, the message storage area including a plurality of storage sub-queues, the message control area for storing and recording the array of read-write information of the storage sub-queues, message reading methods, used to read the messages stored in the storage space according to the message storage methods, the message storage and reading implement the high-speed interaction of the messages by memory copy, solving the problems of the existing communication interaction technology that the message transfer efficiency is low, the high concurrency low-delay requirement can not be satisfied with the "real-time", the configuration is not flexible, the learning and maintenance costs are high.
Description
Technical Field
The invention relates to the field of telecommunication industry support, in particular to message storage methods and message reading methods based on shared memory.
Background
At present, the interaction mode among multiple processes in the charging system in the telecommunication industry is mainly to interact through files, databases and message middleware, wherein the file interaction mode is to write data into a file server through a front process and store the data in a physical disk, a rear process reads the files from the file server and obtains data from the physical disk, the interaction process relates to the opening and closing of files at a read-write end and finally operates the reading and writing of the physical disk, the database interaction mode is to write the data into the database server through a front process, a rear process reads the data from the database server, different processes perform data exchange through the same tables connected with database servers, the message middleware interaction mode performs data exchange through data in front and rear processes through kafka message queue of messages such as rabbitone and the like, and generates messages as messages among messages, and the messages are transmitted to a plurality of consumers through a message queue of messages, and the messages are transmitted to the message server through a plurality of data exchange queue of messages generated by MQ, and the messages are transmitted to the consumers.
However, in the file interaction process, a common file server is required, and the file server generally has a risk, because a file may be tampered, deleted, or divulged, and the like, under the condition of a large data volume, the file can be transmitted through the file, the scheme of the interaction mode is simple, time does not exceed time, and network bandwidth is not occupied, but if the data volume of a single file is large, a file writing process needs to wait for quantification to be written, and a file reading process needs to wait for a long time, so the mode is not suitable for real-time services.
Disclosure of Invention
In view of this, the present invention provides kinds of shared memory based message storage methods and message reading methods to solve the problems that the existing process interaction mode cannot meet the "real-time" requirement under the high concurrency and low latency requirements, configuration is not flexible enough, use is inconvenient, and learning and maintenance costs are high.
According to , the embodiment of the present invention provides kinds of storage spaces, including a message control area and a message storage area, where the message storage area is used to store data, the message storage area includes a plurality of storage sub-queues, and the message control area is used to store an array recording read-write information of the storage sub-queues.
With reference to the aspect, in an embodiment of the aspect, the method, the plurality of storage sub-queues are distributed in a two-dimensional array.
With reference to , in a second embodiment of , the messages are stored in each storage sub-queue according to a first-in-first-out principle.
According to a second aspect, an embodiment of the present invention provides message storage methods, which are used to store a message to be stored in a storage space, where the storage space includes the message control area and a message storage area, the message storage area is used to store data, the message storage area includes a plurality of storage sub-queues, the message control area is used to store an array for recording read-write information of the storage sub-queues, and the method includes obtaining a length of the message to be stored, obtaining, according to the array of the message control area, a start address of the storage sub-queue and a message storage location number of the storage sub-queue corresponding to the message to be stored, determining, according to the message storage location number and the length of the message to be stored, a start address offset of the storage sub-queue, calculating, according to the start address of the storage sub-queue and the start address offset of the storage sub-queue, and storing the message to be stored according to the storage address.
With reference to the second aspect, in an embodiment of the second aspect, the obtaining, according to the array of the message control area, the start address of the storage sub-queue corresponding to the message to be stored includes obtaining lengths of all storage sub-queues before the storage sub-queue of the message to be stored, and obtaining, according to the lengths, the start address of the storage sub-queue of the message to be stored.
With reference to the second aspect, in a second implementation manner of the second aspect, the obtaining a message storage location number of the storage sub-queue includes: acquiring the number of messages of the storage sub-queue of the messages to be stored; acquiring the capacity of the storage sub-queue; and obtaining the message storage position number of the storage sub-queue according to the number of the messages of the storage sub-queue of the messages to be stored and the capacity of the storage sub-queue.
With reference to the second aspect, in a third implementation manner of the second aspect, the calculating a starting address offset of the storage sub-queue according to the message storage location number includes: obtaining the length of a single message; and determining the initial address offset of the storage sub-queue according to the length of the single message numbered by the message storage position.
According to a third aspect, an embodiment of the present invention provides message reading methods, which are used for reading messages stored by the message storage method according to any implementation manner of the second aspect or the second aspect, and include acquiring a length of a message to be read, acquiring a start address of the storage sub-queue corresponding to the message to be read and a position number of the message to be read according to the array of the message control area, determining a start address offset of the storage sub-queue corresponding to the message to be read according to the position number of the message to be read and the length of the message to be read, determining a read address of the message according to the start address of the storage sub-queue corresponding to the message to be read and the start address offset of the storage sub-queue corresponding to the message to be read, and reading the message to be read according to the read address.
With reference to the third aspect, in an embodiment of the third aspect, in the , the obtaining, according to the array of the message control area, the start address of the storage sub-queue corresponding to the message to be read includes obtaining an actual concurrency degree of the message to be read, determining, according to the actual concurrency degree, lengths of all storage sub-queues before the storage sub-queue corresponding to the message to be read, and determining, according to the lengths of all storage sub-queues before the storage sub-queue corresponding to the message to be read, the start address of the storage sub-queue corresponding to the message to be read.
With reference to the third aspect, in a second implementation manner of the third aspect, the obtaining a location number of the message to be read includes: acquiring the number of messages to be read of the storage sub-queue corresponding to the messages to be read; acquiring the capacity of the storage sub-queue; and obtaining the reading position number in the storage sub-queue corresponding to the message to be read according to the number of the message to be read in the storage sub-queue corresponding to the message to be read and the capacity of the storage sub-queue.
With reference to the third aspect, in a third implementation manner of the third aspect, determining a start address offset of the storage sub-queue corresponding to the message to be read according to the position number of the message to be read and the length of the message to be read includes: obtaining the length of a single message; and determining the initial address offset of the storage sub-queue of the message to be read according to the storage position number of the message to be read in the storage sub-queue and the length of the single message.
According to a third aspect, an embodiment of the present invention provides computer devices, including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute a storage space in or of the embodiments , execute a message storage method in the second or of the embodiments, and execute a message reading method in the third or of the embodiments.
According to a fourth aspect, an embodiment of the present invention provides computer readable storage media having stored thereon computer instructions for causing a computer to execute the memory space in any implementation of the or aspect, execute the message storage method in any implementation of the second aspect or the second aspect, and execute the message reading method in any implementation of the third aspect or the third aspect.
The technical scheme of the invention has the following advantages:
1. the invention provides a shared memory-based message storage method and a message reading method, wherein a message to be stored is stored in a shared memory space, the shared memory space comprises a message control area and a message storage area, the message storage area stores messages according to message types and concurrency and is virtualized into a plurality of storage sub-queues, the message control area is used for storing and recording arrays of read-write information of the storage sub-queues, the read-write positions of the messages between different processes/threads are controlled by calculating message storage addresses and message reading addresses, a locking mechanism is not needed, high-speed transmission of message interaction is realized, the instantaneity under high-concurrency and low-delay requirements is met, the shared memory space is divided into a plurality of sub-message queues, different address requirements of message storage are met, the configuration is flexible, the interactive transmission of the messages between multiple processes and multiple threads can be realized, the efficient calculation of access addresses is realized, and the efficiency of quickly processing the messages of the same class is improved.
2. The invention provides message storage methods and message reading methods based on shared memory, which realize message storage based on two-dimensional storage sub-queues by calculating address offset.
3. The invention provides message storage methods and message reading methods based on shared memory, when reading messages based on the accumulation condition of message queues, if the number of reading processes/threads needs to be increased or decreased, the messages can be correctly read according to the rule of calculating and reading message addresses and the method of polling and reading a plurality of message storage sub-queues as long as the number of the reading processes/threads is within the maximum concurrency degree of message reading.
4. The invention provides message storage methods and message reading methods based on shared memory, which greatly expand the application range of a lock-free message queue based on the shared memory in practical application, read-write messages are copied by the memory, the message read-write does not involve reading of a plurality of memories, the read-write data position is calculated by internal routing, the read-write lock-free control is realized, a plurality of processes reading single message data is supported, and high-speed message storage and reading are realized.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a block diagram illustrating embodiments of a shared memory according to the present invention;
FIG. 2 is a diagram illustrating specific views of a storage sub-queue in a storage space according to an embodiment of the present invention;
FIG. 3 is specific diagrams illustrating the storage principle of the storage sub-queue according to the embodiment of the present invention;
FIG. 4 is a flow chart of specific examples of a message storage method in an embodiment of the invention;
fig. 5 is a flowchart of specific examples of S22 corresponding to the message storage method in the embodiment of the present invention;
fig. 6 is a flowchart of another specific examples of the message storage method corresponding to S22 in the embodiment of the present invention;
fig. 7 is a flowchart of specific examples of S13 corresponding to the message storage method in the embodiment of the present invention;
fig. 8 is a flowchart of specific examples of a message reading method in an embodiment of the present invention;
fig. 9 is a flowchart of specific examples of S22 corresponding to the message reading method in the embodiment of the present invention;
fig. 10 is a flowchart of another specific examples of the message reading method corresponding to S22 in the embodiment of the present invention;
fig. 11 is a flowchart of specific examples of S23 corresponding to the message reading method in the embodiment of the present invention;
fig. 12 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it is to be understood that the described embodiments are partial embodiments, but not all embodiments .
In addition, the technical features involved in the different embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
Example 1
In this embodiment, storage spaces are provided, first, a storage space is applied, the storage space is divided into a message control area and a message storage area, the message storage area is used for storing data and includes a plurality of storage sub-queues, the message control area is used for storing and recording an array of read-write information of the storage sub-queues, and the array includes storage sub-queue capacity, write process/thread write message sequence, read process/thread read message sequence, sub-message size, and start address of the storage sub-queues.
A plurality of storage sub-queues are stored in the storage space and stored according to a 'two-dimensional array' mode, each storage sub-Queue corresponds to control messages, the storage distribution of the storage sub-queues corresponds to a two-dimensional array , the th dimension corresponds to a message type, the second dimension corresponds to a maximum reading concurrency degree, the storage sub-queues are divided into M parts (classes) of messages, the M parts (classes) of messages are stored in a fragmentation mode, every parts (classes) of messages are stored according to a maximum concurrency thread T in a fragmentation mode, M.T storage sub-queues exist, corresponding to M.T sub-Queue information, the storage sub-queues are stored through a two-dimensional array cycle _ Queue [ M ] [ T ], each cycle _ Queue comprises the capacity Queue _ capacity of a sub-message Queue, a producer writes a message sequence Wr _ sequ _ seququnno, a consumer reads a message sequence Rd _ seunjun _ No, a sub-message Size Size, a Start address of a sub-message, a drt _ message, and other information sequence Wr _ sequ _ jun _ No, a consumer reads a message sequence, a message, a sub-message sequence of a sub-message, a storage sub-Queue is read message sequence of a read message, a read message sequence of a read message, a read message of.
The storage principle of the storage sub-Queue is shown in fig. 3, the storage sub-Queue is a circular storage area with first-in first-out, messages are sequentially transmitted and received, denier messages are processed by a read process/thread, and the messages are deleted from the Queue, assuming that there are producers (write messages to the storage sub-Queue) and consumers (read messages from the storage sub-Queue), the storage sub-Queue capacity is set, the capacity value is N (i.e., the storage sub-Queue can store N messages), the length of each message is Size, the producer writes the message sequence number Wr _ result _ No, the consumer reads the message sequence number Rd _ result _ No, the write message number Wr _ msg _ count, the read message number _ msg _ count, the Wr _ result _ count _ 1, the No _ result _ No, the consumer reads the message sequence number Wr _ result _ No, the write message number Wr _ result in the memory can be equal to the memory sub-Queue capacity.
The queue _ index value is cycled through the range of 0 to N-1, and the address where the message can be stored is calculated: offset + queue _ index Size;
the queue _ index obtained by the producer based on Wr _ sequnce _ No +1 represents that messages are stored in the position sequence number, and if the queue _ index is the same as the queue _ index obtained by Rd _ sequnce _ No, the message storage sub-queue is full.
The queue _ index obtained by the consumer based on Rd _ sequnce _ No +1 indicates that message position sequence numbers are read, and if the queue _ index is larger than the queue _ index obtained by Wr _ sequnce _ No, the message storage sub-queue is completely read.
The storage space provided by the embodiment, the message storage area stores data, the message control area stores an array for recording read-write information of the storage sub-queue, the message queue is realized based on the shared memory, and the method has the characteristics of fast message read-write, flexible distribution rule, and fast and simple use.
Example 2
This embodiment provides message storage methods, which can be used in a charging system in the telecommunications industry, and implement that a high-concurrency low-latency real-time charging requirement is required to be implemented, and implement that a message is stored in a storage space described in embodiment 1, where first, the message to be stored is stored in the storage space, then, by obtaining a length of the message to be stored, storing a start address of a storage sub-queue corresponding to the message and a message storage location number of the storage sub-queue, determining a start address offset of the storage sub-queue, and further determining a storage address of the message, and storing the message to be stored, as shown in fig. 4, the message storage method includes the following steps:
s21: and acquiring the length of the message to be stored.
S22: and acquiring the initial address of the storage sub-queue corresponding to the message to be stored and the message storage position number of the storage sub-queue according to the array of the message control area. The starting address is the memory position corresponding to the storage sub-queue for starting to store the message, the message storage is started from the corresponding memory position, and the message storage position number is the block number of the storage sub-queue and is used for the storage position of the corresponding message in the storage sub-queue.
S23: and determining the initial address offset of the storage sub-queue according to the message storage position number and the length of the message to be stored. And multiplying the message storage position number by the length of the message to be stored to obtain the initial address offset of the storage sub-queue.
S24: and calculating the storage address of the message according to the initial address of the storage sub-queue and the initial address offset of the storage sub-queue. And the sum of the starting address of the storage sub-queue and the starting address offset of the storage sub-queue is the storage address of the message.
S25: and storing the message to be stored according to the storage address.
According to fig. 2, assuming that the capacities of all the subqueues are N, the length Size of a single message, the Size of the storage space applied is mapped to Start from the applied storage Start address offSet, the maximum number of concurrent threads of each type (share) of messages is T, the type number of the currently stored message is X (i.e. the -th dimension value of the two-dimensional queue is X), the currently stored message needs to be written to the Y-th storage subqueue within the T concurrency (i.e. the second dimension value of the two-dimensional queue is Y), the Start address of the storage subqueue is stored in the Start Addr under the information of the cycle _ queue [ X-1] [ Y-1], and the stored calculation value is offSet + (X-1) × N +(Y-1) × Size, and the calculation of the Start address is completed when the message queue is initialized.
The starting address of the storage sub-queue is offSet + (X-1) T N Size + (Y-1) N Size, wherein the value range of X is 1-M, the value range of Y is 1-T, when messages are actually stored, X is transmitted through a writing process/thread, the value of X indicates that the process/thread needs to confirm which type of messages are written, the value of Y is obtained according to the 'Key' of the written messages, for example, the message Serial Number is Serial _ Number, then Serial _ Number is used for obtaining the remainder of T or the remainder of T according to the total message Number Wr _ g _ allount of some types of messages, the value is obtained by +1, or the value is obtained according to the polling in the range of 1-T, and the value of Start _ Addr of cycle _ ue [ X-1] [ Y-1] is taken as the starting address of the sub-queue needing to be written in the messages;
the message sequence number (written message number-1) written in the current queue is surplus according to the queue capacity, and the message storage position number queue _ index is obtained;
the offset of the starting address of the store sub-queue is: queue _ index _ Size;
determining the storage address of the message according to the initial address of the storage sub-queue and the initial address offset of the storage sub-queue: offSet + (X-1) T N Size + (Y-1) N Size + queue _ index Size, i.e., Start _ Addr + queue _ index Size for cycle _ queue [ X-1] [ Y-1 ].
In the message storage method provided in this embodiment, the storage of the storage sub-queue is implemented by calculating the address offset, and the multiple write processes may be messages of different types, or messages of the same type and different range segments. If the same message is written by a plurality of writing processes, the message source number and the message generation time can be increased, the message is routed based on the source number and corresponds to different message storage sub-queues, the lock-free writing of the message into the storage sub-queues is realized, and the message storage speed is improved.
Optionally, in embodiments of the present invention, as shown in fig. 5, S22 includes the following steps:
s221: and acquiring the lengths of all the storage sub queues before the storage sub queues of the messages to be stored, wherein the lengths are calculated according to the storage sub queues corresponding to the messages to be stored, the different types of messages stored in the storage sub queues corresponding to the messages to be stored and the write processes/thread numbers corresponding to the messages to be stored, namely (X-1) × T × N × Size + (Y-1) × N × Size.
S222: and obtaining the initial address of the storage sub-queue of the message to be stored according to the length. The sum of the start address of the applied storage space and the lengths of all storage sub-queues before the storage sub-queues for storing the messages is the start address of the storage sub-queues for storing the messages, namely, offSet + (X-1) T N Size + (Y-1) N Size.
The initial address of the storage sub-queue is obtained by calculating the lengths of all the storage sub-queues before the storage sub-queue where the message to be stored is obtained, without complicated routing calculation, so that the method is convenient to flexibly and universally use, and the high-efficiency calculation of the storage address is realized.
Optionally, in embodiments of the present invention, as shown in fig. 6, S22 further includes the following steps:
s223: and acquiring the message number of the storage sub-queue of the messages to be stored. The number of messages means that the message to be stored currently is the total number of messages.
S224: and acquiring the capacity of the storage sub-queue.
S225: and obtaining the message storage position number of the storage sub-queue according to the number of the messages of the storage sub-queue of the messages to be stored and the capacity of the storage sub-queue. The storage location number is obtained by taking the surplus of the capacity of the storage sub-queue according to the writing message sequence number (message number-1).
The message storage position number of the storage sub-queue is obtained by taking the surplus according to the capacity of the storage sub-queue according to the write message serial number of the storage sub-queue of the message to be stored, the calculation is simple, flexible and universal, and the high-efficiency calculation of the storage address is realized.
Optionally, in embodiments of the present invention, as shown in fig. 7, S23 includes the following steps:
s231: a single message length is obtained.
S232: and determining the initial address offset of the storage sub-queue according to the message storage position number and the single message length. The product of the message storage position number and the length of the single message is the offset of the starting address of the storage sub-queue.
The initial address offset of the storage sub-queue is determined according to the message storage position number of the storage sub-queue and the length of a single message, complex routing calculation is not involved, and the calculation speed of the storage address is improved.
Example 3
This embodiment provides message reading methods, configured to read a message stored by the method in embodiment 2, as shown in fig. 8, where the method includes:
and S31, acquiring the length of the message to be read, and acquiring the initial position of the storage sub-queue of the message to be read in the memory space in step according to the length of the message to be read.
S32: and acquiring the initial address of the storage sub-queue corresponding to the message to be read and the position number of the message to be read according to the array of the message control area. The starting address is a memory position corresponding to the storage sub-queue where the message is located, and is used for determining a storage address corresponding to the message to be read, and the position number of the message to be read is a position number in the storage sub-queue where the message to be read is located, and is used for reading the message from the storage address corresponding to the storage address of the message in the storage sub-queue.
S33: and determining the initial address offset of the storage sub-queue corresponding to the message to be read according to the position number of the message to be read and the length of the message to be read. And multiplying the position number of the message to be read by the length of the message to be read, namely the initial address offset of the storage sub-queue corresponding to the message to be read.
S34: and determining the reading address of the message according to the initial address of the storage sub-queue corresponding to the message to be read and the initial address offset of the storage sub-queue corresponding to the message to be read. And the sum of the initial address of the storage sub-queue corresponding to the message to be read and the initial address offset of the storage sub-queue corresponding to the message to be read is the reading address of the message.
S35: and reading the message to be read according to the reading address.
According to fig. 2, assuming that the capacity of each storage sub-queue is the same and is N, the length Size of a single message, the Size of the storage space applied is mapped to Start from the storage Start address offSet applied, the maximum number of concurrent threads of each message is T, the type value of the read message is X ( th dimension of the two-dimensional queue), at a certain type of message X, the message of the Y storage sub-queue (second dimension of the two-dimensional queue) is read, the Start _ Addr of the storage sub-queue, the calculated value of the cycle _ queue [ X-1] [ Y-1], Start _ Addr, i.e., offSet + (X-1) × T × Size + (Y-1) × N Size, the actual concurrency degree P (P ═ T), and the number Z (value between 1 and P) of the current read process/thread.
The read process/thread with Z number reads the value of the storage sub-queue, according to the obtained X value and Y value, the Start _ Addr value of the cycle _ queue [ X-1] [ Y-1] is the sub-queue Start address which needs to be read, the read process/thread with Z number reads the value of the storage sub-queue, and only when the read process/thread reads the value of the storage sub-queue, the read process/thread with Z number is the sub-queue Start address which needs to be read, the read process/thread with Y number reads the value of the storage sub-queue, and according to the obtained X value and Y value, the cycle _ queue [ X-1] [ Y-1] is the sub-queue Start address which needs to be read the message, the read process/thread is the storage sub-queue Start address which needs to be read the value of the storage sub-queue, the read process/thread address which needs to be read, the read the number of the storage sub-queue is the sub-queue Start address which needs to be read, the read the number of the storage sub-queue Start _ Addr is the storage sub-queue Start address which needs to be read, the storage sub-queue, the number of the storage sub-queue can be read, the read the message, the storage sub-queue Start _ Addr is the number, the number of the storage sub-queue can be read, the number of the storage sub-queue can be read the storage sub-queue Start _ Addr value of the storage sub-queue can be read, the storage sub-queue can be read, the information can be read, the information read is the information read, the information is the information read, the information read is the information read, the information read is the information, the information is the information, the information is the information, the information;
the message sequence number (read message number-1) read by the current queue is surplus according to the queue capacity, and the position number queue _ index of the message to be read is obtained;
the offset of the starting address of the storage sub-queue corresponding to the message to be read is: queue _ index _ Size;
determining the read address of the message according to the initial address of the storage sub-queue corresponding to the read message and the initial address offset: offSet + (X-1) T N Size + (Y-1) N Size + queue index Size.
According to the message reading method provided by the embodiment, the reading address corresponding to the message is determined, so that when a plurality of reading processes/threads read the message, the address of each reading process/thread for reading the message is not , and the message reading efficiency is improved.
Optionally, in embodiments of the present invention, as shown in fig. 9, S32 includes the following steps:
s321: and acquiring the actual concurrency of the message to be read, wherein the actual concurrency refers to the number of concurrent processes/threads for actually reading the message to be read.
S322: and determining the lengths of all the storage sub queues before the storage sub queue corresponding to the message to be read according to the actual concurrency, wherein the lengths are obtained by calculation according to the number of concurrent processes/threads of different types of messages stored in the storage sub queues corresponding to the message to be read and the message to be read.
S323: and determining the initial address of the storage sub-queue corresponding to the message to be read according to the lengths of all the storage sub-queues before the storage sub-queue corresponding to the message to be read.
Under the condition of message accumulation of the message storage sub-queues, if the number of reading processes/threads is increased or decreased, as long as the number of the reading processes/threads is within the maximum concurrency degree T, the messages can be correctly read according to the calculation rule of the message reading address and the method of polling the plurality of message storage sub-queues, the message reading can be sequential reading or parallel reading, the sequential reading refers to sub-queues, a plurality of reading programs sequentially process messages, namely, the number 1 process/thread finishes reading, the number 2 process/thread continues reading, the number 2 process/thread finishes reading, and the number 3 process/thread reads, the sequential reading is controlled by reading the position number of the message, and the parallel reading refers to the plurality of sub-queues of the concurrency degree T and reads the messages from different storage sub-queues.
The initial address of the message storage sub-queue corresponding to the message to be read is obtained by calculation according to the lengths of all the storage sub-queues before the storage sub-queue where the message to be read is obtained, complicated routing calculation is not carried out, flexibility and universality are facilitated, and efficient calculation of the read address is achieved.
Optionally, in embodiments of the present invention, as shown in fig. 10, S32 further includes the following steps:
s324: and acquiring the number of messages to be read of the storage sub-queue corresponding to the messages to be read, wherein the number of the messages to be read refers to the total number of the messages to be read.
S325: and acquiring the capacity of the storage sub-queue.
S326: and obtaining the reading position number in the storage sub-queue corresponding to the message to be read according to the number of the message to be read in the storage sub-queue corresponding to the message to be read and the capacity of the storage sub-queue. The reading position number is obtained by taking the surplus of the capacity of the storage sub-queue according to the reading message sequence number (the number of the messages to be read-1).
The read message position number of the storage sub-queue is obtained according to the read message sequence number of the storage sub-queue corresponding to the message to be read and the surplus of the capacity of the storage sub-queue, the calculation is simple, flexible and universal, and the efficient calculation of the message read address is realized.
Alternatively, in embodiments of the invention, as shown in fig. 11, S33 includes:
s331: a single message length is obtained.
S332: and determining the initial address offset of the storage sub-queue corresponding to the message to be read according to the storage position number in the storage sub-queue corresponding to the message to be read and the length of the single message. And the product of the storage position number in the storage sub-queue corresponding to the message to be read and the length of the single message is the initial address offset of the storage sub-queue corresponding to the message to be read.
And determining the initial address offset of the storage sub-queue corresponding to the message to be read according to the message storage position number of the storage sub-queue corresponding to the message to be read and the length of the single message, so that the complex routing calculation is not involved, and the calculation speed of the message reading address is improved.
Example 4
The embodiment of the present invention further provides computer devices, as shown in fig. 12, the computer devices may include a processor 41 and a memory 42, where the processor 41 and the memory 42 may be connected by a bus or by other means, and fig. 12 illustrates an example of a connection by a bus.
The Processor 41 may be a Central Processing Unit (CPU), the Processor 41 may also be other general purpose processors, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Array (FPGA) or other Programmable logic device, discrete or transistor logic devices, discrete hardware components, or a combination thereof.
The memory 42 is used as non-transitory computer readable storage media for storing non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions corresponding to the message storage method and the message reading method in the embodiment of the present invention, the processor 41 executes various functional applications and data processing of the processor by executing the non-transitory software programs and instructions stored in the memory 42, that is, implements the message storage method and the message reading method based on the shared memory in the above-mentioned method embodiment.
Memory 42 may include a program storage area that may store an operating system, at least applications needed for functionality, and a data storage area that may store data created by processor 41, etc. additionally, memory 42 may include high speed random access memory, and may also include non-transitory memory, such as at least disk storage devices, flash memory devices, or other non-transitory solid state storage devices, hi embodiments, memory 42 may optionally include memory located remotely from processor 41, which may be connected to processor 41 via a network, examples of which include, but are not limited to, the internet, an intranet, a local area network, a mobile communications network, and combinations thereof.
The one or more modules are stored in the memory 42 and, when executed by the processor 41, perform a shared memory based message storage method and a message reading method as in the embodiments of fig. 1-11.
The details of the computer device can be understood by referring to the corresponding descriptions and effects in the embodiments shown in fig. 1 to fig. 11, and are not described herein again.
Those skilled in the art will appreciate that all or part of the processes in the methods of the above embodiments may be implemented by hardware instructed by a computer program, where the program may be stored in computer readable storage medium, and when executed, the program may include the processes of the above embodiments of the methods, where the storage medium may be a magnetic Disk, an optical Disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid-State Drive (SSD), or the like, and the storage medium may further include a combination of the above memories.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (13)
- kinds of storage spaces are characterized by comprising a message control area and a message storage area, wherein the message storage area is used for storing data and comprises a plurality of storage sub-queues, and the message control area is used for storing and recording arrays of read-write information of the storage sub-queues.
- 2. The storage space of claim 1, wherein the plurality of storage sub-queues are distributed in a two-dimensional array manner.
- 3. The storage space of claim 1, wherein each storage sub-queue stores messages therein on a first-in-first-out basis.
- The message storage method of , which is used to store the message to be stored in the storage space, the storage space includes the message control area and the message storage area, the message storage area is used to store data, the message storage area includes a plurality of storage sub-queues, the message control area is used to store the array of the read-write information of the storage sub-queues, and the method is characterized in that the method includes:acquiring the length of a message to be stored;acquiring the initial address of the storage sub-queue corresponding to the message to be stored and the message storage position number of the storage sub-queue according to the array of the message control area;determining the initial address offset of the storage sub-queue according to the message storage position number and the length of the message to be stored;calculating the storage address of the message according to the initial address of the storage sub-queue and the initial address offset of the storage sub-queue;and storing the information to be stored according to the storage address.
- 5. The message storage method according to claim 4, wherein the obtaining, according to the array of the message control area, the start address of the storage sub-queue corresponding to the message to be stored includes:acquiring the lengths of all the storage sub queues before the storage sub queues of the messages to be stored;and obtaining the initial address of the storage sub-queue of the message to be stored according to the length.
- 6. The message storage method according to claim 4 or 5, wherein obtaining the message storage location number of the storage sub-queue comprises:acquiring the number of messages of the storage sub-queue of the messages to be stored;acquiring the capacity of the storage sub-queue;and obtaining the message storage position number of the storage sub-queue according to the number of the messages of the storage sub-queue of the messages to be stored and the capacity of the storage sub-queue.
- 7. The message storage method of any of claims 4-6 and , wherein the calculating a starting address offset of the storage sub-queue according to the message storage location number comprises:obtaining the length of a single message;and determining the initial address offset of the storage sub-queue according to the length of the single message numbered by the message storage position.
- Message reading method of for reading messages stored according to the message storage method of any of claims 4-7 and , comprising:acquiring the length of a message to be read;acquiring the initial address of the storage sub-queue corresponding to the message to be read and the position number of the message to be read according to the array of the message control area;determining the initial address offset of the storage sub-queue corresponding to the message to be read according to the position number of the message to be read and the length of the message to be read;determining a reading address of the message according to the initial address of the storage sub-queue corresponding to the message to be read and the initial address offset of the storage sub-queue corresponding to the message to be read;and reading the message to be read according to the reading address.
- 9. The method according to claim 8, wherein the obtaining, according to the array of the message control area, a start address of the storage sub-queue corresponding to the message to be read includes:acquiring the actual concurrency of the message to be read;determining the lengths of all the storage sub-queues before the storage sub-queue corresponding to the message to be read according to the actual concurrency;and determining the initial address of the storage sub-queue corresponding to the message to be read according to the lengths of all the storage sub-queues before the storage sub-queue corresponding to the message to be read.
- 10. The message reading method according to claim 8 or 9, wherein obtaining the position number of the message to be read comprises:acquiring the number of messages to be read of the storage sub-queue corresponding to the messages to be read;acquiring the capacity of the storage sub-queue;and obtaining the reading position number in the storage sub-queue corresponding to the message to be read according to the number of the message to be read in the storage sub-queue corresponding to the message to be read and the capacity of the storage sub-queue.
- 11. The message reading method according to any one of claims 8 to 10 and , wherein determining a starting address offset of the storage sub-queue corresponding to the message to be read according to the position number of the message to be read and the length of the message to be read comprises:obtaining the length of a single message;and determining the initial address offset of the storage sub-queue of the message to be read according to the storage position number of the message to be read in the storage sub-queue and the length of the single message.
- 12, a computer device, comprising:a memory and a processor communicatively coupled to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the shared memory based message storage method and the message reading method as recited in any of claims 1-11.
- computer-readable storage media, wherein the computer-readable storage media stores computer instructions for causing the computer to execute the shared memory based message storing method and the message reading method according to any of claims 1-11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910885530.1A CN110737536B (en) | 2019-09-19 | 2019-09-19 | Message storage method and message reading method based on shared memory |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910885530.1A CN110737536B (en) | 2019-09-19 | 2019-09-19 | Message storage method and message reading method based on shared memory |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110737536A true CN110737536A (en) | 2020-01-31 |
| CN110737536B CN110737536B (en) | 2024-03-12 |
Family
ID=69269345
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910885530.1A Active CN110737536B (en) | 2019-09-19 | 2019-09-19 | Message storage method and message reading method based on shared memory |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110737536B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111416858A (en) * | 2020-03-16 | 2020-07-14 | 广州市百果园信息技术有限公司 | Media resource processing platform, method, device and server |
| CN111416825A (en) * | 2020-03-24 | 2020-07-14 | 河南信大网御科技有限公司 | Inter-thread lock-free log management method and system, terminal and storage medium |
| CN111427859A (en) * | 2020-03-25 | 2020-07-17 | 京东数字科技控股有限公司 | Message processing method and device, electronic equipment and storage medium |
| CN111787107A (en) * | 2020-06-30 | 2020-10-16 | 北京字节跳动网络技术有限公司 | Message processing method and device, readable medium and electronic equipment |
| CN111984198A (en) * | 2020-08-25 | 2020-11-24 | 北京一起教育信息咨询有限责任公司 | Message queue implementation method and device and electronic equipment |
| CN113672400A (en) * | 2021-08-26 | 2021-11-19 | 深信服科技股份有限公司 | Data processing method, device and equipment and readable storage medium |
| CN113778948A (en) * | 2021-09-01 | 2021-12-10 | 中国人民银行清算总中心 | Message persistent storage method and device |
| WO2021249029A1 (en) * | 2020-06-09 | 2021-12-16 | 华为技术有限公司 | Apparatus and method for executing data processing, and computing device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060048162A1 (en) * | 2004-08-26 | 2006-03-02 | Bull Hn Information Systems Inc. | Method for implementing a multiprocessor message queue without use of mutex gate objects |
| CN1859325A (en) * | 2006-02-14 | 2006-11-08 | 华为技术有限公司 | News transfer method based on chained list process |
| CN104346229A (en) * | 2014-11-14 | 2015-02-11 | 国家电网公司 | Processing method for optimization of inter-process communication of embedded operating system |
| CN110069438A (en) * | 2018-01-22 | 2019-07-30 | 普天信息技术有限公司 | A kind of method of isomery device Memory communication |
-
2019
- 2019-09-19 CN CN201910885530.1A patent/CN110737536B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060048162A1 (en) * | 2004-08-26 | 2006-03-02 | Bull Hn Information Systems Inc. | Method for implementing a multiprocessor message queue without use of mutex gate objects |
| CN1859325A (en) * | 2006-02-14 | 2006-11-08 | 华为技术有限公司 | News transfer method based on chained list process |
| CN104346229A (en) * | 2014-11-14 | 2015-02-11 | 国家电网公司 | Processing method for optimization of inter-process communication of embedded operating system |
| CN110069438A (en) * | 2018-01-22 | 2019-07-30 | 普天信息技术有限公司 | A kind of method of isomery device Memory communication |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111416858A (en) * | 2020-03-16 | 2020-07-14 | 广州市百果园信息技术有限公司 | Media resource processing platform, method, device and server |
| CN111416858B (en) * | 2020-03-16 | 2023-04-07 | 广州市百果园信息技术有限公司 | Media resource processing platform, method, device and server |
| CN111416825A (en) * | 2020-03-24 | 2020-07-14 | 河南信大网御科技有限公司 | Inter-thread lock-free log management method and system, terminal and storage medium |
| CN111427859A (en) * | 2020-03-25 | 2020-07-17 | 京东数字科技控股有限公司 | Message processing method and device, electronic equipment and storage medium |
| CN111427859B (en) * | 2020-03-25 | 2024-04-05 | 京东科技控股股份有限公司 | Message processing method and device, electronic equipment and storage medium |
| WO2021249029A1 (en) * | 2020-06-09 | 2021-12-16 | 华为技术有限公司 | Apparatus and method for executing data processing, and computing device |
| CN111787107A (en) * | 2020-06-30 | 2020-10-16 | 北京字节跳动网络技术有限公司 | Message processing method and device, readable medium and electronic equipment |
| CN111787107B (en) * | 2020-06-30 | 2023-03-24 | 北京字节跳动网络技术有限公司 | Message processing method and device, readable medium and electronic equipment |
| CN111984198A (en) * | 2020-08-25 | 2020-11-24 | 北京一起教育信息咨询有限责任公司 | Message queue implementation method and device and electronic equipment |
| CN111984198B (en) * | 2020-08-25 | 2024-02-27 | 北京一起教育信息咨询有限责任公司 | Message queue implementation method and device and electronic equipment |
| CN113672400A (en) * | 2021-08-26 | 2021-11-19 | 深信服科技股份有限公司 | Data processing method, device and equipment and readable storage medium |
| CN113778948A (en) * | 2021-09-01 | 2021-12-10 | 中国人民银行清算总中心 | Message persistent storage method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110737536B (en) | 2024-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110737536A (en) | shared memory based message storage method and message reading method | |
| US11010681B2 (en) | Distributed computing system, and data transmission method and apparatus in distributed computing system | |
| US8930593B2 (en) | Method for setting parameters and determining latency in a chained device system | |
| CN102437929B (en) | Method and device for de-queuing data in queue manager | |
| US10140020B2 (en) | Data queue having an unlimited depth | |
| CN111708738B (en) | Method and system for realizing interaction of hadoop file system hdfs and object storage s3 data | |
| CN102567258B (en) | Multi-dimensional DMA (direct memory access) transmitting device and method | |
| CN107783727B (en) | Access method, device and system of memory device | |
| CN109564502B (en) | Processing method and device applied to access request in storage device | |
| CN108897630B (en) | OpenCL-based global memory caching method, system and device | |
| CN116521096B (en) | Memory access circuit, memory access method, integrated circuit, and electronic device | |
| CN109298888B (en) | Queue data access method and device | |
| CN111400212A (en) | Transmission method and device based on remote direct data access | |
| US8630979B2 (en) | Non-blocking input output based storage | |
| CN117707991B (en) | Data reading and writing method, system, equipment and storage medium | |
| CN111984198B (en) | Message queue implementation method and device and electronic equipment | |
| CN111797497B (en) | Communication method and system for electromagnetic transient parallel simulation | |
| CN110968538B (en) | Data buffering method and device | |
| CN118018481A (en) | Data transmission method, device and system | |
| AU2010201275A1 (en) | A high performance and low latency messaging system | |
| JP2009199384A (en) | Data processing apparatus | |
| CN113094328B (en) | Multi-channel parallel computing system for real-time imaging of synthetic aperture radar | |
| CN117196929B (en) | Software and hardware interaction system based on fixed-length data packet | |
| KR101485246B1 (en) | A design of extended ring buffer | |
| CN117081961A (en) | Method, system, device and server for monitoring quantity of accumulated messages |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |