CN104754036A - Message processing system and processing method based on kafka - Google Patents
Message processing system and processing method based on kafka Download PDFInfo
- Publication number
- CN104754036A CN104754036A CN201510099439.9A CN201510099439A CN104754036A CN 104754036 A CN104754036 A CN 104754036A CN 201510099439 A CN201510099439 A CN 201510099439A CN 104754036 A CN104754036 A CN 104754036A
- Authority
- CN
- China
- Prior art keywords
- message
- kafka
- module
- consumer
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
- H04L67/5682—Policies or rules for updating, deleting or replacing the stored data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/21—Monitoring or handling of messages
- H04L51/212—Monitoring or handling of messages using filtering or selective blocking
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a message processing system and processing method based on kafka and belongs to the technical field of message processing. The message processing system and processing method based on kafka are aimed to solve the problems of the prior art that repeated messages appear in an order system frequently, the processing for each message consumes much system resource, the system resource consumption would greatly increase the database visitor volume at the consumer end, the message processing ability is lowered, and the system stability and service effectiveness are lowered. The message processing system based on kafka comprises a message producer cluster composed of one or more message producers, a kafka cluster, and a consumer cluster composed of one or more consumers, the kafka cluster is composed of one or more servers, and the consumer cluster comprises a filter module, a local message queue caching module and a business processing module. The message processing method realized through the system is used for the system especially the order system which needs to process a large number of messages.
Description
Technical field
The present invention is specifically related to a kind of message handling system based on kafka and processing method, belongs to message processing technique field.
Background technology
Kafka is the distributed Message Queuing system (Message Queue) developed by Linkedin.
The main target of kafka exploitation is that structure one is used for processing massive logs, the data processing shelf of user behavior and website optimize statistics etc.Combining data mining, behavioural analysis, when the demands such as monitoring operation, needing can meet various real-time online with batch processed offline application scenario to the requirement of low delay with batch throughput performance.From demand fundamentally, high-throughput is first requirement, is secondly real-time and persistence.
The cluster of kafka is made up of multiple Broker server, the message of each type is defined as topic, the message of same topic inside is partitioned (partition) according to certain key and algorithm and is stored on different Broker, and message producer producer and consumer consumer can produce or consume topic on multiple Broker.
In large concurrent message treatment system, the general Producer-consumer problem pattern of kafka that uses carrys out asynchronous process request, reaches the object eliminating request peak, improves the stability of a system and throughput.Need the system processing a large amount of message as form ordering system for some, need to consume more system resource (such as repeatedly data base querying, access file system or network call etc.) owing to often duplicating message and processing every bar message, consumer's client database visit capacity will be caused to roll up, message handling ability declines, and the stability of a system and availability of service decline.
Summary of the invention
Therefore, the present invention is directed to form ordering system in prior art often to duplicate message and process every bar message and need to consume more system resource, consumer's client database visit capacity will be caused to roll up, message handling ability declines, the problem that the stability of a system and availability of service decline, a kind of message handling system based on kafka is provided, comprise the message producer cluster be made up of one or more message producer, kafka cluster, the consumer's cluster be made up of one or more consumer, kafka cluster is made up of one or more server, each server is according to certain theme storing message, each message producer submits message to one or more server, each consumer is to one or more server subscribe message, it is characterized in that, described consumer comprises filter module, local message queue cache module, Service Processing Module, described filter module is the container not allowing repeating data, for receiving the message that consumer subscribes to from server, described local message queue cache module is used for the message of buffer memory through filter modular filtration, described Service Processing Module is used for obtaining message from message queue cache module, complete the Business Processing to this message.
A kind of message treatment method based on kafka utilizing said system to realize, it is characterized in that, described method is: the message producer in message producer cluster submits message to kafka cluster, message is stored in corresponding server according to type of theme by kafka cluster, server sends this message to the consumer subscribing to this type of theme, the filter module of the consumer of receipt message receives this message and carries out filtration treatment to this message, this message is added to local message queue cache module when this message is not in filter module, this message is supplied to Service Processing Module by local message queue cache module, Service Processing Module completes the Business Processing to this message.
In described method, local message queue cache module arranges message queue maximum.Arranging of maximum needs according to the filter effect expected and the adjustment of Message Processing speed.When queue size major effect filter effect and internal storage data are lost (as restart, machine of delaying), information drop-out number.
After the Business Processing that in described method, Service Processing Module completes this message, prompting and filtering device module deletes this message.Can ensure that filter can not be excessive like this, and new message can be added in filter to.
Beneficial effect of the present invention is: adopt the message handling system based on kafka of the present invention and processing method, while possessing the ability of a large amount of message of high-throughput process of kafka, Service Processing Module does not directly cancel breath from Kafka, but breath is cancelled from local message queue cache module, message in local message queue cache module is through filter module filters, the message repeated is removed, a piece of news only processes once, greatly reduce the access times to consumer's client database, message handling ability rises, the stability of a system and availability of service improve.
Accompanying drawing explanation
Fig. 1 is the structural representation of the message handling system that the present invention is based on kafka.
Embodiment
Below in conjunction with accompanying drawing, the specific embodiment of the present invention is described:
As shown in Figure 1, a kind of message handling system based on kafka, comprise the message producer cluster be made up of 3 message producers, kafka cluster, the consumer's cluster be made up of 2 consumers, kafka cluster is made up of 3 servers, each server is according to certain theme storing message, theme is certain key and algorithm, each message producer submits message to one or more server, each consumer is to one or more server subscribe message, consumer comprises filter module, local message queue cache module, Service Processing Module, filter module is the container not allowing repeating data, for receiving the message that consumer subscribes to from server, local message queue cache module is used for the message of buffer memory through filter modular filtration, Service Processing Module is used for obtaining message from message queue cache module, complete the Business Processing to this message.
Utilize the message treatment method that the above-mentioned message handling system based on kafka realizes, be applied particularly in form ordering system, the message producer 2 i.e. user in message producer cluster submits message to kafka cluster, this message is user ID, user ID is stored in corresponding server 1 according to type of theme by kafka cluster, server 1 sends this message to the consumer 2 subscribing to this type of theme, the filter module receiving the consumer 2 of user ID receives user ID and carries out filtration treatment to user ID, filtered out when user ID is in filter module, this message is added to local message queue cache module when user ID is not in filter module, user ID is supplied to Service Processing Module by local message queue cache module, Service Processing Module completes the Business Processing to user ID.
As preferred embodiment, local message queue cache module arranges message queue maximum.Arranging of maximum needs according to the filter effect expected and the adjustment of Message Processing speed.When queue size major effect filter effect and internal storage data are lost (as restart, machine of delaying), information drop-out number.
As preferred embodiment, after Service Processing Module completes the Business Processing to user ID, prompting and filtering device module deletes user ID.Can ensure that filter can not be excessive like this, and new message can be added in filter to.
If adopt the message treatment method of prior art, a user may have multiple order, each order also may through repeatedly revising, consumer holds needs repeatedly to set up buffer memory for user ID, increase system burden, and use this method only need set up a buffer memory to user ID, can ensure Business Processing complete before user ID can not repeat.
Main Java language algorithm structure is as follows:
Initialization consumer
Function: initialization local message queue size, generates Kafka client, and creates thread receipt message.
Fill message queue
Function: add message to local message queue by filter, completes filtering function.
Obtain the mode of message
Service Processing Module does not directly cancel breath from Kafka, but cancel breath in local message queue, obtain and successfully from filter module, delete corresponding message simultaneously, can ensure that filter module can not be excessive like this, and new message can be added in filter module to.
The size of filter module and local message queue is arranged
Initial value and the maximum of filter capacity need not be set.Local message queue needs to arrange maximum, and arranging of maximum needs according to the filter effect expected and the adjustment of Message Processing speed.When queue size major effect filter effect and internal storage data are lost (as restart, machine of delaying), information drop-out number.
The submission pattern of at-most-once
Use the submission pattern of the at-most-once of Kafka client, need the value ensureing client configuration parameter auto.commit.enable to be true.
The realization of filter module and local message queue
Local message queue adopts supports that the BlockingQueue(of concurrent processing needs more than JDK1.5 to support).Filter module adopts the container not allowing to repeat, as Set.Because the situation of multithreading operation may be there is, therefore the ConcurrentSkipListSet(of support concurrent operations is adopted to need more than JDK1.5 to support).The message got from Kafka server is first stored in ConcurrentSkipListSet, relies on the Set container characteristic that permissible value repeats to complete filtering function.
The above is the preferred embodiment of the present invention; it should be pointed out that for those skilled in the art, under the prerequisite not departing from principle of the present invention; can also make some improvements and modifications, these improvements and modifications also should be considered as protection scope of the present invention.
Claims (4)
1. the message handling system based on kafka, comprise the message producer cluster be made up of one or more message producer, kafka cluster, the consumer's cluster be made up of one or more consumer, kafka cluster is made up of one or more server, each server is according to certain theme storing message, each message producer submits message to one or more server, each consumer is to one or more server subscribe message, it is characterized in that, described consumer comprises filter module, local message queue cache module, Service Processing Module, described filter module is the container not allowing repeating data, for receiving the message that consumer subscribes to from server, described local message queue cache module is used for the message of buffer memory through filter modular filtration, described Service Processing Module is used for obtaining message from message queue cache module, complete the Business Processing to this message.
2. the message treatment method based on kafka utilizing system described in claim 1 to realize, it is characterized in that, described method is: the message producer in message producer cluster submits message to kafka cluster, message is stored in corresponding server according to type of theme by kafka cluster, server sends this message to the consumer subscribing to this type of theme, the filter module of the consumer of receipt message receives this message and carries out filtration treatment to this message, this message is added to local message queue cache module when this message is not in filter module, this message is supplied to Service Processing Module by local message queue cache module, Service Processing Module completes the Business Processing to this message.
3., as claimed in claim 2 based on the message treatment method of kafka, it is characterized in that, in described method, local message queue cache module arranges message queue maximum.
4. as claimed in claim 2 based on the message treatment method of kafka, it is characterized in that, after the Business Processing that in described method, Service Processing Module completes this message, prompting and filtering device module deletes this message.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510099439.9A CN104754036A (en) | 2015-03-06 | 2015-03-06 | Message processing system and processing method based on kafka |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510099439.9A CN104754036A (en) | 2015-03-06 | 2015-03-06 | Message processing system and processing method based on kafka |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104754036A true CN104754036A (en) | 2015-07-01 |
Family
ID=53593124
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510099439.9A Pending CN104754036A (en) | 2015-03-06 | 2015-03-06 | Message processing system and processing method based on kafka |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104754036A (en) |
Cited By (56)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105141687A (en) * | 2015-08-19 | 2015-12-09 | 焦点科技股份有限公司 | Message producing method |
| CN105227662A (en) * | 2015-09-30 | 2016-01-06 | 努比亚技术有限公司 | Message treatment method, device and system |
| CN105630574A (en) * | 2015-12-21 | 2016-06-01 | 中国电子科技网络信息安全有限公司 | Container based message queue bus management system and method |
| CN106170013A (en) * | 2016-06-29 | 2016-11-30 | 上海浦东发展银行股份有限公司信用卡中心 | A kind of Kafka message uniqueness method based on Redis |
| CN106302385A (en) * | 2016-07-26 | 2017-01-04 | 努比亚技术有限公司 | A kind of message distribution device and method |
| CN106293968A (en) * | 2016-08-04 | 2017-01-04 | 华中科技大学 | A kind of intercommunication system based on Kafka message-oriented middleware and method |
| CN106375200A (en) * | 2016-11-08 | 2017-02-01 | 上海找钢网信息科技股份有限公司 | Message system and message processing method |
| CN106603689A (en) * | 2016-12-26 | 2017-04-26 | 努比亚技术有限公司 | Data processing method and device based on distributed message releasing and subscribing system |
| CN106657349A (en) * | 2016-12-29 | 2017-05-10 | 上海理想信息产业（集团）有限公司 | Message subscription processing device, system and method |
| CN106817295A (en) * | 2016-12-08 | 2017-06-09 | 努比亚技术有限公司 | A kind of message processing apparatus and method |
| WO2017114176A1 (en) * | 2015-12-30 | 2017-07-06 | 阿里巴巴集团控股有限公司 | Method and apparatus for coordinating consumption queue in distributed environment |
| CN106953901A (en) * | 2017-03-10 | 2017-07-14 | 重庆邮电大学 | A kind of trunked communication system and its method for improving message transmission performance |
| CN107329832A (en) * | 2017-06-30 | 2017-11-07 | 北京奇艺世纪科技有限公司 | A kind of data receiver method and device |
| CN107369012A (en) * | 2016-07-05 | 2017-11-21 | 康存乐付保数据科技（上海）有限公司 | A kind of method for processing payment information and system and the prepaid card processing method with performance bond insurance mechanism |
| CN107590072A (en) * | 2017-08-30 | 2018-01-16 | 深圳乐信软件技术有限公司 | A kind of application and development and the method and apparatus of test |
| CN107682169A (en) * | 2016-08-02 | 2018-02-09 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus using Kafka collection pocket transmission message |
| CN107911484A (en) * | 2017-12-13 | 2018-04-13 | 浪潮软件股份有限公司 | A kind of method and device of Message Processing |
| CN108009029A (en) * | 2017-11-30 | 2018-05-08 | 中电福富信息科技有限公司 | Method and system based on the data cached decoupling persistence of Ignite grids |
| CN108076103A (en) * | 2016-11-18 | 2018-05-25 | 北京优朋普乐科技有限公司 | A kind of message treatment method, system and message producer's cluster |
| CN108076111A (en) * | 2016-11-15 | 2018-05-25 | 亿阳安全技术有限公司 | A kind of system and method for distributing data in big data platform |
| CN108132845A (en) * | 2016-12-01 | 2018-06-08 | 阿里巴巴集团控股有限公司 | Message storage, delivering method and device and electronic equipment |
| CN108282514A (en) * | 2017-12-12 | 2018-07-13 | 北京奇艺世纪科技有限公司 | A kind of distributed service method for building up and device |
| CN108874562A (en) * | 2018-06-21 | 2018-11-23 | 北京顺丰同城科技有限公司 | Distributed high concurrent message queue supplying system |
| CN108920144A (en) * | 2017-04-11 | 2018-11-30 | 北京京东尚科信息技术有限公司 | A kind of message treatment method and device |
| CN108924041A (en) * | 2018-08-03 | 2018-11-30 | 北京捷易迅信息技术有限公司 | A kind of message efficient reliable diffusion method in instant messaging |
| CN108965355A (en) * | 2017-05-18 | 2018-12-07 | 北京京东尚科信息技术有限公司 | Method, apparatus and computer readable storage medium for data transmission |
| CN108984325A (en) * | 2018-07-20 | 2018-12-11 | 北京北信源信息安全技术有限公司 | Message queue consuming method and device |
| WO2018233592A1 (en) * | 2017-06-20 | 2018-12-27 | 中兴通讯股份有限公司 | Method for maintaining sequencing of messages, network node, and storage medium |
| CN109194736A (en) * | 2018-08-30 | 2019-01-11 | 百度在线网络技术（北京）有限公司 | Message De-weight method, device, electronic equipment, medium and unmanned vehicle |
| CN109451072A (en) * | 2018-12-29 | 2019-03-08 | 广东电网有限责任公司 | A kind of message caching system and method based on Kafka |
| CN109614250A (en) * | 2018-12-04 | 2019-04-12 | 贵州电网有限责任公司 | A kind of message system of the Kafka for electric acquisition system |
| CN109684099A (en) * | 2018-11-23 | 2019-04-26 | 新华三大数据技术有限公司 | Message treatment method and device |
| CN110019240A (en) * | 2017-12-29 | 2019-07-16 | 中国移动通信集团北京有限公司 | A kind of service data interaction method, apparatus and system |
| CN110138574A (en) * | 2019-03-26 | 2019-08-16 | 北京达佳互联信息技术有限公司 | Group managing means, device, electronic equipment and readable storage medium storing program for executing |
| CN110225133A (en) * | 2019-06-20 | 2019-09-10 | 恒生电子股份有限公司 | Message method, node, device, system and relevant device |
| CN110659124A (en) * | 2018-06-28 | 2020-01-07 | 北京京东尚科信息技术有限公司 | Message processing method and device |
| CN110704208A (en) * | 2019-09-19 | 2020-01-17 | 深圳证券交易所 | Message processing method for multiple systems, server and storage medium |
| CN110740145A (en) * | 2018-07-18 | 2020-01-31 | 北京京东尚科信息技术有限公司 | Message consumption method, device, storage medium and electronic equipment |
| CN111049899A (en) * | 2019-12-11 | 2020-04-21 | 贝壳技术有限公司 | kafka message storage system, method, apparatus, and computer-readable storage medium |
| CN111131512A (en) * | 2019-12-31 | 2020-05-08 | 中消云（北京）物联网科技研究院有限公司 | Equipment information processing method and device, storage medium and processor |
| CN111163172A (en) * | 2019-12-31 | 2020-05-15 | 北京奇艺世纪科技有限公司 | Message processing system, method, electronic device and storage medium |
| CN111179081A (en) * | 2019-12-24 | 2020-05-19 | 中国建设银行股份有限公司 | Message processing method and processing device |
| CN111179080A (en) * | 2019-12-23 | 2020-05-19 | 中国建设银行股份有限公司 | Order processing method and order processing device |
| CN111200637A (en) * | 2019-12-20 | 2020-05-26 | 新浪网技术(中国)有限公司 | Cache processing method and device |
| TWI697217B (en) * | 2019-01-30 | 2020-06-21 | 中華電信股份有限公司 | Container control system for repeatedly executing serverless programs and method thereof |
| CN111459689A (en) * | 2020-04-07 | 2020-07-28 | 中国建设银行股份有限公司 | Message processing system and method based on distributed queue |
| CN111555957A (en) * | 2020-03-26 | 2020-08-18 | 孩子王儿童用品股份有限公司 | Kafka-based synchronous message service system and implementation method |
| CN112330420A (en) * | 2020-11-27 | 2021-02-05 | 中国农业银行股份有限公司 | Order cancellation method and related device |
| CN112463411A (en) * | 2020-12-10 | 2021-03-09 | 重庆紫光华山智安科技有限公司 | Data processing method, device, server and storage medium |
| CN112667476A (en) * | 2020-12-30 | 2021-04-16 | 平安普惠企业管理有限公司 | Task-based message filtering method, device, equipment and storage medium |
| CN112751789A (en) * | 2021-01-05 | 2021-05-04 | 浪潮云信息技术股份公司 | Method and system for realizing asymmetric SDN controller cluster |
| TWI735519B (en) * | 2017-01-24 | 2021-08-11 | 香港商阿里巴巴集團服務有限公司 | Distributed environment coordinated consumption queue method and device |
| CN113542314A (en) * | 2020-04-13 | 2021-10-22 | 烽火通信科技股份有限公司 | Method and system for realizing subscription mode based on keyword filtering |
| CN115033401A (en) * | 2022-06-20 | 2022-09-09 | 北京中睿天下信息技术有限公司 | Message system based on isolation environment high-concurrency scene |
| CN115242787A (en) * | 2022-07-22 | 2022-10-25 | 恒生电子股份有限公司 | Message processing system and method |
| CN115914346A (en) * | 2021-08-09 | 2023-04-04 | 中移物联网有限公司 | Internet of things message processing method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102955717A (en) * | 2012-11-05 | 2013-03-06 | 北京奇虎科技有限公司 | Message management equipment and method in distributed message processing system |
| CN103049556A (en) * | 2012-12-28 | 2013-04-17 | 中国科学院深圳先进技术研究院 | Fast statistical query method for mass medical data |
| CN104092767A (en) * | 2014-07-21 | 2014-10-08 | 北京邮电大学 | Posting/subscribing system for adding message queue models and working method thereof |
| CN104092717A (en) * | 2013-12-13 | 2014-10-08 | 深圳市腾讯计算机系统有限公司 | Message processing method and system, message destination end device and distributed message system |
-
2015
- 2015-03-06 CN CN201510099439.9A patent/CN104754036A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102955717A (en) * | 2012-11-05 | 2013-03-06 | 北京奇虎科技有限公司 | Message management equipment and method in distributed message processing system |
| CN103049556A (en) * | 2012-12-28 | 2013-04-17 | 中国科学院深圳先进技术研究院 | Fast statistical query method for mass medical data |
| CN104092717A (en) * | 2013-12-13 | 2014-10-08 | 深圳市腾讯计算机系统有限公司 | Message processing method and system, message destination end device and distributed message system |
| CN104092767A (en) * | 2014-07-21 | 2014-10-08 | 北京邮电大学 | Posting/subscribing system for adding message queue models and working method thereof |
Cited By (87)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105141687B (en) * | 2015-08-19 | 2018-05-04 | 焦点科技股份有限公司 | A kind of method for producing message |
| CN105141687A (en) * | 2015-08-19 | 2015-12-09 | 焦点科技股份有限公司 | Message producing method |
| CN105227662A (en) * | 2015-09-30 | 2016-01-06 | 努比亚技术有限公司 | Message treatment method, device and system |
| CN105630574A (en) * | 2015-12-21 | 2016-06-01 | 中国电子科技网络信息安全有限公司 | Container based message queue bus management system and method |
| CN106933672B (en) * | 2015-12-30 | 2021-04-13 | 阿里巴巴集团控股有限公司 | Distributed environment coordinated consumption queue method and device |
| CN106933672A (en) * | 2015-12-30 | 2017-07-07 | 阿里巴巴集团控股有限公司 | A kind of distributed environment coordinates consumption queuing method and device |
| WO2017114176A1 (en) * | 2015-12-30 | 2017-07-06 | 阿里巴巴集团控股有限公司 | Method and apparatus for coordinating consumption queue in distributed environment |
| CN106170013A (en) * | 2016-06-29 | 2016-11-30 | 上海浦东发展银行股份有限公司信用卡中心 | A kind of Kafka message uniqueness method based on Redis |
| CN107369012A (en) * | 2016-07-05 | 2017-11-21 | 康存乐付保数据科技（上海）有限公司 | A kind of method for processing payment information and system and the prepaid card processing method with performance bond insurance mechanism |
| CN106302385A (en) * | 2016-07-26 | 2017-01-04 | 努比亚技术有限公司 | A kind of message distribution device and method |
| CN106302385B (en) * | 2016-07-26 | 2019-11-15 | 努比亚技术有限公司 | A kind of message distribution device and method |
| CN107682169B (en) * | 2016-08-02 | 2021-03-30 | 北京京东尚科信息技术有限公司 | Method and device for sending message by Kafka cluster |
| CN107682169A (en) * | 2016-08-02 | 2018-02-09 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus using Kafka collection pocket transmission message |
| CN106293968A (en) * | 2016-08-04 | 2017-01-04 | 华中科技大学 | A kind of intercommunication system based on Kafka message-oriented middleware and method |
| CN106293968B (en) * | 2016-08-04 | 2019-05-10 | 华中科技大学 | A kind of intercommunication system and method based on Kafka message-oriented middleware |
| CN106375200A (en) * | 2016-11-08 | 2017-02-01 | 上海找钢网信息科技股份有限公司 | Message system and message processing method |
| CN108076111B (en) * | 2016-11-15 | 2021-07-09 | 亿阳安全技术有限公司 | System and method for distributing data in big data platform |
| CN108076111A (en) * | 2016-11-15 | 2018-05-25 | 亿阳安全技术有限公司 | A kind of system and method for distributing data in big data platform |
| CN108076103A (en) * | 2016-11-18 | 2018-05-25 | 北京优朋普乐科技有限公司 | A kind of message treatment method, system and message producer's cluster |
| CN108132845A (en) * | 2016-12-01 | 2018-06-08 | 阿里巴巴集团控股有限公司 | Message storage, delivering method and device and electronic equipment |
| CN106817295A (en) * | 2016-12-08 | 2017-06-09 | 努比亚技术有限公司 | A kind of message processing apparatus and method |
| CN106603689A (en) * | 2016-12-26 | 2017-04-26 | 努比亚技术有限公司 | Data processing method and device based on distributed message releasing and subscribing system |
| CN106603689B (en) * | 2016-12-26 | 2021-08-03 | 努比亚技术有限公司 | Data processing method and device based on distributed publish-subscribe message system |
| CN106657349A (en) * | 2016-12-29 | 2017-05-10 | 上海理想信息产业（集团）有限公司 | Message subscription processing device, system and method |
| TWI735519B (en) * | 2017-01-24 | 2021-08-11 | 香港商阿里巴巴集團服務有限公司 | Distributed environment coordinated consumption queue method and device |
| CN106953901A (en) * | 2017-03-10 | 2017-07-14 | 重庆邮电大学 | A kind of trunked communication system and its method for improving message transmission performance |
| CN106953901B (en) * | 2017-03-10 | 2020-04-07 | 重庆邮电大学 | Cluster communication system and method for improving message transmission performance |
| CN108920144A (en) * | 2017-04-11 | 2018-11-30 | 北京京东尚科信息技术有限公司 | A kind of message treatment method and device |
| CN108965355B (en) * | 2017-05-18 | 2021-05-25 | 北京京东尚科信息技术有限公司 | Method, apparatus and computer readable storage medium for data transmission |
| CN108965355A (en) * | 2017-05-18 | 2018-12-07 | 北京京东尚科信息技术有限公司 | Method, apparatus and computer readable storage medium for data transmission |
| CN109104446A (en) * | 2017-06-20 | 2018-12-28 | 中兴通讯股份有限公司 | A kind of message order-preserving method, network node and storage medium |
| WO2018233592A1 (en) * | 2017-06-20 | 2018-12-27 | 中兴通讯股份有限公司 | Method for maintaining sequencing of messages, network node, and storage medium |
| CN109104446B (en) * | 2017-06-20 | 2022-04-15 | 中兴通讯股份有限公司 | Message order-preserving method, network node and storage medium |
| CN107329832A (en) * | 2017-06-30 | 2017-11-07 | 北京奇艺世纪科技有限公司 | A kind of data receiver method and device |
| CN107590072A (en) * | 2017-08-30 | 2018-01-16 | 深圳乐信软件技术有限公司 | A kind of application and development and the method and apparatus of test |
| CN107590072B (en) * | 2017-08-30 | 2020-08-18 | 深圳乐信软件技术有限公司 | Application development and test method and device |
| CN108009029B (en) * | 2017-11-30 | 2022-01-04 | 中电福富信息科技有限公司 | Method and system for decoupling and persisting cache data based on Ignite grid |
| CN108009029A (en) * | 2017-11-30 | 2018-05-08 | 中电福富信息科技有限公司 | Method and system based on the data cached decoupling persistence of Ignite grids |
| CN108282514B (en) * | 2017-12-12 | 2020-11-27 | 北京奇艺世纪科技有限公司 | Distributed service establishing method and device |
| CN108282514A (en) * | 2017-12-12 | 2018-07-13 | 北京奇艺世纪科技有限公司 | A kind of distributed service method for building up and device |
| CN107911484A (en) * | 2017-12-13 | 2018-04-13 | 浪潮软件股份有限公司 | A kind of method and device of Message Processing |
| CN110019240B (en) * | 2017-12-29 | 2021-11-09 | 中国移动通信集团北京有限公司 | Service data interaction method, device and system |
| CN110019240A (en) * | 2017-12-29 | 2019-07-16 | 中国移动通信集团北京有限公司 | A kind of service data interaction method, apparatus and system |
| CN108874562B (en) * | 2018-06-21 | 2022-08-02 | 北京顺丰同城科技有限公司 | Distributed high-concurrency message queue pushing system |
| CN108874562A (en) * | 2018-06-21 | 2018-11-23 | 北京顺丰同城科技有限公司 | Distributed high concurrent message queue supplying system |
| CN110659124A (en) * | 2018-06-28 | 2020-01-07 | 北京京东尚科信息技术有限公司 | Message processing method and device |
| CN110740145A (en) * | 2018-07-18 | 2020-01-31 | 北京京东尚科信息技术有限公司 | Message consumption method, device, storage medium and electronic equipment |
| CN110740145B (en) * | 2018-07-18 | 2023-08-08 | 北京京东尚科信息技术有限公司 | Message consumption method and device, storage medium and electronic equipment |
| CN108984325A (en) * | 2018-07-20 | 2018-12-11 | 北京北信源信息安全技术有限公司 | Message queue consuming method and device |
| CN108924041A (en) * | 2018-08-03 | 2018-11-30 | 北京捷易迅信息技术有限公司 | A kind of message efficient reliable diffusion method in instant messaging |
| CN109194736B (en) * | 2018-08-30 | 2021-04-27 | 百度在线网络技术（北京）有限公司 | Message duplicate removal method and device, electronic equipment, medium and unmanned vehicle |
| CN109194736A (en) * | 2018-08-30 | 2019-01-11 | 百度在线网络技术（北京）有限公司 | Message De-weight method, device, electronic equipment, medium and unmanned vehicle |
| US11050814B2 (en) | 2018-08-30 | 2021-06-29 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method, device and vehicle for message deduplication |
| CN109684099A (en) * | 2018-11-23 | 2019-04-26 | 新华三大数据技术有限公司 | Message treatment method and device |
| CN109684099B (en) * | 2018-11-23 | 2020-12-25 | 新华三大数据技术有限公司 | Message processing method and device |
| CN109614250A (en) * | 2018-12-04 | 2019-04-12 | 贵州电网有限责任公司 | A kind of message system of the Kafka for electric acquisition system |
| CN109451072A (en) * | 2018-12-29 | 2019-03-08 | 广东电网有限责任公司 | A kind of message caching system and method based on Kafka |
| TWI697217B (en) * | 2019-01-30 | 2020-06-21 | 中華電信股份有限公司 | Container control system for repeatedly executing serverless programs and method thereof |
| CN110138574A (en) * | 2019-03-26 | 2019-08-16 | 北京达佳互联信息技术有限公司 | Group managing means, device, electronic equipment and readable storage medium storing program for executing |
| CN110138574B (en) * | 2019-03-26 | 2021-10-26 | 北京达佳互联信息技术有限公司 | Group management method, device, electronic equipment and readable storage medium |
| CN110225133A (en) * | 2019-06-20 | 2019-09-10 | 恒生电子股份有限公司 | Message method, node, device, system and relevant device |
| CN110225133B (en) * | 2019-06-20 | 2022-01-21 | 恒生电子股份有限公司 | Message sending method, node, device, system and related equipment |
| CN110704208A (en) * | 2019-09-19 | 2020-01-17 | 深圳证券交易所 | Message processing method for multiple systems, server and storage medium |
| CN110704208B (en) * | 2019-09-19 | 2023-03-14 | 深圳证券交易所 | Message processing method for multiple systems, server and storage medium |
| CN111049899A (en) * | 2019-12-11 | 2020-04-21 | 贝壳技术有限公司 | kafka message storage system, method, apparatus, and computer-readable storage medium |
| CN111200637A (en) * | 2019-12-20 | 2020-05-26 | 新浪网技术(中国)有限公司 | Cache processing method and device |
| CN111200637B (en) * | 2019-12-20 | 2022-07-08 | 新浪网技术(中国)有限公司 | Cache processing method and device |
| CN111179080B (en) * | 2019-12-23 | 2023-10-27 | 中国建设银行股份有限公司 | Order processing method and order processing device |
| CN111179080A (en) * | 2019-12-23 | 2020-05-19 | 中国建设银行股份有限公司 | Order processing method and order processing device |
| CN111179081A (en) * | 2019-12-24 | 2020-05-19 | 中国建设银行股份有限公司 | Message processing method and processing device |
| CN111131512A (en) * | 2019-12-31 | 2020-05-08 | 中消云（北京）物联网科技研究院有限公司 | Equipment information processing method and device, storage medium and processor |
| CN111131512B (en) * | 2019-12-31 | 2023-02-10 | 中消云（北京）物联网科技研究院有限公司 | Equipment information processing method and device, storage medium and processor |
| CN111163172A (en) * | 2019-12-31 | 2020-05-15 | 北京奇艺世纪科技有限公司 | Message processing system, method, electronic device and storage medium |
| CN111555957A (en) * | 2020-03-26 | 2020-08-18 | 孩子王儿童用品股份有限公司 | Kafka-based synchronous message service system and implementation method |
| CN111555957B (en) * | 2020-03-26 | 2022-08-19 | 孩子王儿童用品股份有限公司 | Kafka-based synchronous message service system and implementation method |
| CN111459689A (en) * | 2020-04-07 | 2020-07-28 | 中国建设银行股份有限公司 | Message processing system and method based on distributed queue |
| CN113542314A (en) * | 2020-04-13 | 2021-10-22 | 烽火通信科技股份有限公司 | Method and system for realizing subscription mode based on keyword filtering |
| CN112330420A (en) * | 2020-11-27 | 2021-02-05 | 中国农业银行股份有限公司 | Order cancellation method and related device |
| CN112463411A (en) * | 2020-12-10 | 2021-03-09 | 重庆紫光华山智安科技有限公司 | Data processing method, device, server and storage medium |
| CN112667476B (en) * | 2020-12-30 | 2023-02-14 | 平安普惠企业管理有限公司 | Task-based message filtering method, device, equipment and storage medium |
| CN112667476A (en) * | 2020-12-30 | 2021-04-16 | 平安普惠企业管理有限公司 | Task-based message filtering method, device, equipment and storage medium |
| CN112751789A (en) * | 2021-01-05 | 2021-05-04 | 浪潮云信息技术股份公司 | Method and system for realizing asymmetric SDN controller cluster |
| CN115914346A (en) * | 2021-08-09 | 2023-04-04 | 中移物联网有限公司 | Internet of things message processing method and device, electronic equipment and storage medium |
| CN115033401A (en) * | 2022-06-20 | 2022-09-09 | 北京中睿天下信息技术有限公司 | Message system based on isolation environment high-concurrency scene |
| CN115033401B (en) * | 2022-06-20 | 2024-10-11 | 北京中睿天下信息技术有限公司 | Message system based on isolated environment high concurrency scene |
| CN115242787A (en) * | 2022-07-22 | 2022-10-25 | 恒生电子股份有限公司 | Message processing system and method |
| CN115242787B (en) * | 2022-07-22 | 2023-09-05 | 恒生电子股份有限公司 | Message processing system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104754036A (en) | Message processing system and processing method based on kafka | |
| US20180365254A1 (en) | Method and apparatus for processing information flow data | |
| CN109451072A (en) | A kind of message caching system and method based on Kafka | |
| CN107018042B (en) | Tracking method and tracking system for online service system | |
| CN106170016A (en) | A kind of method and system processing high concurrent data requests | |
| CN105224444B (en) | Log generation method and device | |
| CN108388479A (en) | Postpone information push method, device, computer equipment and storage medium | |
| US20100106915A1 (en) | Poll based cache event notifications in a distributed cache | |
| CN103516807A (en) | Cloud computing platform server load balancing system and method | |
| CN104731956A (en) | Method and system for synchronizing data and related database | |
| CN111651510A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN109450987B (en) | Number generation method, device and system and storage medium | |
| CN104994139B (en) | A kind of system and method to high concurrent network request quick response | |
| CN106789377A (en) | The service parameter update method of network element cluster | |
| CN105095299A (en) | Picture capturing method and system | |
| CN111209126A (en) | Data transmission method and device between microservices and electronic equipment | |
| CN103577251A (en) | Event based Internet computing processing system and method | |
| CN110413978B (en) | Data paging export method, device, computer equipment and storage medium | |
| CN111210340A (en) | Automatic task processing method and device, server and storage medium | |
| CN108076103A (en) | A kind of message treatment method, system and message producer's cluster | |
| CN105208004B (en) | A kind of data storage method based on OBD equipment | |
| CN104539583A (en) | Real-time database subscription system and method | |
| CN107018170A (en) | A kind of method of data synchronization and equipment | |
| CN113360577A (en) | MPP database data processing method, device, equipment and storage medium | |
| CN111352915B (en) | Machine learning system, machine learning parameter server and implementation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150701 |
|
| RJ01 | Rejection of invention patent application after publication |