CN110019240A

CN110019240A - A kind of service data interaction method, apparatus and system

Info

Publication number: CN110019240A
Application number: CN201711479774.7A
Authority: CN
Inventors: 谷鑫; 谭朝丽; 程娟; 张鹏; 侯兆森; 李云; 陈大庆
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Group Beijing Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Group Beijing Co Ltd
Priority date: 2017-12-29
Filing date: 2017-12-29
Publication date: 2019-07-16
Anticipated expiration: 2037-12-29
Also published as: CN110019240B

Abstract

A kind of service data interaction method, apparatus and system, it within the system include Kafka cluster, the first operation system and the second operation system, first operation system obtains the key-value pair shared between the first operation system and the second operation system, zoning schemes according to key-value pair and Kafka cluster determine target partition, and business datum is distributed to target partition.Kafka cluster stores business datum to target partition, and receive the key-value pair that the second operation system is sent, zoning schemes according to key-value pair and Kafka cluster determine target partition, and business datum is obtained in target partition, and business datum is sent to the second operation system.The business datum of second operation system reception Kafka collection pocket transmission.The complexity and expense of operation system interactive process can be reduced through the invention.

Description

A kind of service data interaction method, apparatus and system

Technical field

The present invention relates to field of communication technology more particularly to a kind of service data interaction method, apparatus and system.

Background technique

A kind of message-oriented middleware of the Kafka as open source, extensive utilization are used for the friendship of business datum in a distributed system Mutually.

Service data interaction system based on Kafka mainly includes the message producer (Producer), server (Broker) the Kafka cluster formed and message consumer (Consumer).The message producer gives out information to Kafka cluster, And the message for being each published to Kafka cluster has a classification, this classification can be described as theme (topic).Each topic Comprising one or more subregions (partition), each partition is made of a series of orderly, immutable message. Kafka cluster manages message with topic, and the message that the message producer issues continuously is appended in partition, Each message in partition is assigned a continuous serial number, this serial number can be described as offset (offset), Offset can be used for one message of partition unique identification.Kafka cluster transmits the message to message consumer, and message disappears Expense person handles message.

Service data interaction system based on Kafka generallys use the interaction that publish/subscribe mode carries out business datum.? Under publish/subscribe mode, need to establish multiple publishers and subscriber, wherein publisher and subscriber include message production Person and message consumer have the ability that message is issued and obtained from Kafka cluster to Kafka cluster.Publisher to some Topic gives out information, and subscriber then subscribes to Kafka cluster and obtains the message under some topic.Wherein, topic can be managed Solution is the message flow distinguished according to different classifications for being the maintenance of Kafka cluster.Each publisher needs to safeguard that message is failed to be sold at auction Know the message traffic identifier Hash table with external message stream mapping relations, each subscriber is also required to maintenance message traffic identifier and publication The publisher of mapping relations identifies mapping relations between Hash table and publisher's mark and subscriber's theme between person's mark Publisher's theme Hash table, and publisher, publisher's mark and publisher's theme have one-to-one relationship and globally unique.Hair When cloth person gives out information or Subscriber subscriptions obtain message, by the full-text search of exterior arrangement file, mapping relations Hash table Multiple required treatment processes such as initialization, the full library searching matching of message identifier, the content splicing of message body, realize ordering for message It reads and issues.

However, being carried out in the transmission process of large scale scale heterogeneous class data using aforesaid way, publisher and subscriber need The mapping relations Hash table content complexity of maintenance is higher, and reads exterior arrangement file, mapping table initialization, complete The process of library searching mapping relations Hash table, needs complicated calculation processes, and system consumption is larger.

Summary of the invention

The embodiment of the present invention provides a kind of service data interaction method, apparatus and system, is interacted with reducing operation system The complexity and expense of journey.

In a first aspect, the embodiment of the present invention provides a kind of service data interaction system, the system include Kafka cluster and At least one operation system includes the first operation system for realizing Kafka producer's function at least one described operation system With the second operation system for realizing Kafka consumer's function, first operation system and second operation system are all had To Kafka cluster with Kafka form of message issuing service data and from Kafka cluster with Kafka form of message reception business number According to ability, in which:

First operation system, for obtain between first operation system and second operation system share Key-value pair, the key-value pair is for identifying the business number interacted between first operation system and second operation system According to first operation system determines target partition according to the zoning schemes of the key-value pair and the Kafka cluster, and will The business datum is distributed to the target partition.

The Kafka cluster, for receiving the business datum of first operation system publication, and by the business datum It stores to the target partition, and receives the key-value pair that second operation system is sent, according to the key-value pair and described The zoning schemes of Kafka cluster determine the target partition, and the business datum is obtained in the target partition, will be described Business datum is sent to second operation system.

Second operation system, for obtaining the key-value pair, and to key-value pair described in the Kafka collection pocket transmission, The business datum interacted between first operation system and second operation system is subscribed to, and receives the Kafka cluster The business datum of transmission.

Optionally, first operation system obtains first operation system and second business in the following way The key-value pair shared between system:

The identity information of first operation system is encrypted, the first identity mark of first operation system is generated Know；The identity information of second operation system is obtained, and the identity information of second operation system is encrypted, is generated Second identity；First identity and second identity are encrypted, message queue mark is generated Code, using the message queue identification code as the key assignments shared between first operation system and second operation system It is right.

Second operation system obtains the key-value pair in the following way:

The identity information of second operation system is encrypted, the second identity is generated；Obtain first industry The identity information of business system, and the identity information of first operation system is encrypted, generate the first identity；Foundation Second identity and first identity are encrypted, and message queue identification code are generated, by the message team Column identification code is as the key-value pair shared between first operation system and second operation system.

Optionally, the key-value pair is the identification code that can be resolved to integer.First operation system and described Kafka cluster determines the target point according to the zoning schemes of the key-value pair and the Kafka cluster in the following way Area:

It parses the key-value pair, and integer that the key-value pair obtains will be parsed and the Kafka cluster is the business The number of partitions for including in the theme that data divide carries out remainder operation；Subregion serial number is consistent with the numerical value that remainder operates Subregion, as the target partition.

Optionally, first operation system is difference for including in different operation system or same operation system Operation system.The Kafka cluster is also used to: the business datum issued for different operation system is same operation system In include different subservice systems publication business datum, divide different themes.

Optionally, the quantity of second operation system be at least one, and at least one described second operation system with First operation system shares identical key-value pair.

Optionally, first operation system is also used to: by the attribute information of business datum, with the key-value pair and described Business datum is encapsulated as Kafka message body, and is distributed to the target partition, and the attribute information of the business datum includes business At least one of in the transmission mode of data, the storage format of business datum and traffic data type.The Kafka cluster is also For parsing Kafka message body, and store the attribute information of the business datum.

Further, first operation system is also used to encrypt the business datum, and by encrypted industry Data publication be engaged in the target partition.Second operation system is also used to receive the business datum of encryption, and to the industry Business data are decrypted.

Specifically, the Kafka cluster in the following way stores the business datum to the target partition:

The target partition is divided as unit of being sliced, the business datum is stored to the target partition In slice, offset of the business datum in the slice is generated, and identify the business datum using the offset Storage location in the target partition.

The Kafka cluster obtains the business datum in the target partition in the following way:

The Kafka cluster is using the offset for identifying the business datum, and determination is deposited in the target partition The fragment of the business datum is stored up, and obtains the business datum in the fragment.

Further, the Kafka cluster is also used to: if storing the capacity of the fragment of the business datum beyond setting The storage time of threshold value or the business datum then deletes the business datum beyond setting duration.

Second aspect, the embodiment of the present invention provide a kind of service data interaction method, and this method can be applied to realize Kafka In first operation system of producer's function, which comprises

The business datum interacted between the first operation system and the second operation system is obtained, and obtains first business system The key-value pair shared between system and the second operation system, the key-value pair is for identifying first operation system and described second The business datum of interaction between operation system；

Zoning schemes according to the key-value pair and the Kafka cluster determine target partition, and by the business number According to being distributed to the target partition.

Specifically, described obtain the key-value pair shared between first operation system and the second operation system, comprising:

The identity information of first operation system is encrypted, the first identity mark of first operation system is generated Know；

The identity information of second operation system is obtained, and the identity information of second operation system is added It is close, generate the second identity；

First identity and second identity are encrypted, message queue identification code is generated, it will The message queue identification code is as the key-value pair shared between first operation system and second operation system.

Further, the key-value pair is the identification code that can be resolved to integer；

The zoning schemes according to the key-value pair and the Kafka cluster determine the target partition, comprising:

It parses the key-value pair, and integer that the key-value pair obtains will be parsed and the Kafka cluster is the business The number of partitions for including in the theme that data divide carries out remainder operation；

The consistent subregion of numerical value that subregion serial number and remainder are operated, as the target partition.

Specifically, the business datum is distributed to the target partition, comprising:

By the attribute information of business datum, it is encapsulated as Kafka message body with the key-value pair and the business datum, concurrently For cloth to the target partition, the attribute information of the business datum includes the storage of the transmission mode of business datum, business datum At least one of in format and traffic data type.

The third aspect, the embodiment of the present invention provide a kind of service data interaction method, and this method can be applied to Kafka collection Group, which comprises

The business datum of the first operation system publication is received, and the business datum is stored to target partition, the mesh Marking subregion is that first operation system is determined according to the zoning schemes of key-value pair and the Kafka cluster, the key assignments To for identifying the business datum interacted between first operation system and second operation system；

It receives the key-value pair of the second operation system transmission and subscribes to request, the subscription request is for subscribing to described first The business datum interacted between operation system and second operation system；

Zoning schemes according to the key-value pair and the Kafka cluster determine the target partition, and in the mesh It marks and obtains the business datum in subregion, the business datum is sent to second operation system.

Specifically, the key-value pair is the identification code that can be resolved to integer；

Zoning schemes according to the key-value pair and the Kafka cluster determine the target partition, comprising:

Further, the business datum is stored to before target partition, the method also includes:

The business datum issued for different operation system is the different subservice systems for including in same operation system The business datum of system publication, divides different themes.

Specifically, storing the business datum to target partition, comprising:

By the attribute information of business datum, store with the business datum to target partition；

The attribute information of the business datum includes the transmission mode of business datum, the storage format of business datum and industry At least one of in data type of being engaged in.

Specifically, storing the business datum to target partition, comprising:

The target partition is divided as unit of being sliced, the business datum is stored to the target partition In slice, offset of the business datum in the slice is generated, and identify the business datum using the offset Storage location in the target partition；

The business datum is obtained in the target partition, comprising:

Fourth aspect, the embodiment of the present invention provide a kind of service data interaction method, and this method can be applied to realize Kafka Second operation system of consumer's function, comprising:

The key-value pair shared between the first operation system and second operation system is obtained, the key-value pair is for identifying The business datum interacted between first operation system and second operation system；

To key-value pair described in Kafka collection pocket transmission and request is subscribed to, and receives the business number of the Kafka collection pocket transmission According to；

The subscription request is for subscribing to the business interacted between first operation system and second operation system Data；

The business datum of the Kafka collection pocket transmission is obtained in target partition by the Kafka cluster, the target point Zoning schemes of the area by the Kafka cluster according to the key-value pair and the Kafka cluster determine.

Specifically, obtaining the key-value pair shared between the first operation system and second operation system, comprising:

The identity information of second operation system is encrypted, the second identity is generated；

The identity information of first operation system is obtained, and the identity information of first operation system is added It is close, generate the first identity；

It is encrypted according to second identity and first identity, generates message queue identification code, Using the message queue identification code as the key-value pair shared between first operation system and second operation system.

5th aspect, the embodiment of the present invention provide a kind of service data interaction device, which can be applied to realize Kafka In first operation system of producer's function, comprising:

Acquiring unit, for obtaining the business datum interacted between the first operation system and the second operation system, Yi Ji Between one operation system and the second operation system share key-value pair, the key-value pair for identify first operation system with The business datum of interaction between second operation system；

The zoning schemes of processing unit, key-value pair and Kafka cluster for obtaining according to the acquiring unit determine Target partition；

Release unit, the business datum for obtaining the acquiring unit are distributed to described in the processing unit determination Target partition.

Wherein, the key-value pair is the identification code that can be resolved to integer；

The processing unit is in the following way according to described in the determination of the zoning schemes of the key-value pair and Kafka cluster Target partition:

Optionally, the acquiring unit is obtained in the following way between first operation system and the second operation system Shared key-value pair:

Optionally, the business datum is distributed to the target partition in the following way by the release unit:

6th aspect, the embodiment of the present invention provide a kind of service data interaction device, which can be applied to Kafka collection Group, comprising:

Receiving unit for receiving the business datum of the first operation system publication, and receives what the second operation system was sent Key-value pair and subscription request；

Storage unit, for storing the received business datum of the receiving unit to target partition, the target partition It is determined for first operation system according to the zoning schemes of key-value pair and the Kafka cluster, the key-value pair is used for Identify the business datum interacted between first operation system and second operation system；

Processing unit, the subregion machine of key-value pair and the Kafka cluster for being received according to the receiving unit It makes and determines target partition, and obtain the business datum in the target partition；

Transmission unit, the business datum for determining the processing unit are sent to second operation system.

Wherein, the key-value pair is the identification code that can be resolved to integer.The processing unit in the following way according to The target partition is determined according to the zoning schemes of the key-value pair and the Kafka cluster:

Optionally, the processing unit is also used to: storing the business datum to target partition in the storage unit Before, the business datum issued for different operation system is the different subservice systems hair for including in same operation system The business datum of cloth divides different themes.

Further, the storage unit in the following way stores the business datum to target partition:

By the attribute information of business datum, store with the business datum to target partition；The attribute of the business datum Information includes at least one in the transmission mode of business datum, the storage format of business datum and traffic data type.

The processing unit obtains the business datum in the target partition in the following way: using described in mark The offset of business datum determines the fragment for storing the business datum in the target partition, and in the fragment It is interior to obtain the business datum.

7th aspect, the embodiment of the present invention provide a kind of service data interaction device, which can be applied to realize Kafka Second operation system of consumer's function, comprising:

Processing unit, for obtaining the key-value pair shared between the first operation system and the second operation system, the key assignments To for identifying the business datum interacted between first operation system and second operation system；

Transmission unit, for key-value pair described in Kafka collection pocket transmission and subscription request, the subscription request to be for subscribing to The business datum interacted between first operation system and second operation system；

Receiving unit, for receiving the business datum of Kafka collection pocket transmission, the business datum of the Kafka collection pocket transmission Obtained in target partition by the Kafka cluster, the target partition by the Kafka cluster according to the key-value pair and The zoning schemes of the Kafka cluster determine.

Optionally, the processing unit obtains share between the first operation system and the second operation system in the following way Key-value pair:

Service data interaction method, apparatus and system provided in an embodiment of the present invention, pass through the operation system of interaction data Between share key-value pair, and determine target partition using the zoning schemes of key-value pair and Kafka cluster, by business datum issue with And it is stored in determining target partition, Maintenance free mapping relations Hash table, and initialized, entirely without executing mapping table The process of library searching mapping relations Hash table, therefore can reduce the complexity of operation system interactive process to a certain extent and open Pin.

Detailed description of the invention

Fig. 1 is the product operation system architecture diagram of application of the embodiment of the present invention；

Fig. 2 is the Kafka message system schematic diagram of application of the embodiment of the present invention；

Fig. 3 is that service data interaction system provided in an embodiment of the present invention constitutes schematic diagram；

Fig. 4 is schematic diagram of the issuing service data provided in an embodiment of the present invention to target partition；

Fig. 5 is the schematic diagram that Partition provided in an embodiment of the present invention stores message；

Fig. 6 is the process schematic that Producer provided in an embodiment of the present invention sends message；

Fig. 7 is that the product operation system message provided in an embodiment of the present invention using Kafka pushes networking schematic diagram；

Fig. 8 is the schematic diagram that business datum provided in an embodiment of the present invention stores the slice to the target partition；

Fig. 9 is segment file designation schematic diagram provided in an embodiment of the present invention；

Figure 10 is the corresponding relation schematic diagram of " .index " file provided in an embodiment of the present invention and " .log " file；

Figure 11 is FileMessageSet class storage file schematic diagram provided in an embodiment of the present invention；

Figure 12 is that a kind of service data interaction method applied to the first operation system provided in an embodiment of the present invention is implemented Flow chart；

Figure 13 is the flow diagram that the first operation system provided in an embodiment of the present invention obtains key-value pair；

Figure 14 is the flow diagram that the first operation system provided in an embodiment of the present invention determines target partition；

Figure 15 is a kind of service data interaction method implementing procedure applied to Kafka cluster provided in an embodiment of the present invention Figure；

Figure 16 is that a kind of service data interaction method applied to the second operation system provided in an embodiment of the present invention is implemented Flow chart；

Figure 17 is a kind of service data interaction apparatus structure applied to the first operation system provided in an embodiment of the present invention Schematic diagram；

Figure 18 is that a kind of service data interaction apparatus structure applied to Kafka cluster provided in an embodiment of the present invention is illustrated Figure；

Figure 19 is a kind of service data interaction apparatus structure applied to the second operation system provided in an embodiment of the present invention Schematic diagram；

Figure 20 is a kind of service data interaction device structure schematic diagram provided in an embodiment of the present invention.

Specific embodiment

Below in conjunction with Figure of description, preferred embodiment of the present invention will be described, it should be understood that described herein Preferred embodiment only for the purpose of illustrating and explaining the present invention and is not intended to limit the present invention, and in the absence of conflict, this hair The feature in embodiment and embodiment in bright can be combined with each other.

The embodiment of the present invention is mainly used in and carries out the scene that business datum interacts between multiple operation systems, at this The broadcast of business datum can be carried out under application scenarios between operation system, it is also possible to carry out the unicast of business datum, such as can answer Include in product operation system shown in FIG. 1, in the product operation system product operation platform and with the product operation Each business platform that platform is cooperated.To realize that the ability of affiliate in industrial chain introduces the carrying with business model, Product operation platform needs to provide for each business platform comprising the management of product, set meal management, fee administration, is divided into management, charging A variety of way to manages and means including management, and ensure that its optimal product can be placed in product operation and put down by each business platform Promotion are carried out on platform to sell.Wherein, between product operation platform and each business platform transmission services data data format and biography Defeated mode is also different.For example, product operation platform and the online electronic invoice of middle shifting, flag news information mobile government, small source science and technology gold Melt between the systems such as intelligent SMS due to being related to the transmitting of the information such as finance, government affairs, finance, is examined for real-time and the dual of safety Consider, is widely used synchronous transmitting data mode, and use the data formats such as META, file, XML, JSON, picture, completion must Show the acquisition of information and the circulation of operation flow.Product operation platform with if scientific and technological flow bank, the profound news of Black Warrior science and technology Disappear fastly, AsiaINFO neighbour happy converge etc. needs the reserved customized treatment time due to selling product between systems after order, for production The considerations of product reliability, is widely used asynchronous transfer of data mode, and uses the data lattice such as OFFICE, file, JSON, picture Formula completes the circulation of the necessary acquisition for showing information and operation flow.

Due to business platform substantial amounts, thus the business datum amount that interacts of product operation platform and business platform also compared with Greatly, and the data format that uses of each business platform is different, product operation that therefore, in the embodiment of the present invention can be shown in Fig. 1 Kafka message system shown in Fig. 2 is embedded in system, is each industry using Kafka message system as data pipe and message system Be engaged in module between message-driven provide data transmission capabilities support, and realize different data format business datum transmission and Receive and product operation platform and different kinds of business platform between either synchronously or asynchronously transmission demand.

In Fig. 2, Broker can be understood as the one or more separate servers for including in Kafka cluster, these clothes Business device is commonly known as Broker.Producer, it can be understood as be the message producer, be responsible for giving out information to Kafka Broker.Consumer, it can be understood as be message consumer, be responsible for receiving message from Kafka Broker.It is one or more Consumer can be divided into consumer and be grouped (Consumer Group), can usually be divided according to actual business requirement Consumer Group.Every message that Producer is published to Kafka cluster has a classification, this classification is Topic.Physically the message of difference Topic is stored separately, and in logic although the message of a Topic is stored in one or more On Broker, but user need to only specify the Topic of message can produce or consumption data without be concerned about data where be stored in. Kafka is carried out message management with Topic and is divided into each Topic multiple using subregion (Partition) mechanism The corresponding logic Log of Partition, each Partition, is made of multiple Segment.It is stored in each Segment more Message (message).Message is the basic unit communicated in Kafka message system, and each Producer can be to one Topic issues any multiple messages.Each Partition corresponds to an index (Index) in memory, each to record First message offset in Segment.The message that Producer is published to some Topic can be evenly dispersed to multiple On Partition, Broker is received give out information after add the message on the last one Segment of corresponding Partition. When the message bar number on some Segment reaches Configuration Values or the news release time is more than threshold value, these message can be automatic It is written on disk.Data, Broker meeting will not be written after reaching a certain size in the Segment scale of construction into the Segment again New Segment is created in due course.Each Partition is made of a series of orderly, immutable message, these message are connected It is continuous to be appended in Partition.Each message in Partition has a continuous assigned sequence number to be called Offset, is used for Partition unique identification a piece of news, and Kafka cluster can navigate to the storage of message by message identifier Position effectively prevents the additional mapping for being identified to position.Zookeeper realization can be used between Broker and Consumer Load balancing.All Broker and Consumer can be registered in Zookeeper, and Zookeeper can automatically save some members Data information.If some Broker and Consumer change, other Broker and Consumer can be notified.

The above-mentioned product operation system for being embedded with Kafka message system, it can be understood as be the business datum based on Kafka Interactive system includes Kafka cluster and multiple operation systems (each industry of example as shown in figure 1 in the service data interaction system Business platform).Operation system involved in the embodiment of the present invention can have the function of Producer, can also have the function of Consumer, And have to Kafka cluster with Kafka form of message issuing service data and from Kafka cluster with the reception of Kafka form of message The ability of business datum.The operation system with Producer function is known as the first industry for convenience of description in the embodiment of the present invention Operation system with Consumer function is known as the second operation system by business system, wherein the first operation system and the second industry Business system can be identical operation system, can also be different operation systems.

Fig. 3 show a kind of service data interaction system schematic based on Kafka provided in an embodiment of the present invention.Fig. 3 In, the first operation system has the function of Producer, for obtain first operation system and second operation system it Between the key-value pair shared, the key-value pair is used to identify to interact between first operation system and second operation system Business datum, first operation system determine target according to the zoning schemes of the key-value pair and the Kafka cluster Partition, and the business datum is distributed to the target Partition.The Kafka cluster, it is described for receiving The business datum of first operation system publication, and the business datum is stored to the target Partition.Second industry Business system has the function of Consumer, for obtaining the key-value pair, and to key-value pair described in the Kafka collection pocket transmission, orders Read the business datum interacted between first operation system and second operation system.The Kafka cluster is for receiving The key-value pair that second operation system is sent, the Partition mechanism according to the key-value pair and the Kafka cluster are true The fixed target Partition, and the business datum is obtained in the target Partition, the business datum is sent out It send to second operation system.Second operation system receives the business datum of the Kafka collection pocket transmission.

In the embodiment of the present invention, key-value pair is shared between the first operation system and the second operation system, and utilize key-value pair Target Partition is determined with the zoning schemes of Kafka cluster, and determining target is issued and be stored in business datum Partition, Maintenance free mapping relations Hash table, and without executing mapping table initialization, full library searching mapping pass It is the process of Hash table, therefore can reduce the complexity and expense of operation system interactive process to a certain extent.

In the embodiment of the present invention for convenience of description, the first operation system and Producer are often used alternatingly sometimes, and second Operation system and Consumer are often used alternatingly, it will be understood by those skilled in the art that its meaning is identical concept.

Specifically, Consumer receive data be according to grouping (Customer Group) Lai Jinhang, therefore, the present invention is real It applies in example, the above-mentioned key-value pair being related to can be understood as sharing between each Consumer in Producer and Consumer group Key assignments, the Consumer quantity for wherein including in Consumer group can be one, can also be multiple.Wherein, above-mentioned to be related to Shared key-value pair can be and consulted in advance between Producer and Consumer, be also possible to using scheduled generations rule Then generate.For example, the first operation system and second operation system as described in obtaining under type can be used in the first operation system Between the key-value pair shared: the identity information of first operation system is encrypted, first operation system is generated First identity, first identity can be understood as the first operation system in the service data interaction based on Kafka Unique identity in system.The identity information of second operation system is obtained, and to the body of second operation system Part information is encrypted, and the second identity is generated, which can be understood as the first operation system and be based on Unique information push mark, i.e., can determine first by second identity in the service data interaction system of Kafka Message is pushed to the second operation system by operation system.First identity and second identity are added It is close, message queue identification code is generated, using the message queue identification code as first operation system and second business The key-value pair shared between system.Similar, the key-value pair as described in obtaining under type can be used in second operation system: to institute The identity information for stating the second operation system is encrypted, and the second identity is generated, which can be understood as The unique identity in the service data interaction system based on Kafka of second operation system.Obtain first business system The identity information of system, and the identity information of first operation system is encrypted, generate the first identity, second body Part mark can be understood as the first operation system unique information push mark in the service data interaction system based on Kafka Know.It is encrypted, is generated message queue identification code (Key) according to second identity and first identity, Using the message queue identification code as the key-value pair shared between first operation system and second operation system.

In a kind of possible embodiment, the first operation system and the second operation system generate the above-mentioned message queue being related to Identification code (Key) is synchronized in Kafka cluster in each database for including, so that Kafka cluster can be based on the message queue mark Know the push that code (Key) carries out message.

Such as md5 encryption algorithm can be used to generate first for the first operation system and the second operation system in the embodiment of the present invention Identity and the second identity.Specific encryption realization process, can be used existing prior art and realizes, the embodiment of the present invention No longer it is described in detail.

In a kind of possible embodiment, key-value pair may be configured as the mark that can be resolved to integer in the embodiment of the present invention Know code.Producer determines the target Partition of business datum storage in the key-value pair that can be resolved to integer according to this When, the key-value pair can be parsed, and integer that the key-value pair obtains will be parsed and the Kafka cluster is the business datum The Partition quantity for including in the theme of division carries out remainder operation；The number that Partition serial number and remainder are operated It is worth consistent Partition, as the target Partition.

Wherein, the Partition quantity for including in the theme that the Kafka cluster divides for the business datum can pass through The Paritition.Class of Producer this parameter is specified, which must realize Kafka.Producer.Partitioner interface.In the embodiment of the present invention, Key can be resolved to integer, therefore can will parse The Partition sum remainder for including in the theme that the integer and the business datum that Key is obtained divide, the message can be sent The corresponding Partition of numerical value obtained after to serial number remainder.If using the class in code shown in Fig. 4 as Partition.Class, and 20 message (Key is respectively 0,1,2,3) are sent to Topic3 (comprising 4 by code shown in Fig. 4 A Partition), each Parition can have a serial number, and serial number is since 0, then the identical message of Key can be sent and be deposited It stores up in the same Partition, and the serial number of Key is just identical with Partition serial number.Wherein, Partition serial number Since 0, the Key in this example is also since 0.

Above by the first operation system and the second operation system in the embodiment of the present invention, by business rule it is preset it is unique about Determine Key, the implementation that directly executes data transmission can not only save that read exterior arrangement file, mapping table initial Change, the calculating process that full library searching mapping Hash table etc. complexities are high, system consumption is big, can also using in Kafka Distributed nature improves system performance linearly.

Above-mentioned the first operation system being related to can be different operation system or same business in the embodiment of the present invention The different subservice systems for including in system, the message of the first operation system publication may be polymorphic type in such cases, therefore Kafka cluster can be the business datum or be to wrap in same operation system that different operation systems is issued in the embodiment of the present invention The business datum of the different subservice systems publication included, divides different Topic, is independent from each other between different Topic.By There is inherent distributed nature in Kafka itself, it can will be each in order to realize the high scalability of system, in the embodiment of the present invention Theme (for example, customer information is synchronous, product information is shown, confirmation etc. is ordered in payment) with broad sense is distributed to Kafka On multiple Kafka servers (i.e. Broker) in cluster, at the same for each theme Topic establish it is multiple and different Partition, and the specified corresponding Partition created in Topic creation.Each Partition, which is one, asynchronous to be had Sequence queue is responsible for storage, transmits allocated asynchronous message body (Message), and the same Partition may insure therein Self-holding mode is sent to Consumer to PUSH message in order, but is independent from each other between multiple Partition, specific such as Fig. 5 It is shown.Wherein, Partition serial number is since 0.

In the embodiment of the present invention, Kafka cluster is divided into one or more Partition to Topic, and Message in Partition is that have the message of key-value pair, and the message with identical key-value pair can enter same The message of Partition, the same Partition can be sent to identical Consumer.In this way with identical key assignments message, Theme, Partition and Consumer have certain mapping relations, as shown in fig. 6, Producera passes through Partition1 Message 0 and message 1 are sent to Consumera, are sent to message 0, message 1 and message 2 by Partition2 Consumerb.Message 0 and message 1 are sent to Consumerc by Partition3 by Producerb.

When it is implemented, it is according to grouping (Customer Group) Lai Jinhang, Kafka collection that Consumer, which receives data, Group may insure that the data of each Partition transmission are disappeared by the same Consumer in the same Customer Group Take.If it is intended to Double Spending, then needing other Customer Group to consume.Partition number in each Topic Amount determine this Topic in same Customer Group by can be balanced degree, more Partitions means can be with More Consumer are accommodated, the ability of concurrent consumptions is also just effectively improved.For example, provided in an embodiment of the present invention be based on When the service data interaction system of Kafka is applied to such as production interactive system shown in Fig. 2, product operation platform is needed from red Production sharing information is obtained in the business platforms such as cloud video is logical, flow bank, the pervasive OA of wechat, then is placed them in same It is similar in Customer Group, electronic invoice, financial intelligent communication can be placed in the same Customer Group In, then using the same Partition under same Topic, the Consumer into Customer Group is sent, thus Realize broadcast of the Topic message for several Consumer.If desired wearing is obtained from Intelligent police service wearable device to set The Intelligent police service wearable device, can be individually placed in a Customer Group by standby unique encodings, to realize one Unicast of the Topic message for a Consumer, specific message push networking schematic diagram can be as shown in Figure 7.

Specifically, when the first operation system (Producer) is to Kafka cluster issuing service data in the embodiment of the present invention, Business datum can be encapsulated as to Kafka message body and be published to the target Partition determined in Kafka cluster.

Further, in the embodiment of the present invention the first operation system (Producer) can by the attribute information of business datum, It is encapsulated as Kafka message body with the key-value pair and the business datum, and is distributed to the target Partition, the industry The attribute information for data of being engaged in includes in the transmission mode of business datum, the storage format of business datum and traffic data type At least one of.The Kafka cluster is also used to parse Kafka message body, and stores the attribute information of the business datum, so as to Subsequent Consumer can subscribe to relevant information, Kafka collection according to the attribute information of the business datum recorded in Kafka cluster The message queue that group can also subscribe to according to the attribute information flexible combination Consumer of the business datum of record, the business system made It unites and can be achieved to carry out the interaction of different kinds of business data, and transmission either synchronously or asynchronously between different business systems.

The transmission mode of business datum can pass through configuration parameter Producer.Type by Producer in the embodiment of the present invention It is specified.Specifically, configuration parameter Producer.Type can indicate in background thread the sending method of message be it is synchronous or Asynchronous, default is synchronous mode, i.e. Producer.Type=Sync.It, only need to be by it if business datum needs asynchronous transmission It is arranged to asynchronous mode, i.e. Producer.Type=Async.Setting can also be passed through in the embodiment of the present invention by Producer Batch attribute makes Producer with batch style issuing service data, thus the data processing and transmission of significant increase Broker Performance, but also will increase the risk of loss of data in this way.If it is this kind of right for flow bank, electronic invoice, on-line finance In data transmission accuracy, data real-time response, the high requests such as reliable are returned the result, then must set the side of synchronization for the parameter Formula Sync.

Treatment effeciency, Producer can be improved in asynchronous push data in a manner of batch (Batch) in the embodiment of the present invention Message can be accumulated to after certain amount in memory and send Asynchronous Request as a Batch.The population size of Batch can To be controlled by the parameter of Producer (Batch.Num.Messages).By the size for increasing Batch, it is possible to reduce network The number of request and disk I/O, certain design parameter setting need to do a tradeoff in terms of efficiency and timeliness.The present invention is real Applying in example can also configure the quantity of issuing service data by the way that this parameter of Batch.Size is arranged.

Further, in the embodiment of the present invention, Producer can also by be arranged the Producer buffered message time, The message of largest buffered message number, Producer obstruction waiting time and Producer caching in Producer buffer queue The transmission configurations parameter such as quantity, the transmission mode of configuration service data.

Theme division is carried out to different business systems or different business subsystem by above-mentioned in the embodiment of the present invention, with And the mode of the attribute information of business datum is saved, the horizontal extension of message queue may be implemented, and realize distributed message.

Further, the first operation system described in the embodiment of the present invention is also used to encrypt the business datum, And encrypted business datum is distributed to the target Partition.Second operation system is also used to receive encryption Business datum, and the business datum is decrypted.

Specifically, accessing needs according to the product of different business systems in the embodiment of the present invention, while considering operation flow There is the possibility attacked and stolen in key message, can be to Partial key business number in generation, transmission, forwarding, receive process According to encryption and decryption processing is carried out, to guarantee the safety and reliability in data transmission procedure.

Specifically, in the embodiment of the present invention Producer by Topic to Kafka Broker issuing service data, Kafka cluster is to carry out message category with Topic, and Topic can be grouped in physical layer with Partition again, one Topic is segmented into several Partition.If using Partition as minimum memory unit, when any number of production systems When constantly sending message to product operation platform, expanding without limitation for Partition file will necessarily be caused, in this way for message The maintenance of file and the cleaning tape for having consumed message carry out serious influence, therefore Kafka cluster is using as follows in the embodiment of the present invention Mode stores the business datum to the target Partition: by the target Partition to be sliced (Segment) It is divided, the business datum is stored to the slice of the target Partition, such as shown in Fig. 8 for unit.Each Partition, which is equivalent to a huge file and is averaged, to be assigned in multiple equal-sized Segment data files, certainly every Message number is not necessarily equal in a Segment file, and this characteristic also facilitates the deletion of the longer Segment of storage time, The cleaning of the i.e. convenient message consumed, improves the utilization rate of disk.

It is read and write specifically, each Partition needs support sequence, the file life cycle of Segment is matched by server-side Set parameter (several parameters such as Log.Segment.Bytes, Log.Roll. { Ms, Hours }) decision.Segment file is by two It is grouped as, respectively " .Index " file and " .Log " file, is expressed as Segment index file and data file.This The naming rule of two files are as follows: first Segment of the Partition overall situation is since 0, subsequent each Segment file The Offset value of entitled upper Segment file the last item message, numerical values recited are 64,20 bit digital character lengths, It is specific as shown in Figure 9 without 0 filling of number.Wherein .Index " file and " .Log " file have corresponding management, with Fig. 9 Shown in be illustrated for Segment file, Figure 10 shows Segment:00000000000000170410's The corresponding relationship of " .Index " file and " .Log " file.

In the embodiment of the present invention, " .Index " index file stores a large amount of metadata, and the storage of " .Log " data file is big The message of amount, the metadata in index file are directed toward the physical deflection address of Message in corresponding data file.Wherein with For metadata [3,348] in " .Index " index file, the 3rd message is indicated in " .Log " data file, that is, is existed 170410+3=170413 message is indicated in Partition, the physical deflection address of the message is 348.

Kafka cluster produces offset (Offset) of the business datum in Segment in the embodiment of the present invention, And storage location of the business datum in the target Partition is identified using the offset.Specifically, Kafka After cluster receives the business datum of Producer publication, it can be placed in a certain Partition.Every message (business Data) it in Partition is all that the storage location that is indicated by Offset it in this Partition is certain, this Offset not instead of actual storage locations of the message in Partition data file, a logical value.The present invention is implemented Business datum can be also encapsulated as to Message class in example, there are the attribute informations such as Offset, Messagesize and Data.Its In, Offset is Long type, and Messagesize Int32 indicates size of data, and Data is the particular content of Message, lattice Formula is consistent with the Messageset in Kafka communications protocol.Entire data file then contains symbol in the embodiment of the present invention Several message instances for stating format are closed, are arranged together by Offset is ascending.We be encapsulated for Filemessageset class, class figure are as shown in figure 11.

Kafka cluster is by can be used if under type is described after Offset identification service data in the embodiment of the present invention The business datum is obtained in target Partition: using the Offset for identifying the business datum, in the target The Segment for storing the business datum is determined in Partition, and the business datum is obtained in the Segment, this The opposite mode for crossing interface programming of kind of mode is identified in publisher simply adds between message traffic identifier ": " character and press character The mode being connected of going here and there, which combines, constitutes unique mark key for receiving Kafka message, realizes more simple.For example, When reading the message that Offset is 170418 in Segment, Segment file is first looked for, wherein 00000000000000000000.Index is the file most started, and second file is 00000000000000170410.Index (start offset 170410+1=170411), and third file is 00000000000000239430.Index (start offset 239430+1=239431), so this Offset=170418 It has just fallen among second file.Other subsequent files can with and so on, named with its real offset and arrange these Then file can quickly navigate to specific document location according to binary chop.Secondly, according to index file: [8,1325] in 00000000000000170410.Index navigate in 00000000000000170410.Log file 1325 position is read out.

Further, the Kafka cluster is also used to the business number in the deletion of the deletion strategy based on setting Segment According to discharge memory space.Specifically, can be deleted when determining that the capacity for storing the fragment of the business datum exceeds given threshold Except the storage time of business datum or the business datum is beyond setting duration, then the business datum is deleted.

Kafka cluster can also periodically delete the business datum of setting in the embodiment of the present invention, execute in the embodiment of the present invention Periodically file has been consumed in deletion and Kafka performance boost relationship is little, i.e., how to select deletion strategy only and disk monitors size Specific requirements it is related.Currently disappear in addition, system can be used as some metadata informations of each Consumer Group reservation The record information (message repository location, it is understood that be Offset) taken.This Offset is controlled by Consumer, normally In the case of Consumer can after having consumed a piece of news this linearly increasing Offset.Because Offset is controlled by Consumer System, thus Kafka Broker be it is stateless, it does not need to mark which message by which Consumer post-consumer, does not need It goes to guarantee that same only one Consumer of Consumer Group can consume certain a piece of news by Broker, therefore is not required to It realizes lock mechanism, thus provides powerful guarantee for the high data throughput of Kafka.

Service data interaction system based on above-mentioned offer, the embodiment of the invention also provides a kind of service data interaction sides Method, this method can be applied in the first operation system for realizing Producer function, it is understood that be to be applied to In Producer.Figure 12 show a kind of service data interaction side applied to the first operation system provided in an embodiment of the present invention Method implementation flow chart, refering to fig. 1 shown in 2, this method comprises:

S101: the business datum interacted between the first operation system and the second operation system is obtained.

S102: the key-value pair shared between first operation system and the second operation system is obtained.Specifically, the second industry The quantity of business system be at least one, and at least one described second operation system shared with first operation system it is identical Key-value pair, i.e., the first operation system and one group of second operation system share key-value pair in the embodiment of the present invention, and second in the group Operation system quantity can be one, or multiple.

In the embodiment of the present invention, the key-value pair shared between the first operation system and the second operation system can be used for identifying institute State the business datum interacted between the first operation system and second operation system.

S103: the industry that the zoning schemes according to the key-value pair and Kafka cluster that obtain determine target partition, and will acquire Data publication be engaged in determining target partition.

The first operation system can be by the transmission including business datum in a kind of possible embodiment in the embodiment of the present invention In mode, the storage format of business datum and traffic data type at least one of business datum attribute information, and it is described Key-value pair and the business datum are encapsulated as Kafka message body, and are distributed to the target partition, so that Kafka cluster-based storage The attribute information of the business datum, so that the subsequent attribute information according to business datum is grouped and the division of Topic.

Specifically, above-mentioned first operation system obtains the key-value pair shared between the first operation system and the second operation system The realization of method flow shown in Figure 13 can be used:

S1021: encrypting the identity information of the first operation system, generates the first identity mark of the first operation system Know.

S1022: obtaining the identity information of the second operation system, and encrypt to the identity information of the second operation system, Generate the second identity.

S1033: encrypting the first identity and the second identity, generates message queue identification code, will disappear Queue identity code is ceased as the key-value pair shared between the first operation system and the second operation system.

Further, the above-mentioned key-value pair being related to can be the identification code that can be resolved to integer, the first operation system It is all as shown in figure 14 that zoning schemes according to key-value pair and Kafka cluster determine that the specific implementation process of target partition can be used Method flow, refering to fig. 1 shown in 4, comprising:

S1031: the key-value pair being analyzed and acquired by.

S1032: the integer that the key-value pair obtains and the master that the Kafka cluster is business datum division will be parsed The number of partitions for including in topic carries out remainder operation.

Specifically, the first operation system can be different sub for include in different operation system or same operation system Operation system.Include in the business datum or same operation system that Kafka cluster can be issued for different operation systems The business datum of different subservice system publications, divides different themes.It can determine that Kafka cluster is the in the embodiment of the present invention The Topic that the business datum that one operation system is interacted with the second operation system divides, and determine the number of partitions for including in the Topic Amount.

S1033: the consistent subregion of numerical value that subregion serial number and remainder are operated, as determining target partition.

Service data interaction method provided in an embodiment of the present invention, the first operation system utilize key-value pair and Kafka cluster Zoning schemes determine target partition, business datum is issued and is stored in determining target partition, Maintenance free mapping is closed It is Hash table, and the process without executing mapping table initialization, full library searching mapping relations Hash table, therefore to a certain degree On can reduce the complexity and expense of operation system interactive process.

The embodiment of the invention also provides service data interaction method, which can be applied to Kafka Cluster, it is understood that be to be applied to Broker.Figure 15 show provided in an embodiment of the present invention a kind of applied to Kafka collection The service data interaction method of group, refering to fig. 1 shown in 5, this method comprises:

S201: the business datum of the first operation system publication is received, and the business datum is stored to target partition.

In the embodiment of the present invention, target partition is first operation system according to key-value pair and the Kafka cluster Zoning schemes determine that the key-value pair interacts between first operation system and second operation system for identifying Business datum.

Optionally, the key-value pair is the identification code that can be resolved to integer.

Optionally, the business datum issued for different operation system is difference for including in same operation system The business datum of operation system publication, divides different themes.

Specifically, Kafka cluster can store the attribute information of business datum to target partition with the business datum. Wherein, the attribute information of the business datum includes the transmission mode of business datum, the storage format of business datum and business At least one of in data type.

Further, Kafka cluster can also divide the target partition as unit of being sliced, by the business number According to storing to the slice of the target partition, offset of the business datum in the slice is generated, and described in utilization Offset identifies storage location of the business datum in the target partition.

S202: the key-value pair and subscribe to request that the second operation system is sent are received.

In the embodiment of the present invention, request is subscribed to for subscribing between first operation system and second operation system Interactive business datum.

S203: the zoning schemes according to the key-value pair and the Kafka cluster determine the target partition, and in institute It states and obtains the business datum in target partition.

In the embodiment of the present invention, Kafka cluster is determined according to the zoning schemes of the key-value pair and the Kafka cluster The target partition, which can be used, determines that the similar mode of target partition determines with the first operation system, such as: parse the key assignments It is right, and include by parsing in the theme that the integer that the key-value pair obtains is divided with the Kafka cluster for the business datum Number of partitions carry out remainder operation, the consistent subregion of numerical value that subregion serial number and remainder are operated, as the target Subregion.

Wherein, Kafka cluster can be determined in the target partition using the offset for identifying the business datum The fragment of the business datum is stored, and obtains the business datum in the fragment.

S204: the business datum that will acquire is sent to the second operation system.

Service data interaction method provided in an embodiment of the present invention, Kafka cluster utilize the first operation system and the second industry The zoning schemes of the shared key-value pair of business system and Kafka cluster determine target partition, and business datum is issued and stored In determining target partition, Maintenance free mapping relations Hash table, and it is not necessarily to execute mapping table initialization, full library searching The process of mapping relations Hash table, therefore can reduce the complexity and expense of operation system interactive process to a certain extent.

The embodiment of the present invention also provides a kind of service data interaction method, and this method can be applied to realize Consumer function The second operation system, it is understood that can be applied to Consumer for this method.Figure 16 show offer of the embodiment of the present invention It is a kind of applied to realize Consumer function the second operation system method implementation flow chart, refering to fig. 1 shown in 6, the party Method includes:

S301: the key-value pair shared between the first operation system and second operation system is obtained.

In the embodiment of the present invention, the key-value pair for identify first operation system and second operation system it Between interaction business datum.

Specifically, the mode similar with the first system acquisition key-value pair, which can be used, in the second operation system obtains key-value pair, example Such as: the identity information of second operation system being encrypted, the second identity is generated；Obtain first operation system Identity information, and the identity information of first operation system is encrypted, generates the first identity；According to described Two identity and first identity are encrypted, and generate message queue identification code, the message queue is identified Code is as the key-value pair shared between first operation system and second operation system.

S302: the key-value pair obtained to Kafka collection pocket transmission, and send subscription request.

Wherein, described to subscribe to request and interacted between first operation system and second operation system for subscribing to Business datum.

S303: the business datum of Kafka collection pocket transmission is received.

Specifically, the business datum of the Kafka collection pocket transmission is obtained in target partition by the Kafka cluster, institute Zoning schemes of the target partition by the Kafka cluster according to the key-value pair and the Kafka cluster are stated to determine.

Service data interaction method provided in an embodiment of the present invention, the second operation system send key-value pair to Kafka cluster, So that Kafka cluster determines target partition using the zoning schemes of key-value pair and Kafka cluster, business datum transmission will acquire To the second operation system, so that the second operation system Maintenance free mapping relations Hash table, and without executing mapping table It initializes, the process of full library searching mapping relations Hash table, therefore can reduce answering for operation system interactive process to a certain extent Miscellaneous degree and expense.

Based on identical inventive concept, the embodiment of the present invention provides a kind of service data interaction device, which hands over Mutual device can be applied in the first operation system for realizing Producer function.Figure 17 show provided in an embodiment of the present invention answer For realizing the service data interaction apparatus structure schematic diagram of the first operation system of Producer function, refering to fig. 1 shown in 7, The device includes acquiring unit 101, processing unit 102 and release unit 103.Wherein, acquiring unit 101, for obtaining first Between the business datum and the first operation system and the second operation system interacted between operation system and the second operation system altogether The key-value pair enjoyed, the key-value pair is for identifying the business interacted between first operation system and second operation system Data.The zoning schemes of processing unit 102, key-value pair and Kafka cluster for obtaining according to the acquiring unit 101 are true Set the goal subregion.Release unit 103, the business datum for obtaining the acquiring unit 101 are distributed to the processing unit 102 target partitions determined.

In a kind of possible embodiment, the key-value pair is the identification code that can be resolved to integer.The processing is single Zoning schemes of the member 102 in the following way according to the key-value pair and Kafka cluster determine the target partition:

Optionally, the acquiring unit 101 obtains first operation system and the second operation system in the following way Between share key-value pair:

Optionally, the business datum is distributed to the target partition in the following way by the release unit 103:

Based on identical inventive concept, the embodiment of the present invention provides a kind of service data interaction device, which hands over Mutual device can be applied in Kafka cluster.Figure 18 show the business number provided in an embodiment of the present invention applied to Kafka cluster According to interactive device structural schematic diagram, refering to fig. 1 shown in 8, which includes receiving unit 201, storage unit 202, processing unit 203 and transmission unit 204.Wherein, receiving unit 201 for receiving the business datum of the first operation system publication, and receive the The key-value pair and subscribe to request that two operation systems are sent.Storage unit 202 is used for the received industry of the receiving unit 201 Business data are stored to target partition, and the target partition is first operation system according to key-value pair and the Kafka collection What the zoning schemes of group determined, the key-value pair is handed between first operation system and second operation system for identifying Mutual business datum.Processing unit 203, key-value pair and the Kafka collection for being received according to the receiving unit 201 The zoning schemes of group determine target partition, and the business datum is obtained in the target partition.Transmission unit 204, is used for The business datum that the processing unit 203 determines is sent to second operation system.

Wherein, the key-value pair is the identification code that can be resolved to integer.The processing unit 203 is in the following way Zoning schemes according to the key-value pair and the Kafka cluster determine the target partition:

Optionally, the processing unit 203 is also used to: storing the business datum to mesh in the storage unit 202 It is the business datum or be the different subservices for including in same operation system that different operation systems is issued before marking subregion The business datum of system publication, divides different themes.

Further, the storage unit 202 in the following way stores the business datum to target partition:

The processing unit 203 obtains the business datum in the target partition in the following way: utilizing mark The offset of the business datum determines the fragment for storing the business datum in the target partition, and described The business datum is obtained in fragment.

Based on identical inventive concept, the embodiment of the present invention also provides a kind of service data interaction device, the business datum Interactive device can be applied in the second operation system for realizing Consumer function.Figure 19 show provided in an embodiment of the present invention Applied to the service data interaction apparatus structure schematic diagram for the second operation system for realizing Consumer function, refering to fig. 19 institute Show, which includes processing unit 301, transmission unit 302 and receiving unit 304.Wherein, processing unit 301, for obtaining the Between one operation system and the second operation system share key-value pair, the key-value pair for identify first operation system with The business datum of interaction between second operation system.Transmission unit 302 is used for Kafka collection pocket transmission processing unit 301 Determining key-value pair, and send subscription request.The subscription request is for subscribing to first operation system and second industry The business datum of interaction between business system.Receiving unit 303, it is described for receiving the business datum of Kafka collection pocket transmission The business datum of Kafka collection pocket transmission is obtained in target partition by the Kafka cluster, and the target partition is by described Kafka cluster is determined according to the zoning schemes of the key-value pair and the Kafka cluster.

Optionally, the processing unit 301 is obtained in the following way between the first operation system and the second operation system Shared key-value pair:

The embodiment of the present invention also provides a kind of service data interaction equipment, which can be applied to Producer also can be applied to Broker, can also be applied to Consumer.Figure 20 show provided in an embodiment of the present invention A kind of structural schematic diagram of service data interaction equipment is handed over refering to the business datum shown in Figure 20, provided in the embodiment of the present invention Mutual equipment includes memory 1001, processor 1002 and is stored in the meter that can be run on the memory and on the processor Calculation machine program 1003, the processor 1002 realize above-mentioned the first operation system being related to, the second business when executing described program System or the method for Kafka group service data interaction.

It should be noted that the above-mentioned processor being related to can be central processing unit (Central Processing Unit, CPU), general processor, digital signal processor (Digital Signal Processor, DSP), dedicated integrated electricity Road (Application-Specific Integrated Circuit, ASIC), field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic device, transistor logic, hardware component or Person's any combination thereof.It, which may be implemented or executes, combines various illustrative logic blocks described in present disclosure, Module and circuit.Processor is also possible to realize the combination of computing function, such as combines comprising one or more microprocessors, DSP and the combination of microprocessor etc..

Wherein, the memory can integrate in the processor, can also be provided separately with the processor.

It is understood that illustrate only service data interaction device and business datum friendship in the embodiment of the present application attached drawing Mutual equipment simplifies design.In practical applications, however it is not limited to above structure.

The embodiment of the present invention also provides a kind of computer readable storage medium, is stored thereon with computer program, the program Realize that the first operation system, the second operation system or Kafka cluster as described in above-described embodiment are realized when being executed by processor Step in service data interaction method.

It should be understood by those skilled in the art that, the embodiment of the present invention can provide as the production of method, system or computer program Product.Therefore, in terms of the embodiment of the present invention can be used complete hardware embodiment, complete software embodiment or combine software and hardware Embodiment form.Moreover, it wherein includes computer available programs generation that the embodiment of the present invention, which can be used in one or more, The meter implemented in the computer-usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) of code The form of calculation machine program product.

The embodiment of the present invention be referring to according to the method for the embodiment of the present invention, equipment (system) and computer program product Flowchart and/or the block diagram describe.It should be understood that can be realized by computer program instructions in flowchart and/or the block diagram The combination of process and/or box in each flow and/or block and flowchart and/or the block diagram.It can provide these calculating Processing of the machine program instruction to general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices Device is to generate a machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute For realizing the function of being specified in one or more flows of the flowchart and/or one or more blocks of the block diagram Device.

These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates, Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or The function of being specified in multiple boxes.

These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one The step of function of being specified in a box or multiple boxes.

Obviously, those skilled in the art can carry out various modification and variations without departing from this hair to the embodiment of the present invention Bright spirit and scope.In this way, if these modifications and variations of the embodiment of the present invention belong to the claims in the present invention and its wait Within the scope of technology, then the present invention is also intended to include these modifications and variations.

Claims

1. a kind of service data interaction method, which is characterized in that applied to the first operation system for realizing Kafka producer's function In, which comprises

Obtain the business datum interacted between the first operation system and the second operation system and first operation system and the The key-value pair shared between two operation systems, the key-value pair is for identifying first operation system and second business system The business datum of interaction between system；

Zoning schemes according to the key-value pair and Kafka cluster determine target partition, and the business datum is distributed to The target partition.

2. the method as described in claim 1, which is characterized in that described to obtain first operation system and the second operation system Between share key-value pair, comprising:

The identity information of first operation system is encrypted, the first identity of first operation system is generated；

The identity information of second operation system is obtained, and the identity information of second operation system is encrypted, it is raw At the second identity；

First identity and second identity are encrypted, message queue identification code is generated, it will be described Message queue identification code is as the key-value pair shared between first operation system and second operation system.

3. method according to claim 1 or 2, which is characterized in that the key-value pair is the mark that can be resolved to integer Code；

The zoning schemes according to the key-value pair and Kafka cluster determine the target partition, comprising:

The key-value pair is parsed, and integer and the Kafka cluster that the key-value pair obtains will be parsed as business datum division Theme in include number of partitions carry out remainder operation；

4. the method as described in claim 1, which is characterized in that the quantity of second operation system is at least one, and institute It states at least one second operation system and first operation system shares identical key-value pair.

5. the method as described in claim 1, which is characterized in that the business datum is distributed to the target partition, comprising:

By the attribute information of business datum, it is encapsulated as Kafka message body with the key-value pair and the business datum, and be distributed to The target partition, the attribute information of the business datum include the transmission mode of business datum, the storage format of business datum At least one of and in traffic data type.

6. a kind of service data interaction method, which is characterized in that be applied to Kafka cluster, which comprises

The business datum of the first operation system publication is received, and the business datum is stored to target partition, the target point Area is that first operation system is determined according to the zoning schemes of key-value pair and the Kafka cluster, the key-value pair use The business datum interacted between mark first operation system and second operation system；

It receives the key-value pair of the second operation system transmission and subscribes to request, the subscription request is for subscribing to first business The business datum interacted between system and second operation system；

Zoning schemes according to the key-value pair and the Kafka cluster determine target partition, and in the target partition The business datum is obtained, the business datum is sent to second operation system.

7. method as claimed in claim 6, which is characterized in that the key-value pair is the identification code that can be resolved to integer；

It parses the key-value pair, and integer that the key-value pair obtains will be parsed and the Kafka cluster is the business datum The number of partitions for including in the theme of division carries out remainder operation；

8. the method for claim 7, which is characterized in that the business datum is stored to before target partition, it is described Method further include:

The business datum issued for different operation system is the different subservice systems hair for including in same operation system The business datum of cloth divides different themes.

9. method as claimed in claim 6, which is characterized in that storing the business datum to target partition, comprising:

The attribute information of the business datum includes the transmission mode of business datum, the storage format of business datum and business number According at least one in type.

10. method as claimed in claim 6, which is characterized in that storing the business datum to target partition, comprising:

The target partition is divided as unit of being sliced, the business datum is stored to the slice of the target partition It is interior, offset of the business datum in the slice is generated, and identify the business datum in institute using the offset State the storage location in target partition；

The business datum is obtained in the target partition, comprising:

Using the offset for identifying the business datum, point for storing the business datum is determined in the target partition Piece, and the business datum is obtained in the fragment.

11. a kind of service data interaction method, which is characterized in that applied to the second business system for realizing Kafka consumer's function System, comprising:

The key-value pair shared between the first operation system and the second operation system is obtained, the key-value pair is for identifying described first The business datum interacted between operation system and second operation system；

To key-value pair described in Kafka collection pocket transmission and request is subscribed to, and receives the business datum of the Kafka collection pocket transmission；

The subscription request is for subscribing to the business datum interacted between first operation system and second operation system；

The business datum of the Kafka collection pocket transmission is obtained in target partition by the Kafka cluster, the target partition by The Kafka cluster is determined according to the zoning schemes of the key-value pair and the Kafka cluster.

12. method as claimed in claim 11, which is characterized in that obtain the first operation system and second operation system it Between share key-value pair, comprising:

The identity information of first operation system is obtained, and the identity information of first operation system is encrypted, it is raw At the first identity；

It is encrypted according to second identity and first identity, message queue identification code is generated, by institute Message queue identification code is stated as the key-value pair shared between first operation system and second operation system.

13. a kind of service data interaction device, which is characterized in that applied to the first business system for realizing Kafka producer's function In system, comprising:

Acquiring unit, for obtaining the business datum interacted between the first operation system and the second operation system and the first industry Between business system and the second operation system share key-value pair, the key-value pair for identify first operation system with it is described The business datum of interaction between second operation system；

The zoning schemes of processing unit, key-value pair and Kafka cluster for obtaining according to the acquiring unit determine target Subregion；

Release unit, the business datum for obtaining the acquiring unit are distributed to the target that the processing unit determines Subregion.

14. device as claimed in claim 13, which is characterized in that the key-value pair is the mark that can be resolved to integer Code；

The processing unit determines the target according to the zoning schemes of the key-value pair and Kafka cluster in the following way Subregion:

15. a kind of service data interaction device, which is characterized in that be applied to Kafka cluster, comprising:

Receiving unit for receiving the business datum of the first operation system publication, and receives the key assignments that the second operation system is sent Pair and subscribe to request；

Storage unit, for storing the received business datum of the receiving unit to target partition, the target partition is institute State what the first operation system was determined according to the zoning schemes of key-value pair and the Kafka cluster, the key-value pair is for identifying The business datum interacted between first operation system and second operation system；

The zoning schemes of processing unit, key-value pair and the Kafka cluster for receiving according to the receiving unit are true Set the goal subregion, and the business datum is obtained in the target partition；

16. device as claimed in claim 15, which is characterized in that the processing unit is also used to:

17. a kind of service data interaction device, which is characterized in that applied to the second business system for realizing Kafka consumer's function System, comprising:

Processing unit, for obtaining the key-value pair shared between the first operation system and the second operation system, the key-value pair is used The business datum interacted between mark first operation system and second operation system；

Transmission unit, for key-value pair described in Kafka collection pocket transmission and subscription request, the subscription request to be described for subscribing to The business datum interacted between first operation system and second operation system；

Receiving unit, for receiving the business datum of Kafka collection pocket transmission, the business datum of the Kafka collection pocket transmission is by institute It states Kafka cluster to obtain in target partition, the target partition is by the Kafka cluster according to the key-value pair and described The zoning schemes of Kafka cluster determine.

18. a kind of service data interaction system, which is characterized in that described including Kafka cluster and at least one operation system It include the first operation system and realization Kafka consumer's function for realizing Kafka producer's function at least one operation system The second operation system；

First operation system has the function of realizing any one of claim 1 to 5 the method；

The Kafka cluster has the function of realizing any one of claim 6 to 10 the method；

Second operation system has the function of realizing any one of claim 11 to 12 the method.

19. a kind of service data interaction equipment, including memory, processor and it is stored on the memory and can be at the place The computer program run on reason device, which is characterized in that the processor realizes such as claim 1 to 5 when executing described program Described in any item service data interaction methods, or realize such as the described in any item service data interaction sides of claim 6 to 10 Method, or realize such as the described in any item service data interaction methods of claim 11 to 12.

20. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is by processor It is realized when execution such as service data interaction method described in any one of claim 1 to 5, or realization such as claim 6 to 10 times Service data interaction method described in one, or realize such as the described in any item service data interaction sides of claim 11 to 12 Method.