CN103384551A

CN103384551A - Virtual machine communication method and system based on PCIE network, and server

Info

Publication number: CN103384551A
Application number: CN2013800000429A
Authority: CN
Inventors: 李思聪; 邱军
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2013-01-04
Filing date: 2013-01-04
Publication date: 2013-11-06
Anticipated expiration: 2033-01-04
Also published as: WO2014106321A1; CN103384551B

Abstract

The invention provides a virtual machine communication method and system based on PCIE network. The system comprises a source host having a virtual machine monitor and at least one virtual machine, wherein the virtual machine monitor on the source host is communicated with the virtual machine on the source host, one virtual machine on the source host is used as the source virtual machine and network request blocked on a TCP/IP protocol stack is processed by the source virtual machine and sent to the virtual machine monitor on the source host machine, the data contained in the network request is sent to a PCIE NTB address space corresponded to the source host in a PCIE NTB address space corresponded to a target host of a target virtual machine by the virtual machine monitor on the source host according to the ID of the target virtual machine of the network request. The PCIE network is used as the communication mechanism among the virtual machines, so as to achieve quick communication among the virtual machines; the limit of a PCIE network is broken, and the PCIE NTB address space in a subnetwork management server is managed and shared by a subnetwork based on PCIE network, and the communication space among the virtual machines is improved.

Description

A kind of virtual machine communication method based on the PCIE network, server and system

Technical field

The invention belongs to the computer internet communication technical field, relate to a kind of virtual machine communication method, particularly relate to a kind of virtual machine communication method, server and system based on high-speed peripheral assembly interconnect (Peripheral Component Interconnect Express, PCIE) network.

Background technology

Along with the more deployment of multiple virtualization server, Intel Virtualization Technology development, network virtualization have occupied more and more consequence, and the communication between virtual machine (Virtual Machine is called for short VM) is also more and more frequent.

At present, between virtual machine, communication mainly is based on Ethernet.During two communication between virtual machines, at first the user's attitude application program in virtual machine can be called the Socket interface, by transmission control protocol (Transmission Control Protocol, be called for short TCP)/Internet Interconnection agreement (Internet Protocol, be called for short IP) storehouse, carry out actual forwarding and processing in monitor of virtual machine (Virtual Machine Monitor is called for short VMM).And ICP/IP protocol is more complicated, cause operating system (Operating System, abbreviation OS) protocol processes has occupied the very most of of network delay, in addition, in data transmission procedure, data Replica and interruption are processed and are caused host cpu (Central Processing Unit is called for short CPU) resource can't be absorbed in the processing customer service frequently.

Industry has also proposed to walk around in kernel the ICP/IP protocol stack by the mode of shared drive and has realized the method for high-speed communication between server internal virtual machine, but can only be applied at present the communication between the virtual machine of same physical machine inside.

Summary of the invention

The shortcoming of prior art, the object of the present invention is to provide a kind of virtual machine communication method based on the PCIE network, server and system in view of the above, is used for solving the slow problem of communication speed between the virtual machine on the different physical machine of prior art.

First aspect, the embodiment of the present invention provide a kind of virtual machine communication method based on high-speed peripheral assembly interconnect PCIE network, comprising:

Source virtual machine on source host is tackled network request on the ICP/IP protocol stack, and the data that comprise in described network request are sent to monitor of virtual machine on described source host;

Monitor of virtual machine on described source host is according to the destination address of described network request, determine the PCIE NTB address space of the destination host that the above source host of PCIE switch is corresponding, wherein, described destination host is the main frame under the purpose virtual machine of described network request, and the PCIE NTB address space of the destination host that described the above source host of PCIE switch is corresponding is the PCIE NTB address space of distributing on described PCIE switch in the PCIE NTB address space of described destination host corresponding to described source host;

The PCIE message that monitor of virtual machine on described source host will comprise described data sends to the PCIE NTB address space of destination host corresponding to described the above source host of PCIE switch, so as the monitor of virtual machine of described destination host according to the ID of the purpose virtual machine of the described network request in described PCIE message with the purpose virtual machine of described Data dissemination to described network request.

In the possible implementation of the first of first aspect, the monitor of virtual machine on described source host is determined the PCIE NTB address space of the destination host that the above source host of PCIE switch is corresponding according to the destination address of described network request, is specially:

Monitor of virtual machine on described source host is according to the destination address of described network request, determine the destination host of described network request, and the PCIE NTB address space information of the described destination host corresponding according to the described source host of local cache, determine the PCIE NTB address space of the described destination host that described the above source host of PCIE switch is corresponding;

The PCIE NTB address space information of the described destination host that wherein, described source host is corresponding is monitor of virtual machine on the described source host source virtual machine on described source host buffer memory when sending arp request and carrying out address resolution.

In conjunction with the possible implementation of the first of first aspect, in the possible implementation of the second, buffer memory when the source virtual machine transmission arp request on described source host is carried out address resolution comprises:

Described source virtual machine watch-dog intercepts described source virtual machine after the described arp request of the purpose virtual machine transmission of described network request, and the request message of the PCIE NTB address space information of the described destination host that the described source virtual machine of initiation inquiry is corresponding is to the subnet management server;

Monitor of virtual machine on described source host receives the response message that described subnet management server returns, the PCIE NTB address space information cache of the described destination host that the described source virtual machine that comprises in described response message is corresponding is in this locality, the PCIE NTB address space of the described destination host that the PCIE NTB address space of the described destination host that wherein, described source virtual machine is corresponding and described source host are corresponding is identical.

In conjunction with the possible implementation of the first of first aspect or first aspect or the possible implementation of the second of first aspect, in the third possible implementation, the PCIE message that monitor of virtual machine on described source host will comprise the data in described network request sends to the PCIE NTB address space of destination host corresponding to described the above source host of PCIE switch, comprising:

The data that monitor of virtual machine on described source host comprises described network request, the ID of described source virtual machine and source NTB port id are put into the payload of described PCIE message, with the ID of the purpose virtual machine of the described network request destination address as described PCIE message, send described PCIE message to described PCIE switch, make described PCIE switch described PCIE message be write in the PCIE NTB address space of destination host PCIE NTB address space corresponding to described source host;

Wherein, described source NTB port id is the ID of the NTB port that is connected with monitor of virtual machine on described source host in described PCIE switch.

In conjunction with the possible implementation of the second of first aspect or the third possible implementation of first aspect, in the 4th kind of possible implementation, also comprise:

All PCIE NTB address space carries out maintenance management by described subnet management server, and described whole PCIE NTB address spaces are divided into the subspace, PCIE NTB address of the umber identical with the main frame number of described PCIE switch connection;

The corresponding main frame in each subspace, described address is used for receiving the data that send to described respective hosts;

Be between address microvoid of each other host assignment in each subspace, described address, corresponding described other main frames between each described address microvoid, be used for receiving and come from the data that described other main frames send to subspace, described address respective hosts, wherein, described other main frames are except the respective hosts of subspace, described address, the main frame that is connected with described PCIE switch.

The 4th kind of possible implementation in conjunction with first aspect in the 5th kind of possible implementation, also comprises:

Described subnet management server distributes described whole PCIE NTB address space when described PCIE netinit, and notify All hosts with the PCIE NTB address space information of self, the PCIE NTB address space information of the corresponding described subnet management server of each main frame buffer memory is in order to can send to described subnet management server the request message of described inquiry destination host PCIE NTB address space.

Second to the 5th kind of arbitrary possible implementation in conjunction with first aspect in the 6th kind of possible implementation, also comprises:

Safeguard the corresponding relation that main frame and virtual machine are arranged on described subnet management server, and according to establishment, change, the described corresponding relation of destruction real-time update of virtual machine.

First to the 6th kind of arbitrary possible implementation in conjunction with first aspect or first aspect, in the 7th kind of possible implementation, the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine of the described network request in described PCIE message, comprising:

The monitor of virtual machine of described destination host reads in the PCIE NTB address space of described destination host corresponding to the described PCIE message in the PCIE NTB address space of described source host, according to the ID of the purpose virtual machine of the described network request in described PCIE message with the purpose virtual machine of described Data dissemination to described network request.

Second aspect, the embodiment of the present invention provide a kind of virtual machine communication system based on high-speed peripheral assembly interconnect network, comprising: at least two main frames, a PCIE switch, and described at least two main frames are connected by described PCIE switch communication;

each main frame in described at least two main frames, be used for operation virtual machine and virtual machine monitoring module, wherein, source virtual machine on source host in described at least two main frames is tackled network request on the ICP/IP protocol stack, and the data that comprise in described network request are sent to the monitor of virtual machine of described source host, the monitor of virtual machine of described source host is according to the destination address of the described network request of described source virtual machine transmission, determine the PCIE NTB address space of the destination host that described the above source host of PCIE switch is corresponding, the PCIE message that will comprise the data in described network request sends to the PCIE NTB address space of described destination host corresponding to described the above source host of PCIE switch, so that the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine of the described network request in described PCIE message,

Wherein, described destination host is the main frame under the purpose virtual machine of described network request, and the PCIE NTB address space of the destination host that described the above source host of PCIE switch is corresponding is the PCIE NTB address space of distributing on described PCIE switch in the PCIE NTB address space of described destination host corresponding to described source host.

In the possible implementation of the first of second aspect, described each main frame comprises:

Virtual machine operation module is used for moving described virtual machine;

The virtual machine monitoring module is used for moving described virtual machine monitor.

In the possible implementation of the second of second aspect, described each main frame comprises: at least one processor, memory;

Described memory is used for storage and carries out instruction;

Described processor is used for when described main frame operation, and communicates by letter between described memory, carries out described execution instruction and makes described main frame move described virtual machine and described monitor of virtual machine.

In conjunction with the first of second aspect or second aspect or the possible implementation of the second of second aspect, in the third possible implementation, the monitor of virtual machine of described source host is according to the destination address of the described network request of described source virtual machine transmission, determine the PCIE NTB address space of the destination host that described the above source host of PCIE switch is corresponding, be specially:

Monitor of virtual machine on described source host is according to the destination address of the described network request of source virtual machine transmission, determine the destination host of described network request, and the PCIE NTB address space information of the described destination host corresponding according to the described source host of local cache, determine the PCIE NTB address space of the described destination host that described the above source host of PCIE switch is corresponding;

In conjunction with the third possible implementation of second aspect, in the 4th kind of possible implementation, also comprise the subnet management server;

, buffer memory when the source virtual machine transmission arp request of the monitor of virtual machine on described source host on described source host carried out address resolution, be specially: when the mac address information of the purpose virtual machine that there is no described network request on described source virtual machine, described source virtual machine sends described arp request to the purpose virtual machine of described network request, after monitor of virtual machine on described source host intercepts the described arp request of described virtual machine transmission, initiate the solicited message of PCIE NTB address space corresponding to the described purpose virtual machine of inquiry to described subnet management server, and after receiving the response message that described subnet management server returns, the PCIE NTB address space information cache of the destination host that the described source virtual machine that comprises in described response message is corresponding is in this locality, wherein, the PCIE NTB address space of the described destination host that the PCIE NTB address space of the described destination host that described source virtual machine is corresponding and described source host are corresponding is identical,

Described subnet management server, after the request message of the PCIE NTB address space that the described purpose virtual machine of inquiry that sends for the monitor of virtual machine that receives described source host is corresponding, determine destination host under described purpose virtual machine according to the corresponding relation of main frame and virtual machine, obtain the PCIE NTB address space of described destination host corresponding to described source host, send described response message to the monitor of virtual machine of described source host.

First to fourth kind of arbitrary possible implementation in conjunction with second aspect or second aspect, in the 5th kind of possible implementation, the PCIE message that monitor of virtual machine on described source host will comprise the data in described network request sends to the PCIE NTB address space of described destination host corresponding to described the above source host of PCIE switch, comprising:

the data that monitor of virtual machine on described source host comprises described network request, the ID of described source virtual machine and source NTB port id are put into the payload of described PCIE message, with the ID of the purpose virtual machine of the described network request destination address as described PCIE message, send described PCIE message to described PCIE switch, make described PCIE switch described PCIE message be write in the PCIE NTB address space of destination host PCIE NTB address space corresponding to described source host, wherein, described source NTB port id is the ID of the NTB port that is connected with monitor of virtual machine on described source host in described PCIE switch,

, described PCIE switch also is used for the PCIE NTB address space of the described destination host of the said write PCIE NTB address space corresponding to described source host.

In conjunction with the 4th kind or the 5th kind of possible implementation of second aspect, in the 6th kind of possible implementation:

described subnet management server, also be used for the whole PCIE NTB of maintenance management address space, described whole PCIE NTB address space is divided into the subspace, PCIE NTB address of the umber identical with the main frame number of described PCIE switch connection, the corresponding main frame in each subspace, described address, be used for receiving the data that send to described respective hosts, be between address microvoid of each other host assignment in each subspace, described address, corresponding described other main frames between each described address microvoid, be used for receiving and come from the data that described other main frames send to subspace, described address respective hosts, wherein, described other main frames are except the respective hosts of subspace, described address, the main frame that is connected with described PCIE switch.

In conjunction with the 6th kind of possible implementation of second aspect, in the 7th kind of possible implementation:

Described subnet management server, also be used for distributing described whole PCIE NTB address spaces when described PCIE netinit, and notify All hosts with the PCIE NTB address space information of self, thereby so that the PCIE NTB address space information of the corresponding described subnet management server of each main frame buffer memory can send to described subnet management server the request message of described inquiry destination host PCIE NTB address space.

In conjunction with the 4th kind to the 7th kind arbitrary possible implementation of second aspect, in the 8th kind of possible implementation:

Described subnet management server also is used for the corresponding relation of maintenance host and virtual machine, and according to establishment, change, the described corresponding relation of destruction real-time update of virtual machine.

Eight kinds of arbitrary possible implementations of the first to the in conjunction with second aspect or second aspect, in the 9th kind of possible implementation, the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine of the described network request in described PCIE message, comprising:

The third aspect, the embodiment of the present invention provide a kind of server, comprising:

Virtual machine operation module is used for the operation virtual machine;

The virtual machine monitoring module is used for operation and states virtual machine monitor;

Described virtual machine is used for tackling network request on the ICP/IP protocol stack, and the data that comprise in described network request is sent to monitor of virtual machine;

described monitor of virtual machine, be used for the destination address according to the described network request of described virtual machine transmission, determine the PCIE NTB address space of the destination host that the above server of PCIE switch is corresponding, the PCIE message that will comprise the data in described network request sends to the PCIE NTB address space of described destination host corresponding to described the above server of PCIE switch, so that the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine of the described network request in described PCIE message,

Wherein, described destination host is the main frame under the purpose virtual machine of described network request, and the PCIE NTB address space of the destination host that described the above server of PCIE switch is corresponding is the PCIE NTB address space of distributing on described PCIE switch in the PCIE NTB address space of described destination host corresponding to described server.

In the possible implementation of the first of the third aspect, described monitor of virtual machine is used for the destination address according to the described network request of described virtual machine transmission, determine the PCIE NTB address space of the destination host that the above server of PCIE switch is corresponding, be specially:

Described monitor of virtual machine, be used for the destination address according to the described network request of described virtual machine transmission, determine the destination host of described network request, and the PCIE NTB address space information of the described destination host corresponding according to the described server of local cache, determine the PCIE NTB address space of the described destination host that described the above server of PCIE switch is corresponding;

The PCIE NTB address space information of the described destination host that wherein, described server the is corresponding buffer memory that is described monitor of virtual machine when described virtual machine sends arp request and carries out address resolution.

In conjunction with the possible implementation of the first of the third aspect, in the possible implementation of the second, buffer memory when described monitor of virtual machine carries out address resolution in described virtual machine transmission arp request is specially:

Described virtual machine also is used for, and when the mac address information of the purpose virtual machine that there is no described network request on described virtual machine, sends described arp request to the purpose virtual machine of described network request;

described monitor of virtual machine, after also being used for intercepting the described arp request of described virtual machine transmission, initiate the solicited message of PCIE NTB address space corresponding to the described purpose virtual machine of inquiry to the subnet management server, and after receiving the response message that described subnet management server returns, the PCIE NTB address space information cache of the destination host that the described virtual machine that comprises in described response message is corresponding is in this locality, wherein, the PCIE NTB address space of the described destination host that the PCIE NTB address space of the described destination host that described virtual machine is corresponding and described server are corresponding is identical.

In conjunction with the possible implementation of the first of the third aspect or the third aspect or the possible implementation of the second of the third aspect, in the third possible implementation, the PCIE message that monitor of virtual machine on described source host will comprise the data in described network request sends to the PCIE NTB address space of described destination host corresponding to described the above server of PCIE switch, comprising:

Described monitor of virtual machine, for data, the ID of described virtual machine and the payload that source NTB port id is put into described PCIE message that described network request is comprised, with the ID of the purpose virtual machine of the described network request destination address as described PCIE message, send described PCIE message to described PCIE switch, make described PCIE switch described PCIE message be write in the PCIE NTB address space of destination host PCIE NTB address space corresponding to described server;

Wherein, described source NTB port id is the ID of the NTB port that is connected with described monitor of virtual machine in described PCIE switch.

In conjunction with second or the third possible implementation of the third aspect, in the 4th kind of possible implementation:

The PCIE NTB address space of the described destination host that described server is corresponding is managed by described subnet management server maintenance, is used for receiving coming from the data that described server sends to described destination host;

wherein, all PCIE NTB address space is divided into the subspace, PCIE NTB address of the umber identical with the main frame number of described PCIE switch connection, the corresponding main frame in each subspace, described address, be used for receiving the data that send to described respective hosts, be between address microvoid of each other host assignment in each subspace, described address, corresponding described other main frames between each described address microvoid, be used for receiving and come from the data that described other main frames send to subspace, described address respective hosts, described other main frames are except the respective hosts of subspace, described address, the main frame that is connected with described PCIE switch.

In conjunction with the 4th kind of possible implementation of the third aspect, in the 5th kind of possible implementation:

Described monitor of virtual machine is also for the PCIE NTB address space information that receives the subnet management server that also the described server of buffer memory is corresponding.

First to the 5th kind of arbitrary possible implementation in conjunction with the third aspect or the third aspect, in the 6th kind of possible implementation, the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine in described PCIE message, comprising:

The monitor of virtual machine of described destination host reads in the PCIE NTB address space of described destination host corresponding to the described PCIE message in the PCIE NTB address space of described server, according to the ID of the purpose virtual machine in described PCIE message with the purpose virtual machine of described Data dissemination to described network request.

Fourth aspect, the embodiment of the present invention provides a kind of server, it is characterized in that, comprise processor and memory, described memory stores is carried out instruction, when the operation of described server, to communicate by letter between described processor and described memory, described processor is carried out described execution instruction and is made the following method of described server execution:

Virtual machine on described server is tackled network request on the ICP/IP protocol stack, and the data that comprise in described network request are sent to monitor of virtual machine on described server;

Monitor of virtual machine on described server is according to the destination address of described network request, determine the PCIE NTB address space of the destination host that the above server of PCIE switch is corresponding, wherein, described destination host is the main frame under the purpose virtual machine of described network request, and the PCIE NTB address space of the destination host that described the above server of PCIE switch is corresponding is the PCIE NTB address space of distributing on described PCIE switch in the PCIE NTB address space of described destination host corresponding to described server;

The PCIE message that monitor of virtual machine on described server will comprise described data sends to the PCIE NTB address space of destination host corresponding to described the above server of PCIE switch, so as the monitor of virtual machine of described destination host according to the ID of the purpose virtual machine of the described network request in described PCIE message with the purpose virtual machine of described Data dissemination to described network request.

In the possible implementation of the first of fourth aspect, the described execution instruction of described processor execution makes in the method for described server execution, monitor of virtual machine on described server is according to the destination address of described network request, determine the PCIE NTB address space of the destination host that the above server of PCIE switch is corresponding, be specially:

Monitor of virtual machine on described server is according to the destination address of described network request, determine the destination host of described network request, and the PCIE NTB address space information of the described destination host corresponding according to the described server of local cache, determine the PCIE NTB address space of the described destination host that described the above server of PCIE switch is corresponding;

The PCIE NTB address space information of the described destination host that wherein, described server is corresponding is monitor of virtual machine on the described server virtual machine on described server buffer memory when sending arp request and carrying out address resolution.

In conjunction with the possible implementation of the first of fourth aspect, in the possible implementation of the second, the described execution instruction of described processor execution makes in the method for described server execution, and buffer memory when the virtual machine transmission arp request on described server is carried out address resolution comprises:

Described monitor of virtual machine intercepts described virtual machine after the described arp request of the purpose virtual machine transmission of described network request, and the request message of the PCIE NTB address space information of the described destination host that the described virtual machine of initiation inquiry is corresponding is to the subnet management server;

Monitor of virtual machine on described server receives the response message that described subnet management server returns, the PCIE NTB address space information cache of the described destination host that the described virtual machine that comprises in described response message is corresponding is in this locality, the PCIE NTB address space of the described destination host that the PCIE NTB address space of the described destination host that wherein, described virtual machine is corresponding and described server are corresponding is identical.

In conjunction with the possible implementation of the first of fourth aspect or fourth aspect or the possible implementation of the second of fourth aspect, in the third possible implementation, the described execution instruction of described processor execution makes in the method for described server execution, the PCIE message that the monitor of virtual machine of described server will comprise the data in described network request sends to the PCIE NTB address space of destination host corresponding to described the above server of PCIE switch, comprising:

The data that monitor of virtual machine on described server comprises described network request, the ID of described virtual machine and source NTB port id are put into the payload of described PCIE message, with the ID of the purpose virtual machine of the described network request destination address as described PCIE message, send described PCIE message to described PCIE switch, make described PCIE switch described PCIE message be write in the PCIE NTB address space of destination host PCIE NTB address space corresponding to described server;

Wherein, described source NTB port id is the ID of the NTB port that is connected with monitor of virtual machine on described server in described PCIE switch.

In conjunction with the second or the third possible implementation of fourth aspect, in the 4th kind of possible implementation, described processor is carried out described execution instruction and is made the method for described server execution also comprise:

In conjunction with the 4th kind of possible implementation of fourth aspect, in the 5th kind of possible implementation, described processor is carried out described execution instruction and is made the method for described server execution also comprise:

The monitor of virtual machine of described server receives the PCIE NTB address space information of the subnet management server that also the described server of buffer memory is corresponding, in order to can send to described subnet management server the request message of described inquiry destination host PCIE NTB address space.

In conjunction with first to the 5th of fourth aspect or fourth aspect the possible implementation, in the 6th kind of possible implementation, the described execution instruction of described processor execution makes in the method for described server execution, the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine in described PCIE message, comprising:

The 5th aspect, the embodiment of the present invention provide a kind of computer-readable medium, comprise computer executed instructions, and described computer executed instructions is used for making server to carry out following method:

In the possible implementation of the first aspect the 5th, described computer executed instructions is used for making the method for server execution, monitor of virtual machine on described server is according to the destination address of described network request, determine the PCIE NTB address space of the destination host that the above server of PCIE switch is corresponding, be specially:

In conjunction with the possible implementation of the first of the 5th aspect, in the possible implementation of the second, described computer executed instructions is used for making the method for server execution, and buffer memory when the virtual machine transmission arp request on described server is carried out address resolution comprises:

The first or the possible implementation of the second in conjunction with the 5th aspect or the 5th aspect, the method that is used for making the server execution in computer executed instructions described in the third possible implementation, the PCIE message that the monitor of virtual machine of described server will comprise the data in described network request sends to the PCIE NTB address space of destination host corresponding to described the above server of PCIE switch, comprising:

In conjunction with the second or the third possible implementation of the 5th aspect, in the 4th kind of possible implementation, the method that described computer executed instructions is used for server is carried out also comprises:

In conjunction with the 4th kind of possible implementation of the 5th aspect, in the 5th kind of possible implementation, the method that described computer executed instructions is used for server is carried out also comprises:

In conjunction with first to the 5th of the 5th aspect or the 5th aspect the possible implementation, in the 6th kind of possible implementation, described computer executed instructions is used for making the method for server execution, the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine in described PCIE message, comprising:

As mentioned above, the virtual machine communication method based on the PCIE network of the present invention, server and system have following beneficial effect:

The present invention with the PCIE network as the communication mechanism between virtual machine, utilize the high bandwidth of PCIE network, the advantage of low delay, and by the subnet management based on the PCIE network, make virtual machine on the All hosts in network walk around ICP/IP protocol stack in OS, share PCIE non-transparent bridge (the Non-Transparent Bridge of subnet management server maintenance management, be called for short NTB) address space, further improved the communication speed between virtual machine.

Description of drawings

Fig. 1 is the primary structure schematic diagram of the virtual machine communication system based on the PCIE network of the present invention.

Fig. 2 is the structural representation based on main frame in the virtual machine communication system of PCIE network of the present invention.

Fig. 3 is the structural representation of the virtual machine based on moving on main frame in the virtual machine communication system of PCIE network of the present invention.

Fig. 4 is the structural representation of the monitor of virtual machine based on moving on main frame in the virtual machine communication system of PCIE network of the present invention.

Fig. 5 is the structural representation based on PCIE switch in the virtual machine communication system of PCIE network of the present invention.

Fig. 6 is the structural representation based on subnet management server in the virtual machine communication system of PCIE network of the present invention.

Fig. 7 is the schematic flow sheet of the virtual machine communication method based on the PCIE network of the present invention.

Fig. 8 is the detailed process schematic diagram based on step S1 in the virtual machine communication method of PCIE network of the present invention.

Fig. 9 is the detailed process schematic diagram based on address resolution in the virtual machine communication method of PCIE network of the present invention.

Figure 10 is the server architecture schematic diagram of another embodiment of the present invention.

The specific embodiment

Below by specific instantiation explanation embodiments of the present invention, those skilled in the art can understand other advantages of the present invention and effect easily by the disclosed content of this specification.The present invention can also be implemented or be used by the other different specific embodiment, and the every details in this specification also can be based on different viewpoints and application, carries out various modifications or change under spirit of the present invention not deviating from.

The accompanying drawing that partly relates to for this specific embodiment, need to prove, the diagram that provides in the present embodiment only illustrates basic conception of the present invention in a schematic way, then only show in the diagram with the present invention in relevant assembly but not component count, shape and size drafting when implementing according to reality, during its actual enforcement, form, quantity and the ratio of each assembly can be a kind of random change, and its assembly layout form also may be more complicated.

Along with the development of PCIE technology, it is two data communication bus between main frame that the PCIE bus is further expanded.PCIE the transmission of data bag comes addressing by address realm, namely can read and write the address space of distributing to ancillary equipment during processor access ancillary equipment, final PCIE bus by this address the data message route to ancillary equipment.The address space that this routing mechanism requires to distribute to ancillary equipment in a host computer system can not repeat, but can't guarantee that address space does not repeat when coming interaction data by PCIE between two main frames, this moment, the PCIE bus was introduced non-transparent bridge NTB mechanism.NTB mechanism is to utilize the address translation rule that pre-sets that the address that a main frame mails to the data message of another main frame is changed, and is converted to the address realm that can not conflict in destination host, thereby realizes the data route of PCIE bus between main frame.NTB generally is integrated in the middle of the PCIE exchange chip, and the inter-host communication link of high broadband and low delay is provided, and has the function of PCIE equipment, such as interruption and wrong processing etc.In addition, the PCIE switch mainly plays following two aspect effects for the realization of Host2Host (host-to-host) communication: 1) for PIO (Programming Input Output, process channel) read and write identifies (Identity is called for short ID) route and address route by NTB; 2) sending and receiving for DMA (Direct memory access, direct memory access) passes through the ID route.

because be 1 delicate the time delay of PCIE bus, the delay of Ethernet is close to 10 microseconds, ICP/IP protocol is extremely complicated simultaneously, cause the protocol processes of OS to occupy the very most of of network delay, in addition, in data transmission procedure, the data Replica frequently that the ICP/IP protocol stack is processed and interruption are processed and are caused the host CPU resource can't be absorbed in the processing customer service, so the present invention proposes a kind of virtual machine communication method based on the PCIE network, the method has built a PCIE network for the data exchange service of all VM, the VMM of main frame uses address window and the interrupt mechanism of PCIE NTB to communicate, thereby accelerated the communication between VM.

The present invention is described in detail below in conjunction with embodiment and accompanying drawing.

Embodiment

The present embodiment provides a kind of virtual machine communication system based on the PCIE network, as shown in Figure 1, comprises at least 2 main frames 10 (Host), a PCIE switch 20 (PCIE Switch) and a subnet management server 30; On each main frame 10, operation has a monitor of virtual machine (VMM) 111 and at least one virtual machine (VM) 101; PCIE switch 20 is connected with All hosts, realizes the data retransmission between main frame.Described subnet management server 30 is connected with described PCIE switch 20, is used for the corresponding relation of maintenance system main frame and virtual machine.The main frame of the system that provides in the embodiment of the present invention is specifically as follows common computer, mobile terminal, work station or server, private server etc., the subnet management server can be the logical main frame of the Daepori in system, can be a private host, the present invention do concrete restriction yet.

The below is described in further detail main frame, PCIE switch and subnet management server.

As shown in Figure 2, each described main frame 10 comprises: virtual machine operation module 100, virtual machine monitoring module 110 and hardware platform 120.Virtual machine operation module 100 and virtual machine monitoring module 110 provide respectively hardware platform and the software platform of virtual machine, virtual machine monitoring module 110 has comprised virtual machine monitor 111, be the intermediate layer between physical hardware and virtual machine, be responsible for coordinating each virtual machine to the access of hardware platform.A plurality of virtual machines 101 have been moved in virtual machine operation module 100.Monitor of virtual machine 111 on every main frame is connected with all virtual machines 101 communications on this main frame, and the data that virtual machine 101 is sent send to described PCIE switch 20 with the PCIE agreement.

In addition, the hardware platform 120 of main frame can comprise the low-speed devices such as the high speed I/O equipment such as CPU, internal memory, network interface card, storage and InterWorking Equipment and basic input and output.Virtual machine operates on the hardware platform of main frame, and perception is completed by the PCIE network less than the communication of network cluster inside.

Virtual machine 101 is used for tackling network request on the ICP/IP protocol stack, and the data that comprise in described network request is sent to monitor of virtual machine 111.Preferably, virtual machine 101 is used for the described network request of interception on the ICP/IP protocol stack, obtains the data that comprise in described network request, the data that comprise in described network request are put into own corresponding virtual-machine data bag buffer queue, and notice monitor of virtual machine 111.Preferably, virtual machine 101 can only be tackled for the network request of data type message and carry out subsequent treatment, and the network request of non-data type message also carries out subsequent treatment according to original ICP/IP protocol stack mode.

monitor of virtual machine 111, be used for after the described network request that receives virtual machine 101 transmissions, destination address according to described network request, determine the PCIE NTB address space of the destination host that on the PCIE switch, source host is corresponding, the PCIE message that will comprise the data in described network request sends to the PCIE NTB address space of described destination host corresponding to described the above source host of PCIE switch, so that the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine in described PCIE message.Wherein, described destination host is the main frame under the purpose virtual machine of described network request, and the PCIE NTB address space of the destination host that described the above source host of PCIE switch is corresponding is the PCIE NTB address space of distributing on described PCIE switch in the PCIE NTB address space of described destination host corresponding to described source host.

Preferably, VMM111, after being used for receiving the network request of virtual machine 101 transmissions, according to the destination address in network request (being the IP address of the purpose virtual machine of network request), determine the PCIE NTB address space of the destination host that source host is corresponding, when determining that namely this source host is communicated by letter (being that source VM on source host and the purpose VM on destination host communicate) herein with destination host, the PCIE NTB address space of the destination host that need to write.For convenience of description, in this article, the main frame that sends the virtual machine place of this network request is called source host, the main frame at the purpose VM place of this network request is called destination host.

Preferably, when being cached with the PCIE NTB address space of the corresponding destination host of this network request (being the corresponding physical host in IP address of the purpose virtual machine of network request) on VMM111, VMM111 can determine the PCIE NTB address space that destination host corresponding to described network request is corresponding, i.e. the PCIE NTB address space of this source host destination host that should write.Preferably, send address resolution protocol (Address Resolution Protocol is called for short ARP) when the address resolution flow process is carried out in request at virtual machine 101, the PCIE NTB address space of the corresponding destination host of VMM111 meeting this source host of buffer memory.Further, when virtual machine 101 will send the TCP/IP network request, if there is no the mac address information of the purpose virtual machine of described network request on virtual machine 101, virtual machine 111 can send ARP according to ICP/IP protocol and ask to carry out address resolution.Be specially:

Virtual machine 101 when also being used for the mac address information when the purpose virtual machine that there is no network request on virtual machine 101, sending Ethernet ARP to the purpose virtual machine of network request and asks.The IP address that comprises the purpose virtual machine in the ARP request.; VMM111; after also being used for receiving the ARP request of described virtual machine 101 transmissions; initiate the request message of PCIE NTB address space corresponding to inquiry purpose virtual machine to subnet management server 30; this request message is with the IP address of purpose virtual machine; and after receiving the response message of the PCIE NTB address space information that comprises destination host corresponding to described source VM that described subnet management server 30 returns, the PCIE NTB address space information cache of the destination host that described source VM is corresponding is in this locality.

Subnet management server 30, after request message for PCIE NTB address space corresponding to the inquiry purpose virtual machine that receives the VMM initiation, determine destination host under described purpose virtual machine according to the corresponding relation of main frame and virtual machine, know the PCIE NTB address space of the destination host that source host is corresponding, send response message to monitor of virtual machine 111.

preferably, subnet management server 30, be used for all PCIE NTB address spaces of maintenance management, PCIE NTB address space is configured in described source host and destination host and communicates by letter on the PCIE switch that is connected, described PCIE NTB address space is divided into the subspace, PCIE NTB address of the main frame number equal parts that is connected with the PCIE switch, the corresponding main frame in each subspace, described address, be used for receiving the data that send to this main frame, be between address microvoid of each other host assignment in subspace, address corresponding to each main frame, corresponding other main frames between each described address microvoid, be used for receiving and come from the data that these other main frames send to this main frame corresponding to subspace, described address, here said other main frames are except main frame corresponding to subspace, described address, the main frame that is connected with the PCIE switch.For example, PCIE switch maximum can access N main frame (main frame 1-main frame N), the PCIE NTB address space of subnet management server maintenance also will be divided into subspace, N PCIE NTB address so, and subspace, PCIE NTB address correspondence a main frame.further, subspace, a PCIE NTB address can be told again between N-1 PCIE NTB address microvoid, corresponding other main frames between the microvoid of each PCIE NTB address, the mode of the PCIE NTB address space by writing appointment between main frame and main frame just can realize mutual communication, for example, whole PCIE NTB address space is divided into subspace, N PCIE NTB address, the subspace, PCIE NTB address of main frame 1 correspondence is divided between N-1 address microvoid, be designated respectively H2, H3, ... HN, when main frame 2 and main frame 1 communicate, main frame 2 namely can be completed data by the transmission of main frame 2 to main frame 1 to regional (between the microvoid of the address) data writing of the H2 in the PCIE NTB address space (subspace, address) of main frame 1 correspondence.Subnet management server 30 meetings corresponding address space of all Host in the informing network cluster oneself when the PCIE netinit.The embodiment of the present invention can realize by this mechanism of address space a plurality of main frames and concurrent the communicating of main frame

Further, subnet management server 30 also is used for safeguarding each main frame and the corresponding relation of the virtual machine that moves on it, and according to this corresponding relations of real-time update such as the establishment of virtual machine, change, destructions.The corresponding relation of described subnet management server maintenance can be the mapping relations of IP address or the mapping relations of virtual machine ID, the perhaps mapping relations of other types, for example, can be the corresponding relation between the ID of the IP address of main frame and virtual machine, or the corresponding relation between the ID of PCIE NTB address space corresponding to main frame and virtual machine.The establishment of each VM, destruction, change can notify the subnet management server to refresh its mapping relations.Described subnet management server can be a physical host, can be also a virtual machine.

Concrete, subnet management server 30, be used for receiving the request message that source VMM sends, IP address according to the purpose virtual machine that carries in message, and the corresponding relation of the destination host of local maintenance preservation and purpose virtual machine is determined the destination host that described purpose virtual machine is affiliated, and the PCIE NTB address space of the destination host that further definite source VM is corresponding, it is the PCIE NTB address space of purpose VM corresponding to source VM, send response message to the source virtual machine watch-dog, the PCIE NTB address space information of the destination host that source VM is corresponding returns to source VMM.

; described monitor of virtual machine 111 is after obtaining the PCIE NTB address space message of destination host corresponding to described source VM; also be used for proceeding the ARP flow process of source VM; follow-up ARP flow process and existing ARP flow process are similar: monitor of virtual machine sends to the ARP request of Ethernet the purpose virtual machine of described network request; namely; source VMM sends to ARP request the VMM of the main frame at purpose virtual machine place; after monitor of virtual machine on described destination host is received described ARP request, described ARP request is sent to described purpose virtual machine; The purpose virtual machine returns to arp response, wherein comprises the MAC Address of purpose virtual machine; Described source virtual machine receives the arp response that described purpose virtual machine sends, and has so far completed the ARP address resolution procedure of purpose virtual machine.

preferably, monitor of virtual machine 111, be used for sending to the PCIE NTB address space of destination host corresponding to the PCIE NTB address space of described source host described network request, be specially: monitor of virtual machine 111, for the data that described network request is comprised, source virtual machine ID and source NTB port id are put into the payload of PCIE message, with the ID of the purpose virtual machine of the described network request destination address as described PCIE message, send described PCIE message to PCIE switch 20, make described PCIE switch 20 described data be write in the PCIE NTB address space of destination host PCIE NTB address space corresponding to described source host.Wherein, described source virtual machine ID is the ID that sends the virtual machine of described data, and described source NTB port id is the ID of the NTB port that is connected with monitor of virtual machine on described source host in described PCIE switch.

Further, the monitor of virtual machine of destination host also is used for according to the ID of purpose virtual machine, described Data dissemination being arrived the purpose virtual machine.Particularly, the monitor of virtual machine of destination host reads in PCIE NTB address space corresponding to described destination host corresponding to the data in the PCIE NTB address space of described source host, according to the ID of the purpose virtual machine of described data with Data dissemination in the buffer queue of the purpose virtual machine of described data, and notify purpose virtual machine receive data from described buffer queue of described data.

Further, as shown in Figure 3, described virtual machine 101 comprises: network request blocking module 1011, core protocol module 1012, virtual unit driver module 1013;

Described network request blocking module 1011 is used for the described network request of interception on the ICP/IP protocol stack.

Described core protocol module 1012 is connected with described network request blocking module 1011 communications, is used for the described network request of described network blocking module 1011 interceptions is processed.Concrete processing procedure is: core protocol module 1012 receives described network request, resolves described network request, obtains the data that described network request comprises.

Described virtual unit driver module 1013, be connected with described core protocol module 1012 communications, be used for the data that described network request comprises are put into packet buffering formation corresponding to described virtual machine, and notice and described virtual machine operate in same the monitor of virtual machine on main frame.Particularly, the data that virtual unit driver module 1013 in virtual machine comprises described network request are put into by the monitor of virtual machine supervision and are belonged to the privately owned packet buffering formation (hereinafter to be referred as buffer queue) of described virtual machine, notify simultaneously monitor of virtual machine to have new request to send.Wherein, each virtual machine is a privately owned buffer queue all, owing to can being provided with a plurality of virtual machines on each main frame, therefore the monitor of virtual machine on each main frame is supervised the privately owned buffer queue of whole virtual machines on this main frame.

Further, as shown in Figure 4, described monitor of virtual machine 111 comprises the PCIE packet delivery device 1111, the PCIE communication module 1112 that are connected with virtual machine 101 communications.

Described PCIE packet delivery device 1111, be used for the data that described network request comprises are added that source virtual machine ID and source NTB port id put into the payload of PCIE message, with the ID of the purpose virtual machine of the described network request destination address as described PCIE message.Wherein, the virtual machine of interception network request is called source virtual machine, the virtual machine that receives network request is the purpose virtual machine, main frame under source virtual machine is source host, main frame under the purpose virtual machine is destination host, described source virtual machine ID is the described ID that sends the virtual machine of data, and described source NTB port id is the ID of the NTB port that is connected with monitor of virtual machine on described source host.Owing to may move many VM on a main frame, so the function of PCIE packet delivery device 1111 is that network request with certain VM adds source virtual machine ID and source NTB port id, distinguishes the data flow of each VM.

Described PCIE communication module 1112, be used for the described PCIE message that PCIE packet delivery device 1111 generates is sent to described PCIE switch 20, make described PCIE switch 20 that described data are write in PCIE NTB address space corresponding to destination host PCIE NTB address space corresponding to described source host.

PCIE communication module 1112 on monitor of virtual machine 111 on destination host, also be used for receiving the PCIE message from described PCIE switch 20, described PCIE packet delivery device 1111, also be used for PCIE message that PCIE communication module 1112 is received according to the purpose virtual machine ID of message with the Data dissemination in the PCIE message in the buffer queue of the corresponding purpose virtual machine of message.

As shown in Figure 5, described PCIE switch 20 mainly comprises the non-transparent bridge NTB that is connected with main frame Host, and NTB provides address window mechanism to be used for transceiving data between main frame.In the virtual machine communication system based on the PCIE network of the present invention, described PCIE switch 20 comprises at least 2 NTB ports, and the NTB port is corresponding connected one by one with main frame and subnet management server.For example in Fig. 5, main frame 1 is write the address window at the NTB place of main frame 2, and data can be sent to the NTB of main frame 2 with the form of data message by PCIE switch 20 from the NTB of main frame 1 so, finally send in the internal memory of main frame 2; Main frame 1 also can be write the doorbell register (doorbell) of main frame 2 correspondences, makes the NTB that main frame 2 connects send interruption to main frame 2; At last, PCIE switch 20 also has more senior long-range immediate data access (Remote direct memory access is called for short RDMA) function, utilizes this function, copies internal memory between the main frame that PCIE switch 20 can be realized utilizing NTB to be connected.

As shown in Figure 6, described subnet management server 30 comprises main frame and virtual machine corresponding relation administration module 301, PCIE NTB address space menagement module 302, PCIE communication module 303.

Main frame and virtual machine corresponding relation administration module 301 are for the corresponding relation of maintenance system main frame and virtual machine.Particularly, described main frame and virtual machine corresponding relation administration module 301 have been safeguarded each Host and the corresponding relation that moves VM thereon in the current system.The corresponding relation that described main frame and virtual machine corresponding relation administration module 301 are safeguarded can be IP mapping relations or ID mapping relations, the perhaps mapping relations of other types, the corresponding relation between the PCIE NTB address space that for example attaching relation between the ID of the IP address of main frame and virtual machine or main frame are corresponding and the ID of virtual machine.Main frame and virtual machine corresponding relation administration module 301 can be according to the corresponding relation of the real-time update main frame such as the establishment of virtual machine, change, destruction and virtual machine.

Described PCIE NTB address space menagement module 302, being used for maintenance management is the pre-configured PCIE NTB address space of each main frame.all PCIE NTB address spaces are divided into the subspace, PCIE NTB address of the umber identical with the main frame number of described PCIE switch 20 connections, the corresponding main frame in each subspace, described address, be used for receiving the data that send to main frame corresponding to subspace, described address, further, be between address microvoid of other host assignment in each subspace, described address, corresponding described other main frames between each described address microvoid, be used for receiving and come from the data that described other main frames send to main frame corresponding to subspace, described address, wherein, described other main frames are except main frame corresponding to subspace, described address, the main frame that is connected with described PCIE switch.For example, PCIE switch maximum can access N main frame, main frame 1-main frame N, and the PCIE NTB address space of subnet management server maintenance also will be divided into subspace, N PCIE NTB address so, and subspace, PCIE NTB address correspondence a main frame.further, subspace, a PCIE NTB address can be told again between N-1 PCIE NTB address microvoid, corresponding other main frames between the microvoid of each PCIE NTB address, the mode of the PCIE NTB address space by writing appointment between main frame and main frame just can realize mutual communication, for example, whole PCIE NTB address space is divided into subspace, N PCIE NTB address, the subspace, PCIE NTB address of main frame 1 correspondence is divided between N-1 address microvoid, be designated respectively H2, H3......HN, when main frame 2 and main frame 1 communicate, main frame 2 namely can be completed data by the transmission of main frame 2 to main frame 1 to regional (between the microvoid of the address) data writing of the H2 in the PCIE NTB address space (subspace, address) of main frame 1 correspondence.PCIE NTB address space menagement module 302, meeting all Host subnet management server 30 corresponding address spaces in the informing network cluster when the PCIE netinit, the address space information of the corresponding subnet management server of VMM meeting this main frame of buffer memory on each main frame is so that VMM can correctly send the request message of the PCIE NTB address space of destination host corresponding to query source VM to the subnet management server when the address resolution request of receiving source VM.

Described PCIE communication module 303 is used for communicating by letter with described PCIE switch 20, transmitting-receiving PCIE message.

Further, PCIE communication module 303 is used for the request message that reception sources VMM sends by the PCIE switch; Described main frame and virtual machine corresponding relation administration module 301, for the IP address of the purpose virtual machine that carries according to this request message, and the corresponding relation of the destination host of maintenance preservation and purpose virtual machine is determined the destination host that described purpose virtual machine is affiliated; PCIE NTB address space menagement module 302 is used for the information according to described destination host and source host, determines the PCIE NTB address space of the destination host that source VM is corresponding, the PCIE NTB address space of the purpose VM that namely source VM is corresponding; Send response message to the source virtual machine watch-dog by described PCIE communication module 303, the PCIE NTB address space information of the destination host that source VM is corresponding returns to source VMM.

The present invention as the communication mechanism between virtual machine, has skipped the transmission of Ethernet with the PCIE network, efficiently utilizes the high bandwidth of PCIE network, the advantage of low delay, decrease propagation delay time, realized the high-speed traffic between virtual machine.And, the present invention has also broken through the restriction of PCIE network, realized the subnet management based on the PCIE network, make virtual machine on the All hosts in network walk around ICP/IP protocol stack in OS, safeguarded the corresponding relation of main frame and virtual machine in the system, realize the unified management maintenance of whole system, further improved the integrality based on the virtual machine communication system of PCIE network.In addition, the present invention has realized that by the subnet management server virtual machine to the sharing of PCIE NTB address space, has solved the concurrent reading and writing problem of PCIE network, has further improved the speed of data receiver between virtual machine.

The present embodiment also provides a kind of virtual machine communication method based on the PCIE network, and the method can be realized by described virtual machine communication system based on the PCIE network, but its implementation structure is not limited to the described system of embodiment.

As shown in Figure 7, described virtual machine communication method based on the PCIE network comprises:

S1, the source virtual machine on source host is tackled network request on the ICP/IP protocol stack, and the data that comprise in described network request are sent to monitor of virtual machine on described source host.

Particularly, can be as shown in Figure 8, described step S1 comprises:

S11, source virtual machine is tackled network request on the ICP/IP protocol stack;

Preferably, can only tackle for the network request of data type message and carry out subsequent treatment in the embodiment of the present invention, the network request of non-data type message also carries out subsequent treatment according to original ICP/IP protocol stack mode.Because the network request of non-data type message is little on the impact of host performance, and realize that logic is complicated, adopt the method in the embodiment of the present invention to there is no clear superiority, namely, preferably, the embodiment of the present invention will be controlled message and still all hand to existing ICP/IP protocol stack and carry out subsequent treatment, and the data interception type message carries out the subsequent treatment in the embodiment of the present invention.

S12, described source virtual machine is resolved described network request, obtains the data that described network request comprises;

S13, described source virtual machine is put into packet buffering formation corresponding to described source virtual machine with described data, and notifies the monitor of virtual machine on described source host.

S2, monitor of virtual machine on described source host is after receiving described network request, destination address according to described network request, determine the PCIE NTB address space of the destination host that the above source host of PCIE switch is corresponding, wherein, the PCIE NTB address space of destination host is the PCIE NTB address space of distributing to described destination host on the PCIE switch, the PCIE NTB address space of the destination host that the above source host of PCIE switch is corresponding is the PCIE NTB address space of distributing on described PCIE switch in the PCIE NTB address space of described destination host corresponding to described source host.After VMM on source host receives described network request, determine destination host corresponding to described network request (physical host under the purpose virtual machine) according to the destination address in network request (being the IP address of the purpose virtual machine of network request), and the PCIE NTB address space of the destination host that further definite described source host is corresponding, when determining that namely this source host is communicated by letter (being also that source VM on source host and the purpose VM on destination host communicate) herein with destination host, the PCIE NTB address space of the destination host that need to write.PCIE NTB address space is configured in described source host and destination host and communicates by letter on a PCIE switch that is connected.

Preferably, when being cached with the PCIE NTB address space of the corresponding destination host of this network request (being the corresponding physical host in IP address of the purpose virtual machine of network request) on the VMM on source host, the VMM on source host can determine the PCIE NTB address space that destination host corresponding to described network request is corresponding, i.e. the PCIE NTB address space of this source host correspondence destination host that should write.When source virtual machine transmission arp request is carried out the address resolution flow process, the PCIE NTB address space of the corresponding destination host of VMM meeting this source host of buffer memory on source host, described destination host is exactly the corresponding physical host in IP address of the purpose virtual machine of this network request.Concrete, when source virtual machine will send the TCP/IP network request, if there is no the mac address information of the purpose virtual machine of described network request on source virtual machine, source virtual machine can send ARP according to ICP/IP protocol and ask to carry out address resolution, can be referring to Fig. 9, wherein source VMM refers to the monitor of virtual machine on source host, the monitor of virtual machine on the main frame of purpose VMM feeling the pulse with the finger-tip.The specific implementation process of address resolution is:

S21, source virtual machine is by the purpose virtual machine transmission Ethernet ARP request of the monitor of virtual machine on source host to network request, and the ARP request here is the standard request in ICP/IP protocol, and the inside can comprise the IP address of purpose virtual machine.

S22, source VMM initiate the request message of PCIE NTB address space corresponding to inquiry purpose virtual machine to the subnet management server after receiving ARP request with purpose virtual machine IP address, and this request message is with the IP address of purpose virtual machine.Preserve the PCIE NTB address space information of subnet management server on the VMM of source, send the request message of the PCIE NTB address space of destination host corresponding to query source VM.

S23, described subnet management server is determined destination host under described purpose virtual machine according to the corresponding relation of the main frame of its maintenance management and virtual machine, knows the PCIE NTB address space of the destination host that source host is corresponding.

preferably, the subnet management server maintenance is being managed all PCIE NTB address spaces, all PCIE NTB address spaces are divided into the subspace, PCIE NTB address of the main frame number equal parts that is connected with the PCIE switch, the corresponding main frame in each subspace, described address, be used for receiving the data that send to this main frame, be between address microvoid of each other host assignment in subspace, address corresponding to each main frame, corresponding other main frames between each described address microvoid, be used for receiving and come from the data that these other main frames send to this main frame corresponding to subspace, described address, here said other main frames are except main frame corresponding to subspace, described address, the main frame that is connected with the PCIE switch.

For example, PCIE switch maximum can access N main frame (main frame 1-main frame N), the PCIE NTB address space of subnet management server maintenance also will be divided into subspace, N PCIE NTB address so, and subspace, PCIE NTB address correspondence a main frame.further, subspace, a PCIE NTB address can be told again between N-1 PCIE NTB address microvoid, corresponding other main frames between the microvoid of each PCIE NTB address, the mode of the PCIE NTB address space by writing appointment between main frame and main frame just can realize mutual communication, for example, whole PCIE NTB address space is divided into subspace, N PCIE NTB address, the subspace, PCIE NTB address of main frame 1 correspondence is divided between N-1 address microvoid, be designated respectively H2, H3, ... HN, when main frame 2 and main frame 1 communicate, main frame 2 namely can be completed data by the transmission of main frame 2 to main frame 1 to regional (between the microvoid of the address) data writing of the H2 in the PCIE NTB address space (subspace, address) of main frame 1 correspondence.The embodiment of the present invention can realize by this mechanism of address space a plurality of main frames and concurrent the communicating of main frame.

Preferably, the subnet management server is also safeguarded each main frame and the corresponding relation of virtual machine of operation on it, and according to this corresponding relations of real-time update such as the establishment of virtual machine, change, destructions.The corresponding relation of described subnet management server maintenance can be the mapping relations of IP address or the mapping relations of virtual machine ID, the perhaps mapping relations of other types, for example, can be the corresponding relation between the ID of the IP address of main frame and virtual machine, or the corresponding relation between the ID of PCIE NTB address space corresponding to main frame and virtual machine.

Concrete, the subnet management server receives the request message that source VMM sends, the subnet management server is according to the IP address of the purpose virtual machine that carries in message, and the corresponding relation of the destination host of local maintenance preservation and purpose virtual machine is determined the destination host that described purpose virtual machine is affiliated, and the PCIE NTB address space of the destination host that further definite source VM is corresponding, the PCIE NTB address space of the purpose VM that namely source VM is corresponding.

S24, subnet management server send response message to the source virtual machine watch-dog, and the PCIE NTB address space information of the destination host that source VM is corresponding returns to source VMM.

After S25, the monitor of virtual machine on described source host obtained the PCIE NTB address space information of destination host corresponding to described source VM, the PCIE NTB address space information cache of the destination host that described source VM is corresponding was in this locality.Namely buffer memory the PCIE NTB address space information of destination host corresponding to source host.

After monitor of virtual machine on described source host obtains the PCIE NTB address space message of destination host corresponding to described source VM, proceed the ARP flow process of source VM, follow-up ARP flow process and existing ARP flow process are similar:

S26, monitor of virtual machine on described source host sends to the ARP request of Ethernet the purpose virtual machine of described network request, namely, source VMM sends to ARP request the VMM of the main frame at purpose virtual machine place, after monitor of virtual machine on described destination host is received described ARP request, described ARP request is sent to described purpose virtual machine;

S27, the purpose virtual machine returns to arp response, and the described source virtual machine of MAC Address that wherein comprises the purpose virtual machine receives the arp response that described purpose virtual machine sends, and has so far completed the ARP address resolution procedure of purpose virtual machine.

Preferably, the subnet management server can be when the PCIE netinit pre-configured good PCIE NTB address space, and with the All hosts in the corresponding address space information notice of subnet management server subnet, after each main frame is received the information of subnet management server, the address space information of the corresponding subnet management server of meeting buffer memory oneself is in order to can correctly send the request message of PCIE NTB address space corresponding to inquiry to the subnet management server.

S3, the PCIE message that monitor of virtual machine on described source host will comprise the data in described network request sends to the PCIE NTB address space of destination host corresponding to described the above source host of PCIE switch, so as the monitor of virtual machine of described destination host according to the ID of the purpose virtual machine in described PCIE message with the purpose virtual machine of described Data dissemination to this network request.

particularly, monitor of virtual machine on described source host sends to described network request in the PCIE NTB address space of destination host the PCIE NTB address space corresponding to described source host, implementation procedure is in detail: the data that the monitor of virtual machine on described source host comprises described network request, the ID of described source virtual machine and source NTB port id are put into the payload of PCIE message, with the ID of the purpose virtual machine of the described network request destination address as described PCIE message, send described PCIE message to described PCIE switch, make described PCIE switch described data be write in the PCIE NTB address space of destination host PCIE NTB address space corresponding to described source host.Wherein, described source NTB port id is the ID of the NTB port that is connected with monitor of virtual machine on described source host in described PCIE switch.

Further, described virtual machine communication method based on the PCIE network also comprises step S4, and the monitor of virtual machine of described destination host arrives the purpose virtual machine according to the ID of described purpose virtual machine with described Data dissemination.

Particularly, the detailed implementation procedure of this step is: the monitor of virtual machine of described destination host reads in PCIE NTB address space corresponding to described destination host corresponding to the data in the PCIE NTB address space of described source host, according to the ID of the purpose virtual machine of described data with Data dissemination in the buffer queue of the purpose virtual machine of described data, and notify purpose virtual machine receive data from described buffer queue of described data.Namely, have no progeny in receiving doorbell, the VMM of destination host can resolve rapidly the data of buffering area, and according to zone (domain) ID of target VM with Data dissemination in the buffer queue of target VM, target VMM can inject interrupt notification target VM simultaneously has new data to arrive.Target VM finally can receive described data by the virtual unit driver module in its kernel from buffer queue.Each virtual machine is a privately owned buffer queue all, owing to can being provided with a plurality of virtual machines on each main frame, therefore the monitor of virtual machine in each main frame is supervised the privately owned buffer queue of whole virtual machines on this main frame.

The corresponding relation of main frame and virtual machine in the embodiment of the present invention has been utilized subnet management server maintenance PCIE network, realized the address resolution function of target VM, further realize the unified management maintenance of whole PCIE network, improved the integration of PCIE network.

Figure 10 is the server architecture figure of the embodiment of the present invention, and as shown in the figure, the server 40 of the present embodiment comprises at least one processor 401, memory 405, at least one NIC 404, and at least one communication bus 402.This server 40 optionally comprises user interface 403, comprises display, keyboard or pointing device.Instruction is carried out in memory 405 storages, when server 40 operation, communicate by letter between processor 401 and memory 405, processor 401 execution instructions make the method for the host implementation described in server 40 execution front embodiment of the present invention, be specially: can be as shown in Figure 7, source virtual machine on server is tackled network request on the ICP/IP protocol stack, and the data that comprise in described network request are sent to monitor of virtual machine on server; Monitor of virtual machine on server according to the destination address of described network request, is determined the PCIE NTB address space of the destination host that the above server of PCIE switch is corresponding after receiving described network request; The PCIE message that monitor of virtual machine on server will comprise the data in described network request sends to the PCIE NTB address space of destination host corresponding to described the above server of PCIE switch, so as the monitor of virtual machine of described destination host according to the ID of the purpose virtual machine in described PCIE message with the purpose virtual machine of described Data dissemination to this network request.Realize that the method described in principle and technique effect and front embodiment is similar, repeat no more herein.Operating system 406 comprises various programs, is used for realizing various basic businesses and processing hardware based task.This server 40 also comprises the display interface card.In addition, a kind of computer-readable medium also is provided in the embodiment of the present invention, comprises computer executed instructions, computer executed instructions can make the method for the host implementation described in server 40 execution front embodiment of the present invention, its realize principle and technique effect similar, repeat no more herein.

Further, server 40 in the present embodiment, can also substitute the main frame 10 in the virtual machine communication system of PCIE network in the embodiment of front, realize the function of main frame 10 in above-described embodiment, as the part in the virtual machine communication system that forms the PCIE network, the cooperation of communicating by letter with other parts in the virtual machine communication system of PCIE network, make the virtual machine communication system of described PCIE network realize the PCIE network realizing the high-speed traffic between virtual machine as the communication mechanism between virtual machine.It realizes that communication system and the communication means described in principle and technique effect and front embodiment is similar, repeats no more herein.

The present invention has evaded time delay and host performance loss that the ICP/IP protocol stack causes, has utilized the low delay of PCIE network, the advantage of high bandwidth, decrease propagation delay time, realized the high-speed traffic between virtual machine; Also measured PCIE realizes simultaneously, has solved the concurrent reading and writing problem of PCIE network, has further improved the speed of data receiver between virtual machine.

In sum, the present invention has effectively overcome various shortcoming of the prior art and the tool high industrial utilization.

Through the above description of the embodiments, the those skilled in the art can be well understood to the present invention and can realize with hardware, or the firmware realization, or their combination realizes.When using software to realize, above-mentioned functions can be stored in computer-readable medium or as one or more instructions or code on computer-readable medium and transmit.Computer-readable medium comprises computer-readable storage medium and communication media, and wherein communication media comprises any medium of being convenient to transmit from a place to another place computer program.Storage medium can be any usable medium that computer can access.As example but be not limited to: computer-readable medium can comprise RAM, ROM, EEPROM, CD-ROM or other optical disc storage, magnetic disk storage medium or other magnetic storage apparatus or can be used in the program code that carries or store the expectation with instruction or data structure form and can be by any other medium of computer access.In addition.Any connection can be suitable become computer-readable medium.For example, if software be use coaxial cable, optical fiber cable, twisted-pair feeder, Digital Subscriber Line (DSL) or the wireless technology such as infrared ray, radio and microwave from the website, server or the transmission of other remote source, under coaxial cable, optical fiber cable, twisted-pair feeder, DSL or the wireless technology such as infrared ray, wireless and microwave are included in so in the photographic fixing of medium.As used in the present invention, dish (Disk) and dish (disc) comprise compression laser disc (CD), laser dish, laser disc, digital universal laser disc (DVD), floppy disk and Blu-ray Disc, the copy data of the common magnetic of its mid-game, dish come the copy data of optics with laser.Within top combination also should be included in the protection domain of computer-readable medium.

Above-described embodiment is illustrative principle of the present invention and effect thereof only, but not is used for restriction the present invention.Any person skilled in the art scholar all can under spirit of the present invention and category, modify or change above-described embodiment.Therefore, have in technical field under such as and know that usually the knowledgeable modifies or changes not breaking away from all equivalences of completing under disclosed spirit and technological thought, must be contained by claim of the present invention.

Claims

1. the virtual machine communication method based on high-speed peripheral assembly interconnect PCIE network, is characterized in that, comprising:

2. method according to claim 1, is characterized in that, the monitor of virtual machine on described source host is determined the PCIE NTB address space of the destination host that the above source host of PCIE switch is corresponding according to the destination address of described network request, is specially:

3. method according to claim 2, is characterized in that, buffer memory when the source virtual machine transmission arp request on described source host is carried out address resolution comprises:

4. the described method of according to claim 1 to 3 any one, it is characterized in that, the PCIE message that monitor of virtual machine on described source host will comprise the data in described network request sends to the PCIE NTB address space of destination host corresponding to described the above source host of PCIE switch, comprising:

The data that monitor of virtual machine on described source host comprises described network request, the ID of described source virtual machine and source NTB port id are put into the payload of described PCIE message, with the ID of the purpose virtual machine of the described network request destination address as described PCIE message, send described PCIE message to described PCIE switch, make described PCIE switch described PCIE message be write in the PCIENTB address space of destination host PCIE NTB address space corresponding to described source host;

5. according to claim 3 or 4 described methods, is characterized in that, also comprises:

6. method according to claim 5, is characterized in that, also comprises:

7. the described method of according to claim 3 to 6 any one, is characterized in that, also comprises:

8. the described method of according to claim 1 to 7 any one, it is characterized in that, the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine of the described network request in described PCIE message, comprising:

9. virtual machine communication system based on high-speed peripheral assembly interconnect network comprises: at least two main frames, a PCIE switch, and described at least two main frames are connected by described PCIE switch communication;

10. system according to claim 9, is characterized in that, described each main frame comprises:

Virtual machine operation module is used for moving described virtual machine;

11. system according to claim 9 is characterized in that, described each main frame comprises: at least one processor, memory;

Described memory is used for storage and carries out instruction;

12. according to claim 9 to 11 arbitrary described system, it is characterized in that, the monitor of virtual machine of described source host is according to the destination address of the described network request of described source virtual machine transmission, determine the PCIE NTB address space of the destination host that described the above source host of PCIE switch is corresponding, be specially:

13. system according to claim 12 is characterized in that, also comprises the subnet management server;

14. the described system of according to claim 9 to 13 any one, it is characterized in that, the PCIE message that monitor of virtual machine on described source host will comprise the data in described network request sends to the PCIE NTB address space of described destination host corresponding to described the above source host of PCIE switch, comprising:

the data that monitor of virtual machine on described source host comprises described network request, the ID of described source virtual machine and source NTB port id are put into the payload of described PCIE message, with the ID of the purpose virtual machine of the described network request destination address as described PCIE message, send described PCIE message to described PCIE switch, make described PCIE switch described PCIE message be write in the PCIENTB address space of destination host PCIE NTB address space corresponding to described source host, wherein, described source NTB port id is the ID of the NTB port that is connected with monitor of virtual machine on described source host in described PCIE switch,

15. according to claim 13 or 14 described systems is characterized in that:

16. system according to claim 15 is characterized in that:

17. according to claim 13 to 16 any one described systems, it is characterized in that:

18. the described system of according to claim 9 to 17 any one, it is characterized in that, the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine of the described network request in described PCIE message, comprising:

19. a server comprises:

Virtual machine operation module is used for the operation virtual machine;

20. server according to claim 19, it is characterized in that, described monitor of virtual machine is used for the destination address according to the described network request of described virtual machine transmission, determines the PCIE NTB address space of the destination host that the above server of PCIE switch is corresponding, is specially:

21. server according to claim 20 is characterized in that, buffer memory when described monitor of virtual machine carries out address resolution in described virtual machine transmission arp request is specially:

22. according to claim 19 to the 21 described servers of any one, it is characterized in that, the PCIE message that monitor of virtual machine on described source host will comprise the data in described network request sends to the PCIE NTB address space of described destination host corresponding to described the above server of PCIE switch, comprising:

23. according to claim 21 or 22 described servers is characterized in that:

24. server according to claim 23 is characterized in that:

25. according to claim 19 to the 24 described servers of any one, it is characterized in that:

The corresponding relation of described server and described virtual machine is by described subnet management server maintenance, and according to establishment, change, the described corresponding relation of destruction real-time update of described virtual machine.

26. according to claim 19 to the 25 described servers of any one, it is characterized in that, the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine in described PCIE message, comprising:

27. a server is characterized in that, comprises processor and memory, described memory stores is carried out instruction, when the operation of described server, to communicate by letter between described processor and described memory, described processor is carried out described execution instruction and is made the following method of described server execution:

28. server according to claim 27, it is characterized in that, the described execution instruction of described processor execution makes in the method for described server execution, monitor of virtual machine on described server is according to the destination address of described network request, determine the PCIE NTB address space of the destination host that the above server of PCIE switch is corresponding, be specially:

29. server according to claim 28 is characterized in that, the described execution instruction of described processor execution makes in the method for described server execution, and buffer memory when the virtual machine transmission arp request on described server is carried out address resolution comprises:

Monitor of virtual machine on described server receives the response message that described subnet management server returns, the PCIE NTB address space information cache of the described destination host that the described virtual machine that comprises in described response message is corresponding is in this locality, the PCIE NTB address space of the described destination host that the PCIENTB address space of the described destination host that wherein, described virtual machine is corresponding and described server are corresponding is identical.

30. the described server of according to claim 27 to 29 any one, it is characterized in that, the described execution instruction of described processor execution makes in the method for described server execution, the PCIE message that the monitor of virtual machine of described server will comprise the data in described network request sends to the PCIE NTB address space of destination host corresponding to described the above server of PCIE switch, comprising:

The data that monitor of virtual machine on described server comprises described network request, the ID of described virtual machine and source NTB port id are put into the payload of described PCIE message, with the ID of the purpose virtual machine of the described network request destination address as described PCIE message, send described PCIE message to described PCIE switch, make described PCIE switch described PCIE message be write in the PCIENTB address space of destination host PCIE NTB address space corresponding to described server;

31. according to claim 29 or 30 described servers is characterized in that, described processor is carried out described execution instruction and is made the method for described server execution also comprise:

32. server according to claim 31 is characterized in that, described processor is carried out described execution instruction and is made the method for described server execution also comprise:

33. the described server of according to claim 29 to 32 any one is characterized in that, described processor is carried out described execution instruction and is made the method for described server execution also comprise:

34. the described server of according to claim 27 to 33 any one, it is characterized in that, the described execution instruction of described processor execution makes in the method for described server execution, the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine in described PCIE message, comprising:

35. a computer-readable medium comprises computer executed instructions, described computer executed instructions is used for making server to carry out following method:

36. computer-readable medium according to claim 35, it is characterized in that, described computer executed instructions is used for making the method for server execution, monitor of virtual machine on described server is according to the destination address of described network request, determine the PCIE NTB address space of the destination host that the above server of PCIE switch is corresponding, be specially:

37. computer-readable medium according to claim 36 is characterized in that, described computer executed instructions is used for making the method for server execution, and buffer memory when the virtual machine transmission arp request on described server is carried out address resolution comprises:

38. the described computer-readable medium of according to claim 35 to 37 any one, it is characterized in that, described computer executed instructions is used for making the method for server execution, the PCIE message that the monitor of virtual machine of described server will comprise the data in described network request sends to the PCIE NTB address space of destination host corresponding to described the above server of PCIE switch, comprising:

39. according to claim 37 or 38 described computer-readable mediums is characterized in that, the method that described computer executed instructions is used for server is carried out also comprises:

40. described computer-readable medium, is characterized in that according to claim 39, the method that described computer executed instructions is used for server is carried out also comprises:

41. the described computer-readable medium of according to claim 37 to 40 any one is characterized in that, the method that described computer executed instructions is used for server is carried out also comprises:

42. the described computer-readable medium of according to claim 35 to 41 any one, it is characterized in that, described computer executed instructions is used for making the method for server execution, the monitor of virtual machine of described destination host arrives described Data dissemination the purpose virtual machine of described network request according to the ID of the purpose virtual machine in described PCIE message, comprising: