EP1222753A1

EP1222753A1 - Parallel bus communications over a packet-switching fabric

Info

Publication number: EP1222753A1
Application number: EP00958963A
Authority: EP
Inventors: Michael Kagan; Roni Ashuri; Shimon Rottenberg; Haggai Telem; Boris Spolyansky
Original assignee: Mellanox Technologies Ltd
Current assignee: Mellanox Technologies Ltd
Priority date: 1999-09-08
Filing date: 2000-09-07
Publication date: 2002-07-17
Also published as: AU7036600A; WO2001018989A1; IL148258A0; EP1222753A4

Abstract

A method for bridging between a parallel bus (22) and a packet-switched network (30) includes allocating at least one segment in an address space of the parallel bus for communications between devices (24) on the bus and the packet-switched network, and defining a correspondence between bus addresses within the at least one segment and respective network addresses on the network. When a bus command directed to a specified one of the bus addresses within the at least one segment is received, a packet is transmitted over the network to one of the network addresses responsive to the correspondence.

Description

PARALLEL BUS COMMUNICATIONS OVER A PACKET-SWITCHING FABRIC

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Patent Application 60/152,849, filed September 8, 1999, and of U.S. Provisional Patent Application 60/175,339, filed January 10, 2000. This application is related to another PCT patent application, filed on even date, entitled "Bridge between Parallel Buses over a Packet-Switched Network." All of these co-pending applications are assigned to the assignee of the present patent application and are incorporated herein by reference.

FIELD OF THE INVENTION The present invention relates generally to computing systems, and specifically to devices and methods for bridging between a parallel bus and a switching fabric or network.

BACKGROUND OF THE INVENTION

In current-generation computers, the central processing unit (CPU) is connected to the system memory and to peripheral devices by a parallel bus, such as the ubiquitous Peripheral Component Interface (PCI) bus. PCI bus protocols and rules are described in detail in the PCI Local Bus Specification, Revision 2.2 (1998), published by the PCI Special Interest Group (Hillsboro, Oregon), which is incorporated herein by reference. Briefly, the PCI specification defines a set of bus commands, or cycles, each identified by a 4-bit code on the C/BE (Bus Command and Byte Enable) lines of the bus. A bus master, such as a CPU, sends commands to a target device on the bus by specifying a 32- or 64-bit bus address of the target device. Table I below lists the PCI bus command codes for reference. The different types of read and write commands in the table are based on the division of the overall PCI address space into Memory, I/O and Configuration address spaces.

TABLE I - PCI BUS COMMANDS

As data path-widths grow, and clock speeds become faster, the parallel bus is becoming too costly and complex to keep up with system demands. In response, the computer industry is moving toward fast, packetized, serial input/output (I/O) bus architectures, in which computing hosts and peripheral are linked by a switching network, commonly referred to as a switching fabric. A number of architectures of this type have been proposed, including "Next Generation I/O" (NGIO) and "Future I/O" (FIO), culminating in the "InfiniBand" architecture, which has been advanced by a consortium led by a group of industry leaders (including Intel, Sun, Hewlett Packard, IBM, Compaq, Dell and Microsoft). Storage Area Networks (SAN) provide a similar, packetized, serial approach to high-speed storage access, which can also be implemented using an InfiniBand fabric.

SUMMARY OF THE INVENTION

It is an object of some aspects of the present invention to provide devices and methods for bridging between a parallel bus and a packet-switched network or fabric. It is a further object of some aspects of the present invention to provide methods and devices that map PCI bus commands to serial data packets and vice versa.

In preferred embodiments of the present invention, a bridge device links a parallel bus to a packet-switched network. Preferably, the parallel bus comprises a PCI bus, and the packet-switched network comprises an InfiniBand (IB) network, but the principles of the present invention are similarly applicable to buses and networks of other types. The bridge device receives bus commands that are issued by a master device to a specified address on the bus, and translates them into packets for transmission over the network. The packets are sent to a network address, typically a virtual address on the network, that is assigned to correspond to the specified bus address. Similarly, the bridge device receives packets from the network that are destined for devices on the bus, and translates them into appropriate bus responses or commands, to be conveyed to the proper address on the bus. The bridge device thus enables master devices on the bus to communicate transparently with destination devices on the network, as though the destination devices were actually present on the bus. It similarly enables devices on the network to communicate transparently with target devices on the bus.

In some preferred embodiments of the present invention, in order to allow master devices on the bus to communicate over the network, one or more segments are allocated to the bridge device in the address space of the bus. For the PCI bus, the segments are preferably allocated in each of the Memory, I/O and Configuration address spaces. Each segment is divided into one or more address ranges, and each range is associated with a unique channel on the network. Preferably, a channel lookup table held by the bridge device lists attributes of each of the channels. When the master device on the PCI bus sends a bus command to an address in one of the segments that are allocated to the bridge device, the bridge device looks up the address in the table in order to determine the channel to use and the attributes of the channel. These attributes also including a mapping between bus addresses and network addresses to which the packet is to be sent. The bridge device uses the attributes in the table to construct and transmit a packet of the appropriate type to the proper network address over the assigned channel. This flexible allocation of address segments and ranges, and the use of the lookup table to associate channel properties with address ranges on the PCI bus, make the bridge device particularly convenient to configure and use. When the bridge device receives packets over the network destined for devices on the PCI bus, a similar arrangement is preferably used to map the channels and network addresses to the corresponding bus addresses of the destination devices.

There is therefore provided, in accordance with a preferred embodiment of the present invention, a method for bridging between a parallel bus and a packet-switched network, including: allocating at least one segment in an address space of the parallel bus for communications between devices on the bus and the packet-switched network; defining a correspondence between bus addresses within the at least one segment and respective network addresses on the network; receiving a bus command directed to a specified one of the bus addresses within the at least one segment; and transmitting a packet over the network to one of the network addresses responsive to the correspondence. Preferably, allocating the at least one segment includes allocating a plurality of segments, wherein the bus includes a Peripheral Component Interface (PCI) bus, and the address space includes Memory, Input/Output (I/O) and Configuration address spaces, and wherein allocating the plurality of segments includes allocating the segments in at least two of the addresses spaces. Most preferably, the packet-switched network includes an InfiniBand switching fabric.

Preferably, defining the correspondence includes assigning a channel in the network to correspond to a given range of the bus addresses within the at least one segment, the given range including the specified bus address, and transmitting the packet includes sending the packet over the assigned channel. Optionally, the given range includes substantially all of the at least one segment. Alternatively, assigning the channel includes dividing the segment into a plurality of non-overlapping ranges, one of which is the given range.

Further preferably, assigning the channel includes recording attributes of the channel in a lookup table, and transmitting the packet includes looking up the attributes in the table responsive to the bus command so as to transmit the packet in accordance with the attributes. Preferably, looking up the attributes includes determining an index to the lookup table based on the bus address specified by the bus command. Alternatively or additionally, the recorded attributes of the channel include a status of the channel, and transmitting the packet includes deciding, responsive to the status, whether to transmit the packet or to return to a device on the bus that issued the bus command an indication that the packet could not be sent.

Preferably, transmitting the packet includes transmitting a request packet to a destination device on the network, and the method includes receiving a response packet over the network from the destination device, and generating a response on the parallel bus based on the response packet. Preferably, receiving the bus command includes receiving the command from a master device on the bus, and generating the response includes returning the response to the master device, such that the command and response are substantially indistinguishable to the master device from commands sent to and responses received from other devices on the bus with which the master device is in communication.

In a preferred embodiment, receiving the bus command includes receiving a read command for execution by a destination device at the one of the network addresses, and transmitting the packet includes sending a network read request packet to the destination device. Preferably, receiving the read command includes receiving a command to read a given quantity of data from the destination device, and sending the network read request packet includes setting a data length field in the read request packet responsive to the given quantity. Optionally, receiving the command to read the given quantity includes receiving a command to read one or more cache lines. In another preferred embodiment, receiving the bus command includes receiving a write command for execution by a destination device at the one of the network addresses, and transmitting the packet includes sending a network write request packet to the destination device.

Preferably, defining the correspondence includes defining a relation between the one or more bus addresses and one or more virtual addresses on the network.

There is also provided, in accordance with a preferred embodiment of the present invention, a method for bridging between a packet-switched network and a parallel bus, including: allocating at least one channel on the packet-switched network for communications with devices on the parallel bus; defining a correspondence, for use in the communications over the at least one channel, between one or more network addresses on the network that are accessible via the at least one channel and one or more respective bus addresses in an address space of the parallel bus; receiving a packet at one of the network addresses over the at least one channel; and conveying a bus command over the bus to one of the bus addresses responsive to the correspondence.

Preferably, the at least one channel includes a plurality of channels. In a preferred embodiment, the bus includes a Peripheral Component Interface (PCI) bus, and the network includes an InfiniBand switch fabric, wherein allocating the at least one channel includes associating a plurality of channels with respective InfiniBand Work Queue Pairs (WQPs).

Preferably, allocating the at least one channel includes recording attributes associated with the channel in a lookup table, and conveying the bus command includes looking up the attributes in the table responsive to a characteristic of the packet so as to convey the bus command in accordance with the recorded attributes. Further preferably, the characteristic of the packet includes a channel number assigned to the channel in the network over which the packet is received, and looking up the attributes includes determining an index to the lookup table based on the identification number and finding the attributes in the table indicated by the index.

In a preferred embodiment, determining the index includes using as the index a first portion of the identification number, and looking up the attributes includes matching a second portion of the identification number to a tag stored in the table in order to verify that the attributes indicated by the index pertain to the channel over which the packet is received. Preferably, the method includes refreshing the attributes recorded in the table with respect to the channel over which the packet is received when the second portion of the identification number does not match the tag.

Preferably, conveying the bus command includes determining the bus address to which to convey the command by transforming the network address at which the packet is received, based on the attributes recorded in the table. Most preferably, defining the correspondence includes defining a correspondence between one or more virtual addresses on the network and the one or more respective bus addresses. Further preferably, conveying the bus command includes conveying the command to a destination device on the bus at the one of the bus addresses, and the method includes receiving a response over the bus from the destination device, and transmitting a response packet over the at least one channel of the network responsive to the response. In a preferred embodiment, receiving the packet includes receiving a read request packet for execution by the destination device, and conveying the bus command includes generating a read cycle on the bus addressed to the destination device. Preferably, receiving the read request packet includes receiving a packet specifying a length of data to read from the destination device, and generating the read cycle includes selecting one of a plurality of types of read cycles to generate, responsive to the specified data length. In another preferred embodiment, receiving the packet includes receiving a write request packet for execution by the destination device, and conveying the bus command includes generating a write cycle on the bus addressed to the destination device.

There is additionally provided, in accordance with a preferred embodiment of the present invention, a bridge device for bridging between a parallel bus and a packet-switched network, including: a target channel adapter (TCA), adapted to receive a bus command over the parallel bus directed to a specified one of a plurality of bus addresses within a segment in an address space of the parallel bus, which segment is allocated to the TCA for communications between devices on the bus and the packet-switched network, the TCA being further adapted to generate, responsive to the bus command, a data packet for transmission over the packet-switched network to a network address corresponding to the specified bus address in accordance with a predefined address correspondence; and a network interface adapter, configured to transmit the data packet over the packet-switched network to the network address. In a preferred embodiment, the network interface adapter includes a switch.

There is further provided, in accordance with a preferred embodiment of the present invention, a device for bridging between a packet-switched network and a parallel bus, including: a network interface adapter, configured to receive a data packet sent to a specified network address over a channel on the packet-switched network, which channel is allocated on the network for communications with devices on the parallel bus; and a target channel adapter (TCA), which is adapted to select a bus address in an address space of the parallel bus corresponding to the specified network address in accordance with an address correspondence, defined for use in the communications over the at least one channel, between network addresses that are accessible via the channel and respective bus addresses in the address space of the parallel bus, and to convey a bus command to the selected bus address responsive to the received data packet.

The present invention will be more fully understood from the following detailed description of the preferred embodiments thereof, taken together with the drawings in which: BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram that schematically illustrates a system for bridging between a parallel bus and a packet-switched network, in accordance with a preferred embodiment of the present invention;

Fig. 2 is a block diagram that schematically illustrates details of a bridging device in the system of Fig. 1, in accordance with a preferred embodiment of the present invention; Fig. 3 is a flow chart that schematically illustrates a method for converting a bus command to a packet for transmission over a packet-switched network, in accordance with a preferred embodiment of the present invention;

Fig. 4 is a block diagram that schematically illustrates construction of a channel index for use in the method of Fig. 3, in accordance with a preferred embodiment of the present invention;

Figs. 5 and 6 are block diagrams that schematically illustrate schemes for segmenting memory space on a parallel bus, in accordance with a preferred embodiment of the present invention; and

Fig. 7 is a flow chart that schematically illustrates a method for converting a packet received over a packet-switched network to a bus command, in accordance with a preferred embodiment of the present invention.

DETADLED DESCRIPTION OF PREFERRED EMBODIMENTS

Fig. 1 is a block diagram that schematically illustrates a system 20 for bridging between a PCI bus 22 and an InfiniBand (IB) packet-switched network 30, in accordance with a preferred embodiment of the present invention. In this example, a master device on bus 22, such as a central processing unit (CPU) 24, wishes to read from or write to a destination device, such as an I/O device 38, over network 30. Alternatively, a second bridge device 32 may be used to communicate with a peripheral device 36 on bus 34. The use of bridge devices 28 and 32 in cooperation constitutes a PCI-to-PCI bridge and is described in greater detail in the above-mentioned patent application entitled "Bridge between Parallel Buses over a Packet-Switched Network." While preferred embodiments are described hereinbelow with reference to the PCI bus and the IB network, it will be understood that the principles of the present invention may be applied, mutatis mutandis, to bridge between parallel buses and networks of other types and obeying different standards.

In a typical scenario, CPU 24 issues an appropriate PCI read or write command to an address in the address space of PCI bus 22 that is assigned to bridge device 28. The remainder of the PCI bus address space is typically assigned to peripheral and I/O devices 26 on bus 22. Bridge device 28 generates an IB packet based on the CPU request and sends it over network 30 to device 38. The I/O device returns the appropriate response (typically data, in the case of a read command, or an acknowledgment of a write command) back to bridge device 28. This bridge device generates a corresponding PCI bus cycle, in which it passes the response over bus 22 to the CPU.

In accordance with the PCI standard, each device on PCI bus 22 can have multiple address segments, assigned to it in each of the Memory, Configuration and I/O address spaces on the bus 22. Preferably, bridge device 28 is designed so that thirty-two such segments can be assigned to the device, although alternatively, a larger or smaller number of segments may be available. Further preferably, the segments are divided into ranges, most preferably up to 128 ranges in all for the Memory and I/O segments, and another eight ranges for the Configuration segments. (The configuration ranges are preferably set aside for communication with the Configuration address space of other, remote PCI buses, such as bus 34.) Each range is associated with a channel over network 30, as described in detail hereinbelow. Upon start-up or reconfiguration of system 20, CPU 24 is informed of a PCI address for I/O device 38 that is within one of the address segments assigned to bridge device 28. The CPU addresses all read and write commands destined for device 38 to this address. Preferably, among the segments assigned to bridge device 28, there are also ranges associated with channels over network 30 to bridge device 32, for addressing peripheral device 36 on bus 34, as well as to additional network devices and PCI buses not shown in the figure. Each of the address ranges assigned to bridge device 28 is associated with a different channel, or Work Queue Pair (WQP) in IB network 30. Each channel in the IB network corresponds to a fixed path and level of service through the network, and packet ordering is preferably maintained among all packets carried by a given channel. Similar methods of maintaining packet order are known in other types of networks known in the art. As a consequence of this channel structure, all read and write requests directed to device 38 are received by the device in the exact order in which they were sent by bridge device 28. Responses of device 38 are similarly sent to back to bridge device 28 and are returned to CPU 24 in the proper order. This ordering is used in maintaining PCI bus conventions, as described in the above-mentioned patent application entitled "Bridge between Parallel Buses over a Packet-Switched Network. "

Fig. 2 is a block diagram that schematically illustrates details of bridging device 28, in accordance with a preferred embodiment of the present invention. Device 28 comprises a target channel adapter (TCA) 50, which is coupled to PCI bus 22, and a network interface, such as a switch 52, coupled to network 30. Preferred switches for this purpose are described in the above-mentioned provisional patent applications nos. 60/152,849 and 60/175,339, for example. The design of switch 52 is beyond the scope of the present patent application, however, and substantially any packet switch capable of interfacing with TCA 50 will be suitable for carrying out the function of switch 52. TCA 50 is responsible for converting PCI commands (or cycles) into IB packets, and vice versa. The TCA comprises a PCI target unit (PCIT) 54 and a PCI master unit (PCLM) 56. PCIT 54 responds to transactions originating on PCI bus 22, which are initiated by a bus master such as CPU 24. In particular, the PCIT receives read and write commands from the PCI bus and generates the appropriate IB read and write request packets for transmission via switch 52. When response packets are received from the destination device or bus, PCIT 54 receives these packets and generates the appropriate response cycles on PCI bus 22 (typically returning data in response to a read request or an acknowledgment in response to a write request). PCLM 56, on the other hand, is responsible for receiving IB read and write request packets from network 30, sent by IB device 38 or bridge device 32, for example. It converts these requests into the appropriate read and write commands on bus 22, which in this case is the destination bus. The PCLM then packetizes the resulting response to send back to the originating bus.

Fig. 3 is a flow chart that schematically illustrates a method by which PCIT 54 processes a PCI bus cycle and generates a corresponding IB packet, in accordance with a preferred embodiment of the present invention. As noted above, the PCIT is preferably capable of capturing PCI bus cycles with addresses in any of thirty-two different segments. Each segment is delimited by a lower bar address and an upper limit address, which are preferably programmable for each segment individually. The bar and limit for each segment are listed in a segment header, along with an identification of the PCI address space (Memory, I/O or Configuration) to which the segment belongs. The bar and limit are most preferably set on 1 MB boundaries in the PCI address space, so that the 20 least significant bits (LSB) of the bar are set to 00000, and the 20 LSB of the limit are set to FFFFF.

At a cycle reception step 70, PCIT 54 receives a cycle on the PCI bus and examines its C/BE code and destination address. At an address checking step 72, the address is compared to the bars and limits in the PCIT segment headers in order to determine whether it falls within one of the configured segments. Setting the bar and limit on 1 MB boundaries is useful in this regard, as it allows rapid comparison of the address to the bars and limits without using a great deal of logic or computing power. Cycles having addresses outside all of the PCIT segments are ignored, at a cycle discard step 74. When the cycle address is found to be within one of the segments, PCIT 54 proceeds to assemble the information needed to prepare a packet for transmission, beginning with a segment parameter reading step 76. The PCIT must first determine the IB channel that has been assigned to carry packets associated with the current cycle address, and it must then obtain relevant channel parameters for preparing and sending the packet. As noted above, each such channel corresponds to a range of the addresses within the segment, which may comprise the entire segment. The channel parameters are held in a target channel lookup table (CLT) 58 (Fig. 2). These parameters preferably include, but are not limited to, the following attributes: • EB Work Queue Pair (WQP) context, providing connection information for the channel such as the WQP number and the service type and level, destination ED. • LB channel properties, such as the channel status - whether the channel is enabled or disabled, packet serial number (PSN), and the number of delayed read or write request packets that the channel can handle.

• IB packet properties, for example - - Whether PCI Write cycles are to be conveyed on this channel as LB Remote Direct

Memory Access (RDMA) packets or as Send packets. (RDMA packets specify an address in the memory of the destination device to which data are to be written, while Send packets do not.)

- What data length to specify in RDMA Read packets, depending on the cache line size of the master device on the PCI bus and the type of PCI Read cycle.

- Latency parameters, i.e., how much data to accumulate at LB network device 38 before sending a read response packet back to bridge device 28.

- Address parameters, for mapping the PCI bus address to a corresponding IB virtual address. Storing these parameters in CLT 58 allows them to be accessed quickly and easily when a PCI bus cycle is addressed to one of the configured segments of the PCIT, and also allows simple reprogramming of channel attributes as needed.

The segment headers held by the PCIT preferably include, for each segment, a channel base index (CBI) and offset and mask parameters, which are read at step 76. The CBI and the offset and mask parameters are used in calculating a channel index (CI) at an index computation step 78, based on the PCI address range in which the bus command was received at step 70. The CI identifies the channel over which the packet is to be sent in response to the PCI bus command. It also serves as the index to CLT 58, at a lookup step 80, in order to determine the necessary attributes of the channel. Further details regarding the assignment of channels to PCI address ranges and of the computation of the CI are described hereinbelow.

The attributes retrieved from CLT 58 are used to build and send the corresponding packet, at a packet sending step 82. If the channel is unavailable, because the channel properties indicate that it is disabled or that its queue of delayed requests is full, PCIT 54 will not send the packet, but will rather return a "retry" or failure indication on PCI bus 22. Assuming the channel to be enabled, however, various types of packets may be sent through the LB network switching fabric, invoking different sorts of responses at the packet destination. According to IB convention, the packet type is specified by an Extended Transport Header (ETH) associated with the transport layer of the LB protocol stack. As noted above, read and write requests to a specific virtual address on the IB network are invoked by a Remote Direct Memory Access (RDMA) ETH, while write requests without a specific virtual address are carried by Send packets. The virtual address is preferably calculated by PCIT 54 by applying mask and address parameters, stored in CLT 58, to the PCI address of the corresponding read or write bus command. The first message of a RDMA read or write request contains a RDMA ETH specifying a virtual address for the operation, an access key (R_key) and a total length of the data buffer to read or write. The RDMA write request includes a payload containing the data to be written (which may be continued in subsequent RDMA write packets).

Table LI below lists all of the PCI bus commands that may be received by PCIT 54 on bus 22, along with the corresponding IB packet types that the PCIT will send over network 30 at step 82, in accordance with a preferred embodiment of the present invention:

TABLE II - CONVERSION OF PCI CYCLES TO IB PACKETS

When an entity such as IB device 38 receives the write request, it responds with an acknowledge packet, having an Acknowledge ETH (AETH). The AETH is also present in read response packets returned by device 38 when it receives the read request. Such read response packets also have a payload containing the requested data. Those skilled in the art will recognize that packets types and operations similar to these are also encountered in packet-switched networks of other sorts known in the art. PCIT 54 receives the response packet from device 38 and generates a corresponding PCI bus cycle on bus 22. Further details of the process by which PCIT 54 services read and write transactions initiated on bus 22 are described in the above-mentioned patent application entitled "Bridge between Parallel Buses over a Packet- Switched Network. "

Fig. 4 is a block diagram that schematically illustrates details of index computation step 78, in accordance with a preferred embodiment of the present invention. A channel index (CI) 94 is constructed by concatenating a specified number of mask bits 96 taken from a PCI address 90 at which the command was received, with a number of the most significant bits (MSB) 98 of channel base index (CBI) 92. A configuration bit 100 is preferably added to CI 94 to indicate whether address 90 belongs to one of the 128 ranges assigned to bridge device 28 in the memory and I/O address spaces of the PCI bus to the configuration address space, or to one of the eight special ranges assigned to the configuration space. The mask parameter in the segment header specifies the number of mask bits 96 to select from the PCI address (two bits in the example of Fig. 4), at an offset from the least significant bit (LSB) of the PCI address that is given by the offset parameter. Bits 98 are taken from CBI 92 to give a total CI length of 8 bits. The CBI bits are concatenated between mask bits 96 and configuration bit 100 to obtain the 8-bit CI 94. Figs. 5 and 6 are block diagrams that schematically illustrate the assignment of ranges in PCI Memory spaces 110 and 120 to IB channels, using the method of Fig. 4. In Fig. 5, Memory space 110 includes segments 3 and 4. Segment 3 has a bar and limit on 4 MB boundaries and is divided into four ranges, each 1 MB wide. Given an offset parameter of 20, the two mask bits 96 cycle in order through the segment from 0 to 3 (binary 00 to 11). With CBI 92 set to 8 (binary 100), CBI bits 98 in CI 94 will be 0001. Thus, the four ranges of segment 3 receive channel indices of 8, 9, A and B, respectively. Similarly, segment 4 has its bar aligned on a 16 MB boundary, with an offset of 22. The single mask bit 96 in this case sequentially takes the values 0 and 1, giving two 4MB ranges. With CBI 92 set in this case to C, the two ranges have channel indices of C and D.

In Fig. 6, Memory space 120 includes segments 0, 1 and 2, which in this case are not aligned on 4 MB boundaries. Thus, although segment 2 is divided into four ranges, having the same mask and offset values as segment 3 (Fig. 5), mask bits 96 of segment 2 sequence as 2, 3, 0, 1. With CBI 92 set to 4, the 1 MB ranges in segment 2 receive channel indices 6, 7, 4 and 5, in that order. Segment 0 is an example of non-uniform division of a segment into ranges. The two mask bits 96 for this segment are equal to 00 in the lowest 1 MB range, 01 over the next 2 MB, and 10 in the upper 1 MB, giving channel indices of 0, 1 and 2, respectively. Finally, by setting the offset and mask parameters to zero in segment 1, the entire 5 MB segment receives the same channel index, given by its CBI, which is set to 3. Fig. 7 is a flow chart that schematically illustrates a method by which PCLM 56 (Fig. 2) processes an IB packet and generates a corresponding PCI bus cycle, in accordance with a preferred embodiment of the present invention. In servicing IB packets, the PCLM refers to a master channel lookup table (CLT) 60, which, like target CLT 58, provides the channel parameters pertinent to generating the PCI cycle. The parameters include, but are not limited to:

• LB Work Queue Pair (WQP) context, providing connection information for the channel as in target CLT 58.

• IB channel properties, also as in target CLT 58, and including the message serial number (MSN) to be used in preparing the response packets to LB request packets that PCLM 56 receives.

• PCI cycle properties, for example - - Cycle type - whether this channel corresponds to PCI Memory, I/O or

Configuration cycles.

- Addressing attributes, including address map and mask, for converting IB virtual addresses to PCI bus addresses. PCLM 56 receives an IB packet, at a packet reception step 130, and attempts to look up the corresponding channel parameters in CLT 60. For this purpose, the PCLM preferably extracts the seven LSB of the WQP number for use as an index to the CLT, at an indexing step 132. It uses the index to look up the indicated CLT entry, at a lookup step 134. Preferably, master CLT 60 includes 128 entries, for up to 128 active channels, and the master CLT serves as a sort of data cache for parameters regarding these channels. Parameters of additional channels over which PCLM 56 can receive packets, beyond the active 128, are preferably stored in another memory. In order to determine whether the record in CLT 60 indicated by the channel index at step 134 is indeed the correct one for this channel, the PCLM checks the upper seventeen bits of the WQP number against a Master WQP tag stored in the CLT for this channel entry, at a tag matching step 136. If the tag matches, it means that the cache holds the correct data for this channel. Otherwise, the data in this entry of CLT 60 are preferably refreshed from memory, at a cache refresh step 138. Alternatively, it may be that this channel has not been configured in PCLM 56, in which case the packet is discarded, and an error is reported. Having found the appropriate entry in the CLT, PCLM 56 reads the PCI attributes that it will need in order to construct a corresponding PCI cycle, at an attribute reading step 140. Based on the attributes in the CLT, the PCLM validates the packet, checking, for example, that the packet has arrived in the proper order and carries appropriate protection parameters (P Key and R_Key in LB network convention). In other words, the PCLM implements transport layer functions of network 30. The PCLM then calculates the address on PCI bus 22 to which the cycle is to be addressed, at an addressing step 142. For RDMA read and write packets, the address map and mask parameters in the CLT are used to determine the PCI address based on the virtual address of the LB packet that was received. For IB send packets, PCLM 56 will generate write cycles to a PCI bus address specified in the CLT. The address may be incremented from one packet to the next. Finally, PCLM 56 generates the appropriate PCI cycle for the packet that it has received, at a cycle generation step 144. Table III below lists the types of PCI cycles that are generated in response to incoming IB packets, in accordance with a preferred embodiment of the present invention. Further details of the process by which PCLM 56 services read and write requests received over IB network 30 are described in the above-mentioned patent application entitled "Bridge between Parallel Buses over a Packet-Switched Network."

TABLE III - CONVERSION OF IB PACKETS TO PCI CYCLES

It will thus be observed that system 20 (Fig. 1) meets all of the requirements of both PCI bus 22 and LB network 30. Bridge device 28 enables devices on the PCI bus, such as CPU 24, to carry out transactions transparently with LB device 38 and with PCI peripheral device 36 on PCI bus 34 as though these remote devices were actually located on bus 22. The bridge device similarly enables LB device 38 to interact transparently with PCI bus devices. It will be apparent to those skilled in the art that the principles of system 20 and of the methods described hereinabove may be applied in a straightforward manner to buses, networks and transactions of other types. It will therefore be appreciated that the preferred embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

Claims

1. A method for bridging between a parallel bus and a packet-switched network, comprising: allocating at least one segment in an address space of the parallel bus for communications between devices on the bus and the packet-switched network; defining a correspondence between bus addresses within the at least one segment and respective network addresses on the network; receiving a bus command directed to a specified one of the bus addresses within the at least one segment; and transmitting a packet over the network to one of the network addresses responsive to the correspondence.

2. A method according to claim 1, wherein allocating the at least one segment comprises allocating a plurality of segments.

3. A method according to claim 2, wherein the bus comprises a Peripheral Component Interface (PCI) bus, and the address space comprises Memory, Input/Output (I/O) and

Configuration address spaces, and wherein allocating the plurality of segments comprises allocating the segments in at least two of the addresses spaces.

4. A method according to claim 3, wherein the packet-switched network comprises an InfiniBand switching fabric.

5. A method according to claim 1, wherein defining the correspondence comprises assigning a channel in the network to correspond to a given range of the bus addresses within the at least one segment, the given range including the specified bus address, and wherein transmitting the packet comprises sending the packet over the assigned channel.

6. A method according to claim 5, wherein the given range comprises substantially all of the at least one segment.

7. A method according to claim 5, wherein assigning the channel comprises dividing the segment into a plurality of non-overlapping ranges, one of which is the given range.

8. A method according to claim 5, wherein assigning the channel comprises recording attributes of the channel in a lookup table, and wherein transmitting the packet comprises looking up the attributes in the table responsive to the bus command so as to transmit the packet in accordance with the attributes.

9. A method according to claim 8, wherein looking up the attributes comprises determining an index to the lookup table based on the bus address specified by the bus command.

10. A method according to claim 8, wherein the recorded attributes of the channel comprise a status of the channel, and wherein transmitting the packet comprises deciding, responsive to the status, whether to transmit the packet or to return to a device on the bus that issued the bus command an indication that the packet could not be sent.

11. A method according to any of the preceding claims, wherein transmitting the packet comprises transmitting a request packet to a destination device on the network, and comprising receiving a response packet over the network from the destination device, and generating a response on the parallel bus based on the response packet.

12. A method according to claim 11, wherein receiving the bus command comprises receiving the command from a master device on the bus, and wherein generating the response comprises returning the response to the master device, such that the command and response are substantially indistinguishable to the master device from commands sent to and responses received from other devices on the bus with which the master device is in communication.

13. A method according to claim 11, wherein receiving the bus command comprises receiving a read command for execution by a destination device at the one of the network addresses, and wherein transmitting the packet comprises sending a network read request packet to the destination device.

14. A method according to claim 13, wherein receiving the read command comprises receiving a command to read a given quantity of data from the destination device, and wherein sending the network read request packet comprises setting a data length field in the read request packet responsive to the given quantity.

15. A method according to claim 14, wherein receiving the command to read the given quantity comprises receiving a command to read one or more cache lines.

16. A method according to claim 11, wherein receiving the bus command comprises receiving a write command for execution by a destination device at the one of the network addresses, and wherein transmitting the packet comprises sending a network write request packet to the destination device.

17. A method according to any of claims 1-10, wherein defining the correspondence comprises defining a relation between the one or more bus addresses and one or more virtual addresses on the network.

18. A method for bridging between a packet-switched network and a parallel bus, comprising: allocating at least one channel on the packet-switched network for communications with devices on the parallel bus; defining a correspondence, for use in the communications over the at least one channel, between one or more network addresses on the network that are accessible via the at least one channel and one or more respective bus addresses in an address space of the parallel bus; receiving a packet at one of the network addresses over the at least one channel; and conveying a bus command over the bus to one of the bus addresses responsive to the correspondence.

19. A method according to claim 18, wherein the at least one channel comprises a plurality of channels.

20. A method according to claim 18, wherein the bus comprises a Peripheral Component Interface (PCI) bus, and wherein the network comprises an InfiniBand switch fabric.

21. A method according to claim 20, wherein the address space comprises Memory, Input/Output (I/O) and Configuration address spaces, and wherein allocating the at least one channel comprises allocating a first channel for communicating with one of the address spaces and a second channel for communicating with another one of the address spaces.

22. A method according to claim 20, wherein allocating the at least one channel comprises associating a plurality of channels with respective InfiniBand Work Queue Pairs (WQPs).

23. A method according to claim 18, wherein allocating the at least one channel comprises recording attributes associated with the channel in a lookup table, and wherein conveying the bus command comprises looking up the attributes in the table responsive to a characteristic of the packet so as to convey the bus command in accordance with the recorded attributes.

24. A method according to claim 23, wherein the characteristic of the packet comprises a channel number assigned to the channel in the network over which the packet is received, and wherein looking up the attributes comprises determining an index to the lookup table based on the identification number and finding the attributes in the table indicated by the index.

25. A method according to claim 24, wherein determining the index comprises using as the index a first portion of the identification number, and wherein looking up the attributes comprises matching a second portion of the identification number to a tag stored in the table in order to verify that the attributes indicated by the index pertain to the channel over which the packet is received.

26. A method according to claim 25, and comprising refreshing the attributes recorded in the table with respect to the channel over which the packet is received when the second portion of the identification number does not match the tag.

27. A method according to claim 23, wherein conveying the bus command comprises determining the bus address to which to convey the command by transforming the network address at which the packet is received, based on the attributes recorded in the table.

28. A method according to any of claims 18-27, wherein defining the correspondence comprises defining a correspondence between one or more virtual addresses on the network and the one or more respective bus addresses.

29. A method according to any of claims 18-27, wherein conveying the bus command comprises conveying the command to a destination device on the bus at the one of the bus addresses, and comprising receiving a response over the bus from the destination device, and transmitting a response packet over the at least one channel of the network responsive to the response.

30. A method according to claim 29, wherein receiving the packet comprises receiving a read request packet for execution by the destination device, and wherein conveying the bus command comprises generating a read cycle on the bus addressed to the destination device.

31. A method according to claim 30, wherein receiving the read request packet comprises receiving a packet specifying a length of data to read from the destination device, and wherein generating the read cycle comprises selecting one of a plurality of types of read cycles to generate, responsive to the specified data length.

32. A method according to claim 29, wherein receiving the packet comprises receiving a write request packet for execution by the destination device, and wherein conveying the bus command comprises generating a write cycle on the bus addressed to the destination device.

33. A bridge device for bridging between a parallel bus and a packet-switched network, comprising: a target channel adapter (TCA), adapted to receive a bus command over the parallel bus directed to a specified one of a plurality of bus addresses within a segment in an address space of the parallel bus, which segment is allocated to the TCA for communications between devices on the bus and the packet-switched network, the TCA being further adapted to generate, responsive to the bus command, a data packet for transmission over the packet-switched network to a network address corresponding to the specified bus address in accordance with a predefined address correspondence; and a network interface adapter, configured to transmit the data packet over the packet-switched network to the network address.

34. A device according to claim 33, wherein the segment comprises one of a plurality of segments that are allocated to the TCA.

35. A device according to claim 34, wherein the bus comprises a Peripheral Component Interface (PCI) bus, and the address space comprises Memory, Input/Output (I/O) and Configuration address spaces, and wherein the plurality of segments comprises segments in at least two of the addresses spaces.

36. A device according to claim 35, wherein the packet-switched network comprises an InfiniBand switching fabric.

37. A device according to claim 33, wherein the network interface adapter comprises a switch.

38. A device according to claim 33, wherein the TCA is adapted to assign the data packet to be transmitted by the network interface adapter over a predetermined channel in the network, the channel having been selected to correspond to a given range of the bus addresses within the segment, the given range including the specified bus address.

39. A device according to claim 38, wherein the given range comprises substantially all of the at least one segment.

40. A device according to claim 38, wherein the segment is divided into a plurality of non-overlapping ranges, one of which is the given range, and wherein different, channels in the network are selected to correspond respectively to the plurality of ranges.

41. A device according to claim 38, wherein the TCA is adapted to store a record of attributes of the channel in a lookup table, and to look up the attributes in the table responsive to the bus command so as to generate the packet in accordance with the attributes.

42. A device according to claim 41, wherein the TCA is adapted to derive an index to the lookup table from the bus address specified by the bus command.

43. A device according to claim 41, wherein the record of attributes of the channel comprises an indication of a status of the channel, and wherein the TCA is adapted to decide, responsive to the status indication, whether to transmit the packet or to return to a master device on the bus that issued the bus command an indication that the packet could not be sent.

44. A device according to any of claims 33-43, wherein the packet comprises a request packet for transmission to a destination device at the network address, and wherein the TCA is adapted to receive a response packet over the network from the destination device, and to generate a response on the parallel bus based on the response packet.

45. A device according to claim 44, wherein the bus command is generated by a master device on the bus, and wherein the TCA is adapted to return the response to the master device, such that the command and response are substantially indistinguishable to the master device from commands sent to and responses received from other devices on the bus with which the master device is in communication.

46. A device according to claim 44, wherein the bus command comprises a read command for execution by the destination device, and wherein the data packet generated by the TCA comprises a network read request packet for transmission to the destination device.

47. A device according to claim 46, wherein the read command comprises a command to read a given quantity of data from the destination device, and wherein the TCA is adapted to set a data length field in the read request packet responsive to the given quantity.

48. A device according to claim 44, wherein the bus command comprises a write command for execution by the destination device, and wherein the data packet generated by the TCA comprises a network write request packet for transmission to the destination device.

49. A device according to any of claims 33-43, wherein the network address comprises a virtual address, and wherein the predefined address correspondence comprises a relation between the plurality of bus addresses and respective virtual addresses on the network.

50. A device according any of claims 33-43, wherein the data packet transmitted by the network interface adapter comprises an outgoing packet, and wherein the network interface adapter is further configured to receive an incoming data packet over the packet switched network and to pass the incoming packet to the TCA, which is adapted to generate a bus command to be conveyed over the parallel bus, responsive to the incoming packet.

51. A device for bridging between a packet-switched network and a parallel bus, comprising: a network interface adapter, configured to receive a data packet sent to a specified network address over a channel on the packet-switched network, which channel is allocated on the network for communications with devices on the parallel bus; and a target channel adapter (TCA), which is adapted to select a bus address in an address space of the parallel bus corresponding to the specified network address in accordance with an address correspondence, defined for use in the communications over the at least one channel, between network addresses that are accessible via the channel and respective bus addresses in the address space of the parallel bus, and to convey a bus command to the selected bus address responsive to the received data packet.

52. A device according to claim 51, wherein the channel comprises one of a plurality of channels over which the network interface adapter is configured to receive data packets.

53. A device according to claim 51, wherein the bus comprises a Peripheral Component Interface (PCI) bus, and wherein the network comprises an InfiniBand switch fabric.

54. A device according to claim 53, wherein the address space comprises Memory, Input/Output (I/O) and Configuration address spaces, and wherein the channel comprises one of a plurality of channels over which the network interface adapter is configured to receive data packets, including at least a first channel for communicating with one of the address spaces and a second channel for communicating with another one of the address spaces.

55. A device according to claim 54, wherein the plurality of channels are associated with respective InfiniBand Work Queue Pairs (WQPs).

56. A device according to any of claims 51-55, wherein the TCA is adapted to store attributes associated with the channel in a lookup table, and to look up the attributes in the table responsive to a characteristic of the packet so as to convey the bus command in accordance with the recorded attributes.

57. A device according to claim 56, wherein the characteristic of the packet comprises a channel number assigned to the channel in the network over which the packet is received, and wherein the TCA is adapted to determine an index to the lookup table based on the identification number and to find the attributes in the table indicated by the index.

58. A device according to claim 57, wherein the TCA is adapted to use as the index a first portion of the identification number, and to match a second portion of the identification number to a tag stored in the table in order to verify that the attributes indicated by the index pertain to the channel over which the packet was received.

59. A device according to claim 58, wherein the TCA is adapted to refresh the attributes recorded in the table with respect to the channel over which the packet was received when the second portion of the identification number does not match the tag.

60. A device according to claim 56, wherein the TCA is adapted to select the bus address to which the command is to be conveyed by transforming the network address to which the packet was directed, based on the attributes recorded in the table.

61. A device according to any of claims 51-55, wherein the address correspondence comprises a correspondence between one or more virtual addresses on the network and the one or more respective bus addresses.

62. A device according to any of claims 51-55, wherein the TCA is adapted to convey the command to a destination device on the bus and to receive a response over the bus from the destination device, and further to generate a response packet for transmission by the network interface adapter over the at least one channel of the network responsive to the response.

63. A device according to claim 62, wherein the packet comprises a read request packet for execution by the destination device, and wherein the bus command comprises a read cycle on the bus addressed to the destination device.

64. A device according to claim 63, wherein the read request packet comprises a specification of a length of data to read from the destination device, and wherein the TCA is adapted to select one of a plurality of types of read cycles to generate, responsive to the specified data length.

65. A device according to claim 62, wherein the packet comprises a write request packet for execution by the destination device, and wherein the bus command comprises a write cycle on the bus addressed to the destination device.

66. A device according to any of claims 51-55, wherein the network interface adapter comprises a switch.