User-Level Network Interface Protocols

Modern high-speed local area net-works offer great potential for communication- intensive applications, but their performance is limited by the use of traditional communication protocols, such as TCP/IP. In most cases, these protocols require that all network access be through the operating system, which adds significant overhead to both the transmission path (typically a system call and data copy) and the receive path (typically an interrupt, a system call, and a data copy). To address this performance problem, several user-level communication architectures have been developed that remove the operating system from the critical communication path. This article describes six important issues to consider in designing communication protocols for user-level architectures. The issues discussed focus on the performance and semantics of a communication system. These issues include data transfer, address translation, protection, and control transfer mechanisms, as well as the issues of reliability and multicast. To provide a basis for analyzing these issues, the authors present a simple network interface protocol for Myricom's Myrinet network, which has a programmable network interface. Researchers can thus explore many protocol design options, and several groups have designed communication systems for Myrinet. In this article, the authors refer to 11 such systems, all of which differ significantly in how they resolve these design issues but all of which aim for high performance and provide a lean, low-level, and more or less generic communication facility.