KR102479757B1 - Offloading method and system of network and file i/o operation, and a computer-readable recording medium - Google Patents

Abstract

A method for offloading file input/output operations according to the present invention is a method for offloading network and file input/output operations in a central processing unit (CPU) of a file-based content transmission server and a network interface card (NIC) having an operation function. , an offload step of offloading data plane operations to the NIC; and a performing step in which the NIC performs the data plane operation and the CPU performs a control plane operation excluding the offloaded data plane operation, wherein the data plane operation includes: fetching content; It includes at least one operation of data buffer management, data packet division, TCP/IP checksum calculation, data packet generation and transmission, wherein the control plane operation includes TCP protocol connection management, data transmission stability management, data transmission congestion management, It includes an operation of at least one of flow control and error control.

Description

Network and file input/output operation offloading method, operation offloading system, and computer readable recording medium

The present invention improves content transmission performance per unit CPU capacity by offloading file I/O and network I/O, which are the bottlenecks of a file-based content transmission server, from the CPU and having them performed in an auxiliary device (eg SmartNIC) having an arithmetic function. It relates to a network and file input/output operation offloading method, an operation offloading system, and a computer readable recording medium that can be significantly improved.

Recent advances in high-bandwidth I/O devices promise fast delivery of high-quality online content. Unfortunately, however, today's programming models are still bound by CPU-centric approaches, whose bottlenecks severely limit the true potential of modern I/O devices. More than 85% of CPU cycles are used for simple but repetitive operations such as disk and network I/O operations when transferring online content.

On the other hand, recent computing hardware trends have shown that advances in I/O device capacity have substantially outpaced growth in CPU capacity. Over the past 20 years, network interface card (NIC) bandwidth and disk I/O speeds have improved by a factor of two to four. Conversely, progress in CPU capacity has slowed and it is widely accepted that Moore's Law for processors ended more than a decade ago. This trend calls for a rethinking of the CPU-centric programming model on which modern operating systems are based. Current OS APIs can perform operations only when all data from the I/O device is fetched from the main memory directly connected to the CPU. For this reason, in the case of a video content delivery server, the CPU is used for simple I/O operations (reading the contents of the fast disk and transmitting them through a high-bandwidth NIC) for more than 85% of the total cycle, so I/O-intensive applications such as video content delivery performance bottlenecks frequently occur. To effectively take advantage of modern innovations in I/O devices, current program structures must mitigate their dependence on the CPU and its memory system to perform I/O operations.

Fortunately, programmable I/O devices such as SmartNICs or Computational SSDs have recently emerged to help reduce the computational burden on the CPU without CPU involvement. The PCI-e standard allows for peer-to-peer DMA (P2PDMA) between two PCI-e devices without CPU intervention, so you can think of a server system where the NIC completely offloads disk I/O operations from the CPU. Indeed, recent work such as DCS and DCS-Ctrl demonstrates that FPGA-based coordinators can perform all disk I/O operations via P2PDMA for video delivery servers. Unfortunately, however, the major drawback of these systems is that the content delivery runs over a protocol similar to UDP, as opposed to the TCP-based transport commonly employed in today's video streaming. However, supporting TCP in an I/O offload server raises the "feature placement" problem of where to place the TCP functions. That is, if all disk I/O is done on a programmable NIC, the question arises where to run the TCP stack. One approach would be to run it on the CPU side, but this introduces the problem of "missing data" for data packets because the disk content is only available on the NIC. The problem with moving data back to the CPU negates the benefits of offloaded disk I/O.

Another approach is to run a TCP stack on the NIC side. Implementing a full TCP stack on an FPGA board is not easy, but it is doable on a SmartNIC. In fact, modern SmartNIC platforms support Arm-based embedded processors running on Linux with full TCP stacks. However, running the full TCP stack on the NIC has the drawback that those applications must co-run on the same platform with limited resources.

The present invention has been devised in view of the above problems, and the present invention offloads file I/O and network I/O, which are the bottlenecks of the content delivery server, from the CPU, so that the CPU is complex, but the control plane (which must be performed quickly) Only control plane) operations are performed, and simple but repetitive and mechanical data plane operations are performed in peripheral devices (SmartNIC, etc.), but network and file input/output operation offloading methods that are perfectly compatible with the TCP protocol, operation offloading It is to provide a system and a computer readable recording medium.

In order to achieve the above object, a file input/output operation offloading method according to the present invention is provided by offloading network and file input/output operations in a central processing unit (CPU) of a file-based content transmission server and a network interface card (NIC) having an operation function. As a loading method, an offload step of offloading, by the CPU, a data plane operation to the NIC; and a performing step in which the NIC performs the data plane operation and the CPU performs a control plane operation excluding the offloaded data plane operation, wherein the data plane operation includes: fetching content; It includes at least one operation of data buffer management, data packet division, TCP/IP checksum calculation, data packet generation and transmission, wherein the control plane operation is TCP protocol connection management, data transmission stability management, and data transmission congestion management , flow control and error control.

The offloading step may include transmitting, by the CPU, an "offload_open()" command received from an application program to the NIC; the NIC opening a file on the NIC stack and returning a result to the CPU; and transmitting, by the CPU, an “offload_write()” command called from the application program to the NIC.

A virtual operation step of, by the CPU, virtually performing the data plane operation when actual data is missing from the host stack of the CPU.

The virtual operation step may include, by the CPU, when the application program calls the "offload_write()" command, updating only meta data and virtually writing it in a transmission buffer; and posting a "SEND" command to the NIC stack of the NIC after the CPU determines the number of bytes to be sent along with the congestion and flow control parameters of the data.

The CPU transmits the "SEND" command using a TCP packet, and the TCP packet may include at least one of file ID, starter offset to read, and data length information. .

Upon receiving the “SEND” command, the NIC may convert and transmit a plurality of TCP packets based on length information included in the TCP packet.

When the client tries to transmit a data packet for the “SEND” command, it transmits an echo packet to the CPU, and the CPU may not transmit a retransmission timer for a predetermined time until receiving the echo packet. .

The predetermined time may be determined by a packet processing delay time in the NIC and a unidirectional processing time of the echo packet.

A packet processing delay time in the NIC may be a delay time between when the "SEND" command arrives at the NIC and a data packet for the "SEND" command departs from the client.

The CPU includes the "OPEN" command used to open the file of the IO-TCP application program, the "SEND" command used to send the contents of the file to the client, the "ACKD" command for efficient processing of retransmission, and the IO-TCP application program You can define the "CLOS" command used to close a file in .

The CPU offloads only the "SEND" command to the NIC, and can directly process packets received from clients.

When the CPU confirms an ACK for the offloaded sequence space range, the CPU may periodically transmit the “ACKD” packet to the NIC.

On the other hand, a calculation offloading system for achieving the above object is a network and file input/output calculation offloading system including a central processing unit (CPU) of a file-based content transmission server and a network interface card (NIC) having a calculation function, The CPU offloads data plane operations to the NIC, the NIC performs the data plane operations, and the CPU performs a control plane excluding the offloaded data plane operations An operation is performed, wherein the data plane operation includes at least one operation of fetching content, managing data buffers, dividing data packets, calculating a TCP/IP checksum, generating and transmitting data packets, and the control plane operation is It includes at least one operation of connection management, data transmission stability management, data transmission congestion management, flow control, and error control.

When the CPU transmits the “offload_open()” command received from the application program to the NIC, the NIC opens a file on the NIC stack and returns a result to the CPU, and the CPU responds with an “offload_write() command called from the application program. )" command to the NIC.

When actual data is missing from the host stack of the CPU, the CPU may virtually perform the data plane operation.

When the application program calls the "offload_write()" command, the CPU updates only the meta data and virtually writes it to the transmission buffer, determines the number of bytes to be transmitted along with the congestion and flow control parameters of the data, and then " SEND" command to the NIC stack of the NIC.

The CPU transmits the "SEND" command by utilizing a TCP packet, the TCP packet includes at least one information of a file ID, a starter offset to read, and data length information, Upon receiving the “SEND” command, the NIC may convert and transmit a plurality of TCP packets based on length information included in the TCP packet.

The client transmits an echo packet to the CPU when attempting to transmit a data packet for the "SEND" command, and the CPU does not transmit a retransmission timer for a predetermined time until receiving the echo packet. The predetermined time is determined by the packet processing delay time in the NIC and the one-way processing time of the echo packet, and the packet processing delay time in the NIC is determined by the “SEND” command It may be the delay between data packets departing for the client.

The CPU includes the "OPEN" command used to open the file of the IO-TCP application program, the "SEND" command used to send the contents of the file to the client, the "ACKD" command for efficient processing of retransmission, and the IO-TCP application program defines the "CLOS" command used when closing the file of , the CPU offloads only the "SEND" command to the NIC, directly processes packets received from clients, and the CPU If the ACK for the ACK is confirmed, the “ACKD” packet may be periodically transmitted to the NIC.

Meanwhile, the computer readable recording medium according to the present invention may record a program for executing the above-described operation offloading method in a computer.

According to the present invention, it is possible to design a network stack that completely supports the TCP protocol while performing disk access offloading using P2P DMA and controlling the TCP operation on the CPU side. In addition, in the prior art, P2P DMA was implemented with an FPGA device to directly access the NVMe disk without CPU intervention, but the TCP protocol was not supported. Therefore, considering that most of the modern video streaming services are based on TCP, the commercialization potential of the prior art is inevitably low. On the other hand, the present invention has the advantage that it can be directly applied to video services by supporting the TCP protocol.

Figure 1 shows an architectural overview of the IO-TCP stack.
Figure 2 shows the NVMe disk utilization of a single CPU core as measured by fio, a widely used disk benchmark tool.
Figure 3 shows the result of a single CPU core in the performance comparison of web servers.
Figure 4 shows the architecture of the Mellanox Blue-Field NIC used in the platform.
Figure 5 shows a subset of operations using API functions in an HTTP server context.
Figure 6 shows how the "SEND" command packet is processed.
7 illustrates round-trip time of Ethernet for communication between a host and a NIC stack.
8 shows the result of evaluating the effect of IO-TCP on large-capacity file content delivery.
Figure 9 shows the throughput (a) for a fraction of CPU cycles consumed by the host stack and the throughput (b) for a single SmartNIC as a function of the number of Arm cores.
Figure 10 shows the effect (a) of the echo packet starting the retransmission timer at the time the actual data packet is transmitted and the effect (b) of TCP timestamp modification in the NIC stack.
11 shows throughput test results for the number of simultaneous communication connections.
Figure 12 shows the throughput of serving a 4KB file for different numbers of concurrent communication connections.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar elements are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffix "part" for components used in the following description is given or used interchangeably in consideration of ease of writing the specification, and does not itself have a meaning or role distinct from each other. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

First of all, in the present invention, I/O Offloading TCP (IO-TCP), which is a split TCP stack design for I/O intensive applications, solves the above-mentioned problems. The core idea of IO-TCP is to run only the control plane on the CPU while delegating all data plane operations to a SmartNIC that can access the disks (e.g. PCIe connected disks like NVMe disks) via P2PDMA. will be. The control plane refers to all the core functions of the TCP protocol (connection management, reliability management of data transmission, data flow control and congestion management). Refers to all aspects of transmission.

The present invention allows the CPU side to still fully control all operations. Internally, the actual disk and network I/O operations are offloaded to the SmartNIC. This allows CPU cycles to be reserved for complex control operations and excluded from simple but repetitive I/O operations.

IO-TCP offers great potential for saving CPU cycles, but it also presents a series of new challenges. First, the host TCP's RTT measurement fluctuates due to disk-induced delays. In IO-TCP, the NIC's data packet I/O is combined with the disk I/O as it fetches the packet content from the disk. However, disk I/O can be much slower and often adds significant delay to packet transmission even when there is no congestion on the network path. IO-TCP solves this problem by carefully removing disk-induced delays in the RTT measurement. Echo packets are used so that the host stack can accurately track packet departure time. Second, IO-TCP has to handle packet retransmission in the NIC stack without information about when to retransmit the packet. If you're not careful, the NIC stack can issue redundant disk I/Os to resend packets. To avoid inefficiencies, IO-TCP uses an internal ACK protocol to signal when it is safe to discard data packets. Third, IO-TCP must provide a well-defined API for applications to flexibly compose file or non-file content for data transfer. To do this, IO-TCP slightly extends the Berkeley Sockets API with some "offload" functions to open files and send file contents from the NIC.

Figure 1 shows an architectural overview of the IO-TCP stack. IO-TCP is implemented with a Mellanox BlueField Smart-NIC that can directly access NVMe disks with P2PDMA. For the host stack, the existing user-level TCP stack is extended to support I/O offloading while the NIC stack is implemented with the DPDK library. In the case of the host stack, 1,793 lines of code modification is required, and in the case of the NIC stack, 1,853 lines of C code are required. To evaluate the effect of real application, Lighttpd is ported to use IO-TCP with only about 10 lines of code modification.

As a result of the evaluation, the IO-TCP port Lighttpd achieves 44Gbps video content transfer with a single CPU core. In contrast, the original Lighttpd on Linux TCP stack needs 6 CPU cores to reach the same performance. We know the current bottleneck is in the BlueField NIC's low memory bandwidth, but future versions may achieve better performance. The invention has the following technical significance.

(1) Analysis of the effect of the CPU bottleneck on disk I/O on the performance of the latest content delivery system

(2) Introducing the design and implementation of IO-TCP, a new TCP stack design that allows content delivery systems to fully utilize the recent I/O advances in SmartNICs by separating the TCP control and data planes.

(3) IO-TCP breaks the limits of the CPU bottleneck and presents a way to achieve much greater I/O bandwidth than the CPU can normally do.

Mismatch between I/O Device Advances and CPU Capacity

Twenty years ago, the fastest hard disks could achieve around 200 random I/O operations per second (IOPS), but recent NVMe disks can do over 1 million IOPS, nearly 40 times faster. Over the same period, the bandwidth of Ethernet NICs has improved by more than 200 times (from 1 Gbps in 1997 to 200 Gbps in 2019), and 800 Gbps/1.6 Tbps Ethernet is expected to become standard within a few years.

In contrast, CPU capacity growth was greatly hampered by the end of Moore's Law and the collapse of the Dennard extension. The first general-purpose multi-core CPU appeared in 2005, but the maximum number of cores per CPU remains at 32 to date.

Figure 2 shows the NVMe disk utilization of a single CPU core as measured by fio, a widely used disk benchmark tool. It uses an Intel Xeon Silver 4210 (2.20 GHz) for the CPU and an Intel Optane 900P for the NVMe device. Figure 2 shows that a single CPU core cannot saturate even two NVMe disks with a unit block size of 4KB. Even with a block size of 16KB, a single core can only process up to 3 NVMe disks in parallel. Given that NVMe disks typically occupy 4 PCIe lanes and modern server-grade CPUs can support 40-48 PCIe lanes, a content delivery server with a single CPU can host up to 8-10 NVMe disks in addition to a high-speed NIC. You can do it. The difference in performance between CPUs and I/O devices requires a reexamination of current OS abstractions for I/O operations. Existing OSs require CPU intervention to perform I/O operations, such as reading disk contents and sending them over the NIC. This is because the current programming model requires that the contents of the I/O device be moved to main memory before performing the next operation, so memory operations are interrupted and CPU cycles are wasted.

Inefficiencies in Content Delivery System Stacks

Modern content delivery systems consist of numerous geographically distributed content delivery webs or reverse proxy servers. These systems are the basis for many applications such as video streaming and web page access. Among them, video traffic accounts for about 60% of all Internet traffic, and the total volume has increased due to the recent explosive increase in demand. To optimize performance, server design has traditionally focused on optimizing disk access and CPU utilization, as hard disk I/O has been much slower. For fetching small objects such as web page content, the server is optimized to minimize disk searches while maintaining a small memory footprint for indexing. For large object accesses such as video downloads, the server uses sequential disk reads to maximize disk throughput. It also supports sendfile() in general to avoid frequent memory copying and context switching between user processes and the kernel.

To increase CPU utilization, servers typically adopt an event-driven architecture. Traditional disk-based optimization has largely become obsolete with the advent of cheap, high-capacity RAM and flash-based disks (such as SSDs and NVMe disks) that have removed the limitations imposed by seek. Now that the major disk bottleneck has been resolved, the memory subsystem becomes the next bottleneck in today's servers. This is exacerbated by multiple memory copies resulting from disk and network I/O as well as content scanning for encryption and decryption. Recent work optimizes the disk access layer and utilizes Intel Data Direct I/O (DDIO) to arrange all these operations to be performed with data in the CPU cache, but without offloading the CPU's computational load. So, when the workload exceeds the CPU cache size, performance drops to that of the memory subsystem. To understand the performance of web-based content delivery, two popular web servers, Lighttpd (v1.4.32) and nginx (v1) for a disk-bound workload simulating a typical setup for HTTP adaptive video streaming. .16.1) to run the experiment. The server settings are the same as in Section 5.1, and we use file sizes of 100 KB, 300 KB, and 500 KB to simulate different video qualities. Use 1600 persistent connections and make sure the client is not a bottleneck.

Compare the performance of web servers with and without sendfile() optimization. Figure 3 shows the result of a single CPU core in the performance comparison of web servers. In general, larger file sizes improve performance, with sendfile() improving performance from 12% to 52%. Without sendfile() nginx shows better performance, but Lighttpd shows similar performance to sendfile(). Given that a single NVMe disk achieves around 2.5GB/s (or 20Gbps) for random file reads, a single CPU core has just over half the performance of a single NVMe disk with a 300KB file.

[Table 1] shows nginx's CPU cycle analysis at the function call level as measured by perf.

Function %CPU sendfile() 71.59% open() 14.55% recv() 1.76% ngx_http_finalize_request() 3.52% ngx_http_send_header() 1.17% others 7.41%

Not surprisingly, sendfile() and open() take up most of the CPU cycles, accounting for 86.1% of the cycles consumed. This clearly shows where most of the CPU cycles are being used on the content delivery server (disk and network I/O). Offloading these operations from the CPU has great potential to improve performance.

Opportunities with SmartNIC

The core idea of the present invention is to offload data I/O from the CPU to a programmable I/O device while supporting TCP-based content delivery. Ideally, any programmable device that can do direct disk I/O and network packet I/O would accomplish that goal. For its implementation, a modern SmartNIC platform with P2PDMA capability can be selected, and the design could be implemented on an FPGA board as well. Modern SoC-based SmartNICs such as the Mellanox Blue-Field and Broadcom Stingray provide an Arm-based embedded system on top of the NIC data processing unit. These systems support direct access to NVMe disks in the same domain without CPU or main memory involvement. For example, BlueField NICs support P2PDMA with NVMe over Fabrics (NVMe-oF) target offload. It offloads all NVMe-oF operations to hardware accelerators, and Arm processors can connect to the system and read directly from local NVMe disks. These disks are mounted directly into the Linux environment running on the Arm processor and run on the same filesystem as is visible to the host OS. In terms of performance, the fio of the BlueField NIC achieves 2.5GB/sec per Intel Optane 900P NVMe disk, which is similar to what was achieved on the host CPU side.

Figure 4 shows the architecture of the Mellanox Blue-Field NIC used in the platform. It is equipped with 16 Armv8 cores and 16GB of DDR4 memory running on Linux. The Arm subsystem allows you to run DPDK applications to perform fast packet I/O on remote systems or on the local host. Applications can also offload TCP/IP checksum computation and TCP segmentation (ie, TSO) to the ConnectX-5 NIC hardware. However, having all packets pass through an embedded system can introduce significant packet forwarding overhead. To prevent this, the NIC (eSwitch in Fig. 4) can be configured so that all received packets are directly forwarded to the host CPU side and the NIC embedded system processes only outbound packets. The processor and memory are not as powerful compared to the CPU. The maximum memory bandwidth of the BlueField NIC's Arm subsystem is only 19.2 GB/s, which limits overall performance, but this is unlikely to be an inherent limitation as Armv8-based SoCs can support memory bandwidths in excess of 100 GB/s.

Design

In the present invention, we present a design of IO-TCP that allows content delivery systems to take advantage of recent SmartNIC I/O advancements. A key design choice for IO-TCP is to separate the control and data planes of the TCP stack, allowing the CPU side to fully control all operations while individual I/O operations are offloaded to the SmartNIC. The key rationale for this is to save most CPU cycles on I/O operations while keeping the NIC stack easy to implement. Simplicity is key to performance scalability.

IO-TCP has three design goals: First, IO-TCP must conform to the TCP protocol and be able to support various congestion control implementations. For example, handling disk I/O on the NIC stack should not break the host stack's congestion control logic due to inaccurate RTT measurements due to disk access latency (discussed below). Second, modifications to existing applications should be minimized to migrate to IO-TCP. Except for file I/O offloading (discussed below), you must use the same socket API. Third, the IO-TCP host stack must communicate with the NIC stack for I/O offloading and reduce overhead. It also needs to notify the host stack of any errors in the NIC stack in a timely manner so that it can properly handle them (described below).

Separating TCP control and data planes

To save host CPU cycles, determine which operations will benefit most from offloading based on the capabilities of the SmartNIC and CPU. Embedded processors in SoCs or ASIC-based SmartNICs are better suited for simpler data plane operations, while x86 CPUs with advanced features can handle complex control plane operations faster. Therefore, the TCP stack is divided into control plane and data plane operations.

Control plane functions represent key TCP protocol functions such as connection management, reliable data transmission, congestion/flow control, and error control. Actions depend on the response of the other side, so complex state management is usually required. For example, for reliable data delivery at the receiver side, all separated received data ranges must be tracked for proper sequential delivery and ACK generation. It is also tightly coupled with congestion control with loss detection and packet retransmission, resizing the transmission window for reliable delivery. Likewise, flow control also affects window size, so it should be run in conjunction with congestion control. Error control cannot be run alone as it needs to track detailed flow states in order to infer error behavior. These operations can be offloaded to the SmartNIC, but it is better to offload them together for efficiency. Each of the above operations can be offloaded to the SmartNIC, but for efficiency it is better to offload all of these operations together. However, as will be discussed later, a single CPU core is sufficient to handle thousands of connections, so offloading it does not save many CPU cycles. Connection management is the only function that can be offloaded independently as described in AccelTCP. However, it only provides the greatest benefit when flow sizes are small, which may not fit the content delivery workload due to the relatively large overhead of connection management. Therefore, in the present invention, they are all stored on the CPU side.

Data plane operations refer to all operations related to preparing and transmitting data packets that support the implementation of control plane functions. This includes managing data buffers, splitting data into packets, calculating TCP/IP checksums, and more. Because IO-TCP is simple, iterative, and stateless, it offloads data plane operations from the transmission path. IO-TCP also offloads file/disk I/O and couples it to TCP data plane operations. The rationale for this decision is that these operations take up most of the TCP cycles in large file content delivery, thus saving significant CPU cycles. Also, SmartNICs tend to have hardware-based crypto accelerators, which could enable TLS data encryption/decryption at line rate in the near future.

IO-TCP Offload API Functions

First, we will explain how to create an IO-TCP application. Porting an application to IO-TCP requires little modification of the core logic, but it is desirable to have the flexibility to enable what the application wants.

For example, an IO-TCP application must be able to construct content to be transmitted, whether it is file content or not. For this goal, it extends the existing socket API by adding only four functions for offloaded file and network I/O. The function is shown below.

int offload_open(const char *filename, int mode) - opens a file in the NIC and returns a unique file id (fid).
int offload_close(int fid) - closes the file for fid in the NIC.
int offload_fstat(int fid, struct stat* buf) - retrieve the metadata for an opened file, fid.
size_t offload_write(int socket, int fid, off_t offset, size_t length) - sends the data of the given length starting at the offset value read from the file, fid, and returns the number of bytes virtually copied to the send buffer.

IO-TCP provides offload_open() and offload-close() to open and close files on the NIC stack. offload_open() asks the NIC stack to open a file and report the result (success or error). Returns a file ID (instead of a file descriptor) that identifies the open file on the NIC stack for further operation. offload_open() is an asynchronous function whose result must be checked with epoll() since opening a file can fail for various reasons. When all file operations are complete, the application can ask the NIC stack to close the file using offload_close(). IO-TCP also adds offload_fstat() to retrieve metadata about open files (e.g. file size and permission information). With the open file ID, the application can call offload_write() to send the contents of the file as TCP data packets for the connection. Basically, offload_write() does the same operation as sendfile() in Linux, but works with open files on NIC embedded systems. Applications can send custom data (such as HTTP response headers) using existing socket APIs such as write() or send content from multiple files opened by the NIC stack. Currently we only implement plaintext data transfer, but it should be easy to offload cryptographic operations (eg using SSL_offload_write()) to the NIC stack. Figure 5 shows a subset of operations using API functions in an HTTP server context.

In particular, as shown in Fig. 5, the offloading function to the NIC is to offload only the "SEND" function by filling the packet data with the file content, and the ACK packet for this data packet coming from the client bypasses the NIC. It is "directly sent" to the host stack on the CPU side by bypassing it, where it is processed. More specifically, IP TCP offloads only the "SEND" command to the NIC, and all receive-side operations are implemented in the host stack of the CPU. In other words, all packets with ACKs or payloads received from the client bypass the processor of the NIC and are directly delivered to the host stack for processing. Considering that ACK processing and message receiving are much more complex operations, efficiency can be increased by processing them in the CPU side.

According to the present invention adopting this design, the CPU side processes complex parts such as TCP ACK processing, and the SmartNIC can perform only simple repetitive tasks such as data transmission, so overall efficiency is significantly improved. In other words, it is okay to use a processor with low computational performance in the SmartNIC, which promotes the advantage of reducing the manufacturing cost of the SmartNIC.

IO-TCP Host Stack

The biggest problem with the I/O offload TCP stack is that the actual data for transmission can be missing from the host stack. Because file I/O is offloaded to the NIC stack, when data needs to be read from a file, the host stack cannot generate data packets. Likewise, retransmission of data packets becomes impossible because there are no data packets in the host stack.

IO-TCP solves this problem by virtually performing data plane operations on the host stack when data is not available. The IO-TCP host stack keeps track of "missing" data in the sequence number space and does only bookkeeping while delegating the actual I/O operations to the NIC stack.

For example, when an application calls offload_write(), the host stack only updates meta data and virtually writes it to the transmit buffer. This function returns immediately with the number of "virtual" bytes written to the transmit buffer. The host stack then determines how many bytes need to be sent, along with congestion and flow control parameters, and posts a "SEND" command to the NIC stack (see ③ and ④ in Figure 5).

The "SEND" command is performed on the TCP packets destined for the NIC stack (including the NIC's internal MAC address). The command packet's TCP/IP header contains full concatenation information (e.g. 4 concatenation tuples for the next data packet, sequence and sequence and ACK number, etc.) and the payload contains the "SEND" command that is eventually actually replaced. Before the content is sent to the client, the "SEND" command specifies the file ID, starter offset to read, and data length. Using this information, the NIC stack reads the contents of the file and creates and transmits the actual data packet with the header information. Depending on the size of the file contents, one "SEND" command can be converted into multiple MTU-sized data packets. That is, the implementation of the "SEND" command utilizes a general TCP packet from the host stack on the CPU side and transfers it to the NIC stack. The header of this packet uses the same header as that of a general TCP packet, but uses a virtual payload. Have. The virtual payload contents have the file ID, starter offset to read, and data length information of a specific file, and the NIC reads the contents of this file, considering the length, It is converted into n real TCP packets and transmitted. How the "SEND" command is processed is described in more detail below.

Figure 6 shows how the "SEND" command packet is processed. The host stack handles packet retransmission in the same way. Send "SEND" command with file content information for retransmission. The rationale for this design is to make the NIC stack as simple as possible. One obvious alternative is to let the NIC stack handle retransmissions, ensuring reliable delivery of all data for the "SEND" command. The NIC stack must then track all ACKs from the client and run congestion control logic to determine when to retransmit packets.

This can make the NIC stack stateful and more complex, and can be difficult to implement efficiently on some SmartNIC platforms (such as FPGA-based platforms). For all other tasks, IO-TCP behaves like a normal TCP stack. All complex tasks such as connection-specific state and buffer management of the receive path, timer management, reliable data transfer, congestion/flow control and error control are executed in the host stack. Also, for control packets or packets that have data available on the host stack, the host stack creates them and sends them directly to the client, bypassing the NIC stack. All packets coming from the client are also passed directly to the host stack (see ② in Figure 5 and the ACK sent by the client).

This is because delay time overhead occurs as shown in FIG. 7 when passing through the NIC's embedded system, and unnecessary burden is placed on the NIC stack. 7 illustrates round-trip time of Ethernet for communication between a host and a NIC stack. This packet arbitration can easily be done in a detached mode on Mellanox BlueField NICs where the NIC's embedded system has different IP and MAC addresses.

IO-TCP NIC Stack

The IO-TCP NIC stack does all the actual data plane operations for the host stack and handles offloaded file I/O and network I/O for data packet transmission. It works by processing user-specified commands on the host stack, where each command is delivered in a special packet destined for the NIC stack. Currently, four commands are defined: "OPEN", "CLOS", "SEND" and "ACKD". "OPEN" and "CLOS" are used to open or close files in IO-TCP applications. "SEND" is the basic command for sending the contents of a file to the client. Finally, "ACKD" is used to efficiently handle retransmissions without redundant disk accesses. The "SEND" instruction is the core driver for I/O operations. When a "SEND" command is given, the NIC stack checks if the target file is open and reads the contents of the file into a fixed-size memory buffer. The file read offset and length are aligned to NVMe disk page boundaries (e.g. 4KB) and the actual file I/O is executed asynchronously to avoid blocking the main thread. If the file contents are available in the memory buffer, the NIC stack creates a TSO packet with a TCP/IP header in the "SEND" command packet and sends it to the NIC hardware data plane. The NIC hardware data plane handles TCP packet splitting and TCP/IP checksum computation.

In addition, when the host stack on the CPU side receives the ACK packet sent from the client, it notifies the NIC of the ACKed sequence number periodically (eg, in units of 32KB), and accordingly, the NIC sends the ACKed sequence number ( sequence number) need not be maintained in memory. In TCP, if this data packet is lost in the network during data transmission, retransmission is performed again to ensure reliable data transmission. You should be doing it, but ACCD allows you to discard this data.

Challenges with Integrated I/O

Coupling file I/O to the NIC stack's network I/O introduces some unique challenges to the correctness of the TCP stack's operations.

(Retransmit timer and RTT measurement)

TCP relies on delay measurements for decisions such as retransmission timers. However, delays due to disk I/O can confuse RTT measurements. Even with fast NVMe disks, the disk access latency to read a few KB of data is on the order of microseconds, and can take up to milliseconds if I/O requests to the same disk are backlogged.

Early implementations of IO-TCP often retransmit packets even if the original packets were not sent to the client. To solve this problem, when the NIC stack wants to send a data packet for that "SEND" command, it sends an echo packet back to the host stack. The host stack does not start the retransmit timer until it receives the echo packet for the SEND command. For high accuracy, the host stack subtracts the one-way delay of the echo packet (~3.7 microseconds on the platform) from the timeout value on the NIC stack. The CPU overhead for echo packets is small because they are sent per "SEND" command, and normal "SEND" commands translate into tens to hundreds of MTU-sized packets when transferring large files over less congested network paths.

Also, for accurate RTT measurement, the NIC stack reflects the actual "packet processing" delay into the value of the TCP timestamp option. That is, the delay between the "SEND" command arriving at the NIC stack and the corresponding data packet leaving the NIC stack. That is, the "SEND" command packet carries the TCP timestamp options populated by the host stack and the NIC stack updates the values before sending the packet. Since the timestamp option value is in milliseconds and the host stack's time feed is in microseconds, the host stack sends additional time information in microseconds to the NIC stack. The NIC stack can then round the timestamp value if necessary.

In summary, the CPU does not transmit the retransmission timer for a predetermined time until the echo packet is received, and the predetermined time is determined by the packet processing delay time in the NIC and the unidirectional processing time of the echo packet. A packet processing delay time in a NIC may be defined as a delay time between when the "SEND" command arrives at the NIC and a data packet for the "SEND" command departs to the client. That is, the echo packet has a function in which the NIC transfers information about disk IO delay to the host stack in order to accurately measure the RTT of the flow in the host stack.

(handling retransmission)

As described above, retransmission of I/O offloaded packets is also implemented with the "SEND" command. However, simple implementations can waste disk and memory bandwidth by rereading the file contents. To avoid inefficiency, the NIC stack keeps the original data contents in memory until the host stack acknowledges the delivery to the client. When the host stack acknowledges an ACK for an I/O offloaded sequence space range, it periodically informs the NIC stack of the portion passed in the "ACKD" command packet. The NIC stack can then recycle the memory buffer holding the passed data. To minimize overhead, the host stack informs the NIC stack whenever the client sees the last critical amount of data it saw (e.g. 32KB currently used). This buffer memory basically acts as a socket transmit buffer in a regular TCP stack, but the actual memory required is approximately the bandwidth delay product. A 100Gbps NIC with an average RTT of 30ms for a connection might require 375MB of buffer memory.

Handling Errors

In IO-TCP, the host stack handles all TCP-level errors, such as handling packet loss, bad packets, or sudden connection failures. Because the NIC stack only sends packets on behalf of the host stack, and all incoming packets bypass the NIC stack, the host stack can infer TCP-level errors just like any other TCP stack.

However, the NIC stack must report file I/O errors to the host stack. For the "OPEN" command, the NIC stack responds to the host stack whether or not the file open was successful. The host stack then fires an event on that file ID, letting the application learn the result. The host stack caches the metadata of offloaded files, so offload_write() may return an error if an invalid parameter value is passed. If an error occurs in the file read operation itself, it is reported to the host stack with an "error" command packet along with the file ID and error code. Then offload_write() returns 1 with an error code in errno the next time the application calls it.

Implementation

(IO-TCP host stack)

It implements an IO-TCP host stack by modifying mTCP, a high-performance user-level TCP stack. Choose mTCP because its socket API is similar to the Berkeley socket API and supports event-driven programming using epoll.

The host stack includes all mTCP API functions and adds four offload functions (see Table 2) to the application. Each offload function is implemented by exchanging special command packets with the NIC stack. The NIC stack detects command packets by checking a special value in the ToS field of IP packets. Since the "SEND" command packet has a valid TCP/IP header with full connection information, only the payload (with checksum) needs to be swapped with the actual file content before being sent to the client.

In the case of offload_open(), the host stack receives the opened file's metadata (e.g. the output of fstat()) in response from the NIC stack on a successful operation. You can then cache for offload_stat() so the function can handle itself locally. This reduces round trips to the NIC stack. For file operations, both the host and NIC stack share the same file system. The host OS mounts the NVMe disk's filesystem as read/write, and mounts the NIC stack as read-only. One problem is that updates to the host-side filesystem are not automatically propagated to the NIC stack when running on a separate operating system. We currently assume that files do not change during content delivery services, but we may add support for dynamic synchronization of the two filesystems in the future.

In the present invention, in order to efficiently implement the offload API, the TCP header can be reused for communication between the host stack and the NIC. Specifically, if the host stack decides to transmit a data packet after sending "offload_write()", it piggybacks the first part of the TCP/IP header to be sent when sending the "SEND" command and sends it to the NIC. Since it is reused as it is, implementation is very convenient and efficient. Piggybacking is a method of improving transmission efficiency by reducing the number of transmissions of response frames by creating a protocol in which an information frame can be transmitted and a response function can be performed at the same time by properly adjusting and redefining the structure of the information frame. it means.

(IO-TCP NIC Stack)

The NIC stack is implemented as a DPDK application. Its basic operations are to read command packets sent by the host stack, read data from files, create data packets, and send them to the client.

Multiple main threads perform these tasks in parallel, while each thread is anchored to a separate Arm core that operates independently of each other. Command packets from the host stack are distributed to these threads by RSS (receive-side extensions) in the NIC hardware, allowing the sequential delivery of data packets on the same concatenation of different "SEND" command packets. Since the command packets between the host and the NIC stack are localhost communication, they are delivered reliably without changing their order. For efficient memory buffer management, the NIC stack pre-allocates all buffers for file contents at startup. All buffers are of a configurable fixed size. Each main thread owns 1/n of these buffers to avoid lock contention, and a simple user-level memory manager allocates and frees them inexpensively. When a per-thread buffer is exhausted, the memory manager dynamically allocates a buffer. Even with faster NVMe disks, file reading is slower than memory operations, so each main thread uses a few disk reading threads to avoid blocking the main thread. The disk-reading thread uses direct I/O (i.e. open() with the O_DIRECT flag) to bypass inefficiencies in the operating system's filesystem cache and communicates efficiently with the main thread via shared memory. An alternative is to use a user-level disk I/O library such as Intel SPDK. In practice, SPDK reaches its maximum performance with up to 2x fewer Arm processor cycles than direct I/O, but here it sticks to a regular filesystem (i.e. using XFS on Linux) with SPDK's support.

(Porting Lighttpd to IO-TCP)

To evaluate the effectiveness of IO-TCP in real applications, Lighttpd v1.4.32 can be ported to IO-TCP, and the mTCP ported Lighttpd code can be taken from Github and modified to support offloaded I/O operations. Porting to IO-TCP is quite straightforward, as only about 10 of the 41871 lines of Lighttpd code needed to be modified.

Evaluation

The IO-TCP evaluation criteria in the present invention are as follows. (1) whether the new architecture can improve the throughput of the content delivery system; (2) whether CPU cycles are significantly saved; and (3) whether the design and implementation serve the purpose well. Before running the experiment for evaluation, we verified that the accuracy of the IO-TCP stack is maintained with the IO-TCP port Lighttpd server even in the case of high packet loss and multiple simultaneous connections.

Experiment Setup

The experimental setup consists of 1 server and 2 clients. The server system has two Intel Xeon Gold 6142 CPU @2.60GHz (16 cores). Experiments use only one CPU. I used two 25G Mellanox BlueField SmartNICs and four Intel Optane 900P NVMe disks. I connected 1 Smart NIC and 2 NVMe disks. You can set up each NUMA domain and NVMe-oF target offload so that the NIC can directly access 2 NVMe disks in the same domain.

The host CPU runs on Linux 4.14 and the Arm subsystem runs on Linux 4.20. The client systems are each equipped with an Intel E5-2683v4 CPU @2.10GHz (16 cores) and a 100Gbps Mellanox ConnectX-5 NIC. The client's NIC is configured to use Large Receive Offload (LRO). All clients run on Linux 4.14, confirming that the client is not a bottleneck in the experiment.

All NICs connect to 100Gbps Dell EMC Networking Z9100-ON switches. Fill the NVMe disk with 300KB files, the average size of video chunks used in recent work. Make sure the workload is disk bound so that the working set size exceeds the main memory size. Each disk has an advertised read throughput of 2500MB/sec, meaning there is a theoretical limit of 80Gbps when reading from 4 disks.

IO-TCP Throughput with Lighttpd

To evaluate the effectiveness of IO-TCP in the delivery of large file contents. Compare the throughput of IO-TCP-ported Lighttpd with that of the stock version using sendfile() for various numbers of CPU cores.

It uses 1600 concurrent requests for a persistent connection requesting another video chunk of 300 KB. Run the experiment for 5 minutes and report the average throughput. 8 shows the result of evaluating the effect of IO-TCP on large-capacity file content delivery.

For Lighttpd on IO-TCP, it uses only a single CPU core on the host side and uses a full 16 Arm cores on each SmartNIC. This is because a single CPU core is sufficient to handle the control plane operations for all 1600 clients, and using more CPU cores does not improve performance, as the NIC bandwidth limit is approaching.

Each NIC achieves about 22 Gbps, which shows that performance scales with the number of NICs. Conversely, Lighttpd on Linux TCP achieves about 11 Gbps with a single CPU core, 4x smaller than IO-TCP. Performance increases non-linearly with CPU cores, reaching performance similar to IO-TCP port Lighttpd with 6 CPU cores. This clearly shows that IO-TCP effectively offloads I/O operations from the CPU and provides significant benefits in saving CPU cycles.

It also evaluates whether IO-TCP provides bandwidth fairness between competing connections. Jain's IO-TCP fairness index ranges from 0.91 to 0.97 depending on the number of concurrent connections. A similar range (0.90 to 0.97) can be seen for Linux TCP in the same experiment. This proves that there is a significant effect considering the severe disk I/O contention in the experiment.

Analysis on CPU Cycle Saving

We further analyze the savings in CPU cycles by offloading I/O. First, we investigate whether full CPU cores are required to achieve peak performance in the experiments in the previous section. Write a custom program that steals a configured amount of CPU cycles, and locks it to the same CPU core the IO-TCP host stack is running on.

9(a) shows the throughput for a portion of CPU cycles consumed by the host stack. It was confirmed that 70% of single CPU core cycles can achieve 97.3% of peak performance. 80% of the cycle is sufficient for maximum throughput. Even having half the CPU cores is only 23% of the performance loss due to peak throughput. This shows that there is no need to use an entire CPU core to handle control plane operations for 1600 simultaneous connections. Offloading data plane operations to the SmartNIC saves significant CPU cycles.

Figure 9(b) shows the throughput of a single SmartNIC according to the number of Arm cores used. The maximum performance that can be achieved is 22.1 Gbps, which is realized only with 12 Arm cores. Even with just 6 Arm cores, the performance loss is only about 10% of the maximum throughput. This indicates that the prototype's Arm core is not the bottleneck. In practice, the bottleneck for SmartNICs is memory bandwidth, which will be discussed in detail in Section 6.

Evaluation of IO-TCP Design Choices

Here we evaluate the effect of design choices on IO-TCP.

(fix retransmit timer)

First, we evaluate the effect of the echo packet starting the retransmission timer at the right time when the actual data packet is transmitted. Figure 10(a) compares the throughput of Lighttpd with and without timer modification. Without echo packets, many premature timeouts occur on the IO-TCP host stack ("no RTO fix"), so performance remains at 37.9 Gbps. With the timer modification, IO-TCP improves performance by 16.2%. In fact, IO-TCP without proper timer correction produces 97.6 times more timeouts than using timer correction. The performance hit due to premature timeouts would have been much more severe in wide area networks (WANs) with larger end-to-end RTTs.

(RTT measurement correction)

We also compare the impact of TCP timestamp modification on the NIC stack. The average RTT value recorded in the TCP stack every second is measured and displayed in (b) of FIG. 10 . With TCP timestamp modification disabled, the average RTT with 1600 concurrent connections reaches 42-44ms even when the normal RTT is less than 1ms. This is because RTT includes disk access latency that can reach 10 milliseconds due to contention. Enabling timestamp correction reduces the average RTT to 3-5 ms. Although much lower, the RTT is still above 1 ms due to excessive contention on the NIC. A more accurate latency measurement is important to trigger a timeout in the presence of packet loss.

Overhead Evaluation

IO-TCP's split architecture can suffer from the communication overhead between the host and the NIC stack, as well as the low computing capacity of the Arm-based subsystem on the NIC. For this reason, Linux TCP's CPU-only approach performs better than IO-TCP for a small number of simultaneous connections, as the CPU can comfortably handle connections without any overhead.

However, this trend changes as the number of connections increases. 11 shows the results of throughput experiments for different numbers of simultaneous connections. With a single persistent connection, Linux TCP outperforms IO-TCP by a factor of 2 or more. However, maximum performance is reached with a minimum of 4 connections and beyond that the performance remains the same. Conversely, the throughput of IO-TCP increases slowly due to the overhead, but reaches its maximum throughput at 128 connections, while exceeding that of Linux TCP at 8 connections. The performance trend continues even with smaller file sizes.

Figure 12 shows the throughput of serving 4KB files for different numbers of connections. As in the previous case, Linux TCP and IO-TCP reach peak performance at 4 and 128 connections, but the performance is much lower than that shown in Figure 11 due to the increased overhead of file operations. Nonetheless, IO-TCP outperforms Linux TCP at 16+ concurrent connections. To evaluate the latency overhead, we compare the latency of downloading small files using IO-TCP and Linux TCP. IOTCP shows an additional delay of about 200-500us when downloading a file between 1KB and 16KB. This additional latency is unavoidable due to the overhead of the current prototype, but is negligible when delivering content over a wide area network.

The present invention has the potential to be used very promisingly in the future.

(memory bandwidth limit)

The current prototype's performance bottleneck is the BlueField NIC's low memory bandwidth. We verify this by running the same test as in Figure 8 using a 100Gbps BlueField NIC. The NIC stack implementation has two main memory copies: (1) copying from NVMe disk to memory buffer and (2) copying from memory buffer to TSO packet payload buffer. Disabling (1) performance goes up to 36 Gbps, disabling both brings performance to 83 Gbps per NIC. Current BlueField NICs have only 1/6 the memory bandwidth of the Intel Xeon Gold 6142. Nevertheless, the method according to the present invention is promising for the future. First, Arm-based SoCs can be designed with much higher memory bandwidth, and future SmartNICs could benefit from that. For example, Cavium ThunderX2, an Armv8-based SoC server, has a maximum memory bandwidth of 166GB/s, which is much larger than an Intel Xeon CPU. Second, improving the memory bandwidth of SmartNICs is more cost-effective as Arm SoCs are less expensive than server-class x86 CPUs and the overall performance can be easily scaled using multiple NICs.

(encryption supported)

The current prototype does not implement SSL/TLS, but it can be supported by performing cryptographic operations on the NIC stack. At this point, it is important whether the performance of the CPU-based implementation is better or comparable. The Arm subsystem of the NIC platform supports AES-NI, but the AES-GCM performance is only 8.1% to 12.7% of the Intel CPU (Gold 6142) as shown in Table 3 below.

Key Size 64B 256B 1025B 8092B 128bit (Arm) 212 324 365 377 256 bit (Arm) 192 287 321 333 128 bit (CPU) 1,632 3,063 4,649 5,657 256 bit (CPU 1,507 2,550 3,572 4,103

AES-GCM is today the most popular symmetric key cryptography that determines the performance of large file transfers in TLS. Interestingly, running 256-bit keyed AES-GCM with TSO payload on IO-TCP, the current prototype achieves 19 Gbps per NIC (or 38 Gbps for both), losing only 14% of the performance in plaintext transfers. . On the same setup, Lighttpd on the Linux TCP stack only produces 5.5 Gbps with a single core and requires 8 cores to reach 38 Gbps. Since 6 Arm cores are enough to achieve 19 Gbps in plain text transmission (see Fig. 9(b)), and the remaining 10 cores can achieve 19 Gbps by performing AES-GCM, the performance is very reasonable. BlueField-2 has hardware accelerators for AES-GCM, so TLS support on the NIC stack will make more sense in the near future. (QUIC supported)

The present invention can easily extend the design to support other transport layer protocols such as QUIC. To support QUIC in file-based content delivery, you only need to update offload_write() for QUIC while maintaining other API functionality. Internally, each "SEND" command packet must be converted to a QUIC data packet with a header on the NIC stack. Also, since QUIC works over TLS, it must support cryptographic operations.

(other applications supported)

Although the present invention focuses primarily on HTTP-based video streaming applications using IO-TCP, any application that needs to serve large files can benefit from the stack. This includes software downloads, web browsing, file-level synchronization, and more. Also, for small file transfers, the IO-TCP stack still provides benefits as long as there are enough simultaneous connections as described above.

Work related to the present invention will be described.

(PCIe P2P communication)

Enabling PCIe P2P communication between external devices can significantly reduce CPU overhead when transferring data between external devices. NVIDIA GPUDirect RDMA, GPUDirect Async and AMD DirectGMA technologies expose GPU memory directly to PCIe memory space, providing a way for other devices to directly access data on the GPU. EXTOLL proposes to enable direct communication between the Intel Xeon Phi coprocessor (accelerator) and the NIC so that the accelerators can communicate with each other over the network without CPU intervention. Morpheus enables communication between NVMe devices and other PCIe devices. DCS and DCS-ctrl propose a hardware-based framework that enables peer-to-peer communication between various types of external PCIe devices. However, all these peer-to-peer solutions only consider data communication in hardware without considering the kernel stack. As a result, these solutions still suffer from kernel stack overhead when running content delivery applications.

(Accelerated Networking Stack)

There is some existing work to improve the performance of the networking stack. Some work seeks to improve the performance of the existing kernel stack. Fastsocket improves TCP stack performance by achieving table-level connection partitioning, increasing connection locality, and eliminating lock contention. StackMap provides application-specific network interfaces and provides a zero-copy, low-overhead network interface for applications. Megapipe utilizes split lightweight sockets and batches system calls to improve performance. Another approach is to bypass the heavy kernel stack and run the full stack at the user level. mTCP, IX, Sandstorm, and F-Stack utilize user-level packet I/O libraries and leverage multiple CPU cores to process incoming flows concurrently to increase processing throughput and reduce the latency of kernel calls. ZygOS, Shinjuku, and Shenango further improve the tail latency of packet processing by improving workload distribution among CPU cores. Arrakis completely removes kernel involvement in the data path while preserving the kernel's security features. TAS builds TCP fast path as a separate OS service aimed at improving the performance of RPC calls in data centers. Disk | Crypt | Net is building a new video streaming stack that includes a new kernel bypass storage stack and an existing kernel bypass network stack to achieve low latency and high throughput for video streaming applications. However, all these solutions still require a lot of CPU for packet processing, wasting a lot of CPU power when transferring data between external devices. A recent work called AccelTCP offloads TCP connection management and connection relaying to the SmartNIC, offloading some of the packet processing computations from the host CPU core. However, it focuses only on improving throughput for short-lived connections and L7 proxies.

(NIC Offload)

Traditionally, NIC offload methods have ranged from TCP segmentation offload (TSO), large receiver offload (LRO), and TCP/IP checksum offload to stateful systems such as TCP Engine Offload (TOE) and Microsoft Chimney Offload. varied. Stateless offload schemes have become common in modern NICs by effectively reducing CPU cycles for large message transfers. Conversely, stateful schemes have not been very popular due to various security and maintenance issues due to their complexity. Despite potential dependencies on disk I/O, the IO-TCP NIC stack is designed to be stateless to accommodate easy implementation and high performance on a variety of NIC platforms. Recently, there have been several efforts focused on offloading various tasks to SmartNICs to improve the performance of specific applications. AccelTCP improves the performance of TCP by offloading short-lived connection management and L7 proxies to the SmartNIC. KV-Direct leverages FPGA-based SmartNICs to improve the performance of in-memory key-value stores. Floem, ClickNP and UNO leverage SmartNICs to accelerate packet processing for network applications. Metron reduces packet processing latency for network functions by offloading packet tagging to the NIC. iPipe builds a generic framework for offloading decentralized applications to SmartNICs. IO-TCP is obviously the first attempt to leverage SmartNICs to accelerate disk and packet I/O for content delivery systems.

The present invention presents IO-TCP, a split TCP stack design that offloads I/O operations from the CPU for scalable content delivery. IO-TCP provides a new abstraction to perform I/O operations by leveraging the SmartNIC processor, greatly reducing the burden on the CPU and main memory system. In addition, the present invention maintains the simplicity of the NIC stack design so that it can be easily implemented with a low-power processor in an I/O device. According to the evaluation described above, IO-TCP offers significant savings in CPU cycles while providing benefits even for small file transfers when providing sufficient connections.

In summary, the present invention divides the TCP stack implementation into a data plane that processes data and a control plane that oversees all other control functions. It relates to a method of maximizing transmission performance per unit CPU capacity by making the control plan function, which prevents waste and is complex but must be performed quickly, operate in the CPU. Proposes a communication protocol between the host stack, which is a TCP stack executed in the CPU, and the NIC stack executed in the NIC, and defines an API that directs file access in the NIC. By providing it to the host stack, it is designed to control the operation of sending the contents of files read from the NIC in TCP packets. All other control functions (connection management, reliable data transfer, congestion/flow control, error control) are all designed to operate on the host stack. Only a small part of the existing TCP stack was transferred to the NIC stack to make implementation easy, but to maximize the advantage of CPU savings.

This can be implemented in various types of SmartNICs. Utilizing the protocol between the host stack and the NIC stack, the packet RTT measurement in the host stack does not include the disk access delay, so the accuracy is increased. It is designed to periodically inform the NIC stack of the sequence space to release the file contents that have been transferred at the right time. In the performance evaluation of implementing a prototype system and randomly downloading a 300KB file, it showed 50Gbps performance with only one CPU core, which showed the same performance with only 1/6 the CPU capacity of the server using the existing method. .

Even the online video transmission service, which is a part of the content distribution system service, has already exceeded 60% of the total internet traffic, and the amount of traffic continues to increase as non-face-to-face daily life continues due to the corona virus pandemic. are on trend In addition, although the video quality is gradually improving through HD and UHD, there is a problem in that the number of clients that can be simultaneously supported per unit server is relatively reduced due to the burden of increasing bandwidth demand, thereby increasing the video transmission cost. The present invention has the potential to greatly increase the number of clients that can be supported per server by increasing transmission performance per unit CPU capacity by utilizing SmartNIC. Currently, technology development for SmartNICs is actively developing, and the computing performance of SmartNICs is showing a trend of increasing, so it will spread to server platforms in the near future. It has the advantage of being easy to implement because the offloading operation is simple.

The present invention presents IO-TCP, a split TCP stack design that dramatically reduces the CPU burden for online content transmission. Allows you to focus only on function. This division of labor realizes the separation of the control plane and the data plane of the TCP stack, assuming full control of the stack operations on the CPU side, while only the data plane operations are offloaded to the SmartNIC. IO-TCP can saturate two 25Gbps NICs with a single CPU core for disk-bound workloads.

Computational offloading system

The calculation offloading system according to the present invention is implemented as a system for the above calculation offloading method, and includes a central processing unit (CPU) of a file-based content transmission server and a network interface card (NIC) having a calculation function. Since the specific operation method has been described in detail above, only the main operation contents will be briefly described here.

The CPU offloads data plane operations to the NIC, the NIC performs the data plane operations, and the CPU performs control plane operations excluding the offloaded data plane operations. do.

In this case, the data plane operation includes at least one operation of fetching contents, managing data buffers, dividing data packets, calculating TCP/IP checksums, and generating and transmitting data packets.

The control plane operation includes at least one operation of TCP protocol connection management, data transmission stability management, data transmission congestion management, flow control, and error control, and each operation area is divided.

The CPU transfers the “offload_open()” command received from the application program to the NIC. The NIC opens a file on the NIC stack and returns the result to the CPU of the transmission server, and the CPU transmits the “offload_write()” command called from the application program to the NIC.

At this time, if actual data is missing from the host stack of the CPU, the CPU virtually performs the data plane operation.

When the application program calls the “offload_write()” command, the CPU updates only meta data, writes virtual data into the transmission buffer, determines the number of bytes to be transmitted along with data congestion and flow control parameters, and then sends “SEND " command to the NIC stack of that NIC.

Meanwhile, the CPU transmits the “SEND” command using a TCP packet, and the TCP packet includes at least one of file ID, starter offset to read, and data length information, Upon receiving the “SEND” command, the NIC may convert and transmit a plurality of TCP packets based on length information included in the TCP packet.

In addition, when the client tries to transmit a data packet for the “SEND” command, it transmits an echo packet to the CPU, and the CPU does not transmit a retransmission timer for a predetermined time until receiving the echo packet. , The predetermined time is determined by the packet processing delay time in the NIC and the unidirectional processing time of the echo packet, and the packet processing delay time in the NIC is determined by the “SEND” command after the “SEND” command arrives at the NIC. is the delay between the departure of data packets to the client.

The CPU uses the “OPEN” command to open the file of the IO-TCP application, the “SEND” command to send the contents of the file to the client, the “ACKD” command to efficiently process retransmission, and the IO-TCP application’s Defines the "CLOS" command used when closing a file, the CPU offloads only the "SEND" command to the NIC, directly processes packets received from the client, and the CPU If ACK is confirmed, the “ACKD” packet may be periodically transmitted to the NIC. ACPD enables efficient processing of retransmission.

IO-TCP offloads only “SEND” and all receive-side operations are implemented in the host stack of the CPU. That is, all packets with ACKs or payloads coming from the client bypass the processor of the NIC and are directly delivered to the host stack for processing. This is because ACK processing and message receiving are much more complex operations, so it is more efficient to process them in the CPU.

The calculation offloading system according to the present invention implements offloading in which the NIC reads and sends the file to be transmitted instead of the CPU when a network web server such as a video server system transmits a file on a disk through TCP. In this case, the TCP It has technical characteristics of how the implementation should be redesigned. The CPU performs all the functions necessary for performing TCP, but only data input/output is offloaded to the NIC so that simple repetitive and time-consuming tasks are performed by the NIC. And, TCP redesign to enable this is a major feature of the present invention, which dramatically reduces the CPU load for online content transmission compared to the prior art.

The method described above may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. A computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the medium may be those specially designed and configured for the present invention or those known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to act as one or more software modules to perform the operations of the present invention, and vice versa.

Features, structures, effects, etc. described in the embodiments above are included in one embodiment of the present invention, and are not necessarily limited to only one embodiment. Furthermore, the features, structures, and effects illustrated in each embodiment can be combined or modified with respect to other embodiments by those skilled in the art in the field to which the embodiments belong. Therefore, contents related to these combinations and variations should be construed as being included in the scope of the present invention.

Host Stack: Host Stack
NIC Stack: Network Interface Card (NIC) Stack

Claims (20)

A network and file input/output operation offloading method in a CPU (Central Processing Unit) of a file-based content transmission server and a NIC (Network Interface Card) having an operation function,
an offload step of, by the CPU, offloading data plane operations to the NIC; and
A performing step of the NIC performing the data plane operation and the CPU performing a control plane operation excluding the offloaded data plane operation;
The data plane operation includes at least one of content fetching, data buffer management, data packet splitting, TCP/IP checksum calculation, data packet generation and transmission, and the control plane operation includes TCP protocol connection management and data transmission. Including the operation of at least one of stability management, data transmission congestion management, flow control and error control,
In the off-road phase,
transmitting, by the CPU, an “offload_open()” command received from an application program to the NIC;
the NIC opening a file on the NIC stack and returning a result to the CPU; and
and transmitting, by the CPU, an “offload_write()” command called from the application program to the NIC. delete According to claim 1,
and a virtual operation step of, by the CPU, virtually performing the data plane operation when actual data is missing from the host stack of the CPU. According to claim 3,
The virtual operation step,
The CPU, when the application program calls the "offload_write()" command, updating only meta data and virtually writing them in a transmission buffer; and
and posting a "SEND" command to the NIC stack of the NIC after the CPU determines the number of bytes to be transmitted along with congestion and flow control parameters of the data. According to claim 4,
The CPU transmits the "SEND" command using a TCP packet, and the TCP packet includes at least one of file ID, starter offset to read, and data length information. How to offload. According to claim 5,
The NIC receiving the "SEND" command converts and transmits a plurality of TCP packets based on length information included in the TCP packet. According to claim 6,
The client transmits an echo packet to the CPU when attempting to transmit a data packet for the "SEND"command;
The operation offloading method of claim 1 , wherein the CPU does not transmit a retransmission timer for a predetermined time until receiving the echo packet. According to claim 7,
The predetermined time is determined by a packet processing delay time in the NIC and a unidirectional processing time of the echo packet. According to claim 8,
The packet processing delay time in the NIC is a delay time between when the "SEND" command arrives at the NIC and a data packet for the "SEND" command departs to the client. According to claim 1,
The CPU includes the "OPEN" command used to open the file of the IO-TCP application program, the "SEND" command used to send the contents of the file to the client, the "ACKD" command for efficient processing of retransmission, and the IO-TCP application program An operation offloading method that defines the "CLOS" command used to close a file in . According to claim 10,
wherein the CPU offloads only a "SEND" command to the NIC, and directly processes packets received from clients. According to claim 10,
The operation offloading method of claim 1 , wherein the CPU periodically transmits the “ACKD” command to the NIC when confirming an ACK for an offloaded sequence space range. A network and file input/output operation offloading system including a CPU (Central Processing Unit) of a file-based content transmission server and a NIC (Network Interface Card) having an operation function,
The CPU offloads data plane operations to the NIC;
The NIC performs the data plane operation, and the CPU performs a control plane operation excluding the offloaded data plane operation,
The data plane operation includes at least one operation of fetching contents, managing data buffers, dividing data packets, calculating TCP/IP checksums, and generating and transmitting data packets;
The control plane operation includes at least one operation of TCP protocol connection management, data transmission stability management, data transmission congestion management, flow control, and error control;
When the CPU transmits the “offload_open()” command received from the application program to the NIC, the NIC opens a file on the NIC stack and returns a result to the CPU, and the CPU responds with an “offload_write() command called from the application program. )" command to the NIC. delete According to claim 13,
When actual data is missing from the host stack of the CPU, the CPU virtually performs the data plane operation. According to claim 15,
When the application program calls the "offload_write()" command, the CPU updates only the meta data and virtually writes it to the transmission buffer, determines the number of bytes to be transmitted along with the congestion and flow control parameters of the data, and then " A compute offloading system that posts a "SEND" command to the NIC's NIC stack. According to claim 16,
The CPU transmits the "SEND" command using a TCP packet, the TCP packet includes at least one of a file ID, a starter offset to read, and data length information,
The NIC receiving the "SEND" command converts and transmits a plurality of TCP packets based on length information included in the TCP packet. According to claim 17,
The client transmits an echo packet to the CPU when attempting to transmit a data packet for the "SEND"command;
The CPU does not transmit the retransmission timer for a predetermined time until receiving the echo packet,
The predetermined time is determined by a packet processing delay time in the NIC and a unidirectional processing time of the echo packet;
The packet processing delay time in the NIC is a delay time between when the "SEND" command arrives at the NIC and a data packet for the "SEND" command departs to the client. According to claim 13,
The CPU includes the "OPEN" command used to open the file of the IO-TCP application program, the "SEND" command used to send the contents of the file to the client, the "ACKD" command for efficient processing of retransmission, and the IO-TCP application program Defines the "CLOS" command used to close the file of
The CPU offloads only the "SEND" command to the NIC, and directly processes packets received from clients;
When the CPU checks the ACK for the offloaded sequence space range, the CPU periodically transmits the "ACKD" command to the NIC. A computer-readable recording medium recording a program for executing the operation offloading method according to any one of claims 1 and 3 to 12 in a computer.

Images