A Survey on Architectures, Hardware Acceleration and Challenges for In-Network Computing

Two important questions arise when transmitting network packets. The first is how the packets are transmitted from one device to another, and the second is how the packets are handled by each device (how they move through a device). To abstract the problem, the functionality of network devices can be logically divided into control plane and data plane [89].

The control plane is responsible for handling packet traffic transmitted via or generated by the network device. These packet processing policies include functions such as updating headers, forwarding packets, and device management, including the allocation and assignment of resources for the data plane. Since the control plane needs to make more complex decisions, these are usually implemented in software running on GPPs [137] rather than on specialized hardware such as an ASIC. However, since software processing on a GPPs is quite slow compared to a hardware implementation, it is often referred to as the slow path.

The data plane of a device, on the other hand, is responsible for processing or forwarding of packets that are defined by the policies of the control plane. To achieve high throughput and low latency, path switching/routing is usually implemented on an ASIC. However, for Virtual Switches (vSwitch), which are used for communication with Virtual Machines (VMs) and thus enable the simultaneous use of the physical infrastructure, the data plane can also be implemented in software. Although the data level of a vSwitch is slower than the data plane of a physical switch, it is still orders of magnitude faster than the control plane of a device.

In legacy networks and network devices, the control and data plane were tightly coupled. They therefore lacked flexibility and expandability. The introduction of SDN decouples the control and data plane and creates a (locally) centralized control plane through a network controller. The network controller enables functions such as global policy management, event-based triggering, centralization of network information, and other global network management and monitoring applications. According to the Open Network Foundation, the SDN architecture consists of three different layers that are connected to each other via interfaces [45] (Figure 1):

Because a centralized view of the network in the control plane is pursued the network controller was realized on a single machine. However, while this goal can be easily achieved for small networks, for larger networks like in data centers or for IoT-related networks with a large number of diverse actors each of them having different demands, a single centralized network controller becomes fast a bottleneck. Therefore, several approaches target multiple distributed controller in a network and create the global by synchronizing with each other [125]. This (physically distributed but) logically centralized can be realized in different ways. While the simplest approach is to synchronize between all controllers it scales not well. Therefore, approaches like Kandoo have an hierarchical approach. All small transactions are handled individually by each local controller. Larger transactions on the other hand are controlled by a single network controller, which has more computational capacities compared to the local controllers. SDN not only provides us with the capabilities to address high-bandwidth demands by monitoring and controlling the network traffic but also to adapt to changing business needs and a reduction of network management complexity and operations [54]. Researchers and network operators are capable to rethink the idea what networks are, as well as how to design and interact with them.

In legacy network architectures, any network function beyond simple forwarding, such as firewalls, load balancing, and NAT, was tightly coupled to dedicated middlebox hardware. As a result, any change in network functionality, such as an update to a new protocol, is expensive and time-consuming [128]. NFV virtualizes this Physical Network Function (PNF), allowing them to be deployed on commodity servers as software implementations running on VMs, eliminating the need for (expensive) dedicated middlebox hardware. This allows different network services to run on the same equipment, generally reducing network Capital Expenditure and Operation Expense compared to the traditional PNF approach. NFV is complementary to SDN, but interdependent. This means that SDN can be deployed and benefit from NFV and vice versa, but both can be deployed independently [1]. However, NFV has in common with SDN that both technologies aim to decouple control and application software from proprietary hardware to enable a more flexible and dynamic network infrastructure.

As the GPPs architectures used in commodity servers offer a high degree of flexibility when processing data packets, they are widely used in data centers. However, their high flexibility comes at the price of low energy efficiency and latency that is difficult or impossible to predict. They are generally not able to provide the performance of inline packet processing alone. Therefore, they are typically not used as standalone processors for the data plane, but for control plane processing and slow path offloading instead. They are either connected to an accelerator as a standalone processor via a (PCIe) interface or are an integrated component/subsystem of an SoC. An example of a standalone processor together with an accelerator is the Cisco 3550-T. The Cisco 3550-T switch has all of its 48 25G SFP28-ports directly connected to an AMD/Xilinx Virtex UltraScale Plus FPGA with 8 GB High-Bandwidth Memory (HBM) 2 for data plane processing. The FPGA is connected via a PCIe Gen3 x8 interface to an Intel Atom processor with 8 cores and 16 GB DDR4 for control plane processing.

While the advantage of such a design is that all components are already available and therefore no custom chip design is required, the disadvantage is that the data transfer between accelerator and processor is limited by the interface (e.g., about 63 Gb/s data transfer without protocol overhead for PCIe Gen3 x8) and each transfer has a communication overhead that increases latency. Therefore, GPPs (with the exception of slow-path offloading) are not used for data plane processing on these devices. The trend is to use GP cores tightly coupled with dedicated (network) accelerators on a single SoC. Typical processor cores for this type of device are ARM or MIPS based. Further details on such designs are discussed in Section 8.4.

An ASIC is a chip that is specialized and optimized for a specific task. Hardware developers focus only on implementing the minimum number of operations required to accomplish the task at hand. ASIC-based designs have the advantage of being able to perform their task with order of magnitudes better energy-to-performance ratio than a GPP [83], but they lack any flexibility. Since they also cannot be updated or fixed once they are manufactured and shipped, they have long and costly development cycles to ensure the correctness of the design.

Before the introduction of SDN and NFV, ASIC-based network devices with functionality tightly coupled to hardware were the state of the art—not only for power, but also for performance reasons. Earlier GPPs were simply not capable of processing packets at line speed, and in many cases still are not. As a result, many ASIC-based network chips use GPPs only for control plane functionality or to handle unusual/special cases of data plane packet processing (slow path offloading), while fast packet processing on the data plane packet is still implemented with ASICs. However, some ASIC-based network chips, such as the Intel Tofino (formerly Barefoot Tofino) [34], do offer some flexibility. These so-called programmable ASICs provide a basic level of flexibility by using MAT pipelines inspired by the RISC architecture. In addition, such designs are often equipped with programmable parsers and deparsers to enable protocol-independent packet processing according to the PISA architecture design. This allows network operators to define and use their own protocol, or switch to a different protocol if required, without having to replace expensive hardware, unlike fixed-function network chips.

FPGAs are reconfigurable devices consisting of Configurable Logic Blocks connected by a Switch Matrix. In general, any digital circuit can be designed on an FPGA, as long as enough resources are available. The downside of this flexibility is that FPGAs are larger and the achievable frequency is typically lower compared to ASICs. To at least partially address these issues, FPGAs typically also contain slices of specialized components such as Digital Signal Processing units and Block-RAM (BRAM).

Before P4, Xilinx had already targeted network programmability for their FPGAs. They proposed their own packet processing language called PX [24] for SDNet. SDNet allows the import of custom IP cores implemented in VHDL and Verilog by defining the interface to the user engine [23]. In addition, a C behavioral description must be provided for simulation. However, with the increasing popularity of P4, a \(P4_{16}\) compiler has been introduced into the SDNet environment. The Xilinx adaptation of the P4 compiler generates a JSON file for the runtime control software and the PX program, which is compiled as before by the SDNet compiler, which generates the Hardware Description Language (HDL) module [70]. In addition, a SystemVerilog testbench is generated. With the P4 \(\rightarrow\) NetFPGA design flow the P4-SDNet compiler was integrated by the NetFGPA project [70] to provide easier programmability of their NetFPGA-SUME boards without the need for HDL knowledge. NetFPGA PLUS [7] introduced a codebase using parts of the AMD/Xilinx OpenNIC project [2] to support AMD/Xilinx Alveo 200, 250, and 280 boards. With the introduction of Vitis Networking P4 (VNP4) [53], AMD/Xilinx provides direct P4 integration, generating SystemVerilog files for simulation and synthesis from a P4 description using the P4C-vitisnet compiler.

Since P4 is not a Turing-complete language, it is not possible to express many constructs required for FPGA data plane programming. However, since the introduction of extern in \(P4_{16}\), it is possible to interact with objects and functions provided by the architecture. This means it is possible to configure the FPGA and connect functions exposed to the data plane. Nevertheless, many features of modern FPGAs, such as Dynamic Partial Reconfiguration (DPR), cannot yet be used. Also, the incomplete standard for SmartNICs in P4 is an existing problem related to portability. The OpenNIC project [2] only partially addresses these problems. First, as the authors state, the OpenNIC project is not a full-featured SmartNIC solution. Second, the project originated as a research project within Xilinx Labs and targets only Xilinx FPGAs.

Research to provide programmability for the data plane has made significant progress. However, many features are still missing or proprietary. Also, programming FPGAs for network processing is still challenging and still requires sometimes lot of manual work and expert knowledge. For example, while it is possible to use HLS with OpenNIC, the RTL wrapper still has to be created manually because the interfaces generated by HLS are not directly compatible with the interfaces expected by OpenNIC.

SoC is a collective term for any design that realizes a system of multiple heterogeneous components interconnected on a single chip to solve specific tasks. Over time, SoC architectures have been proposed and realized for many domains, such as mobile processors with integrated graphics units, integrated tensor units for ML, or NPUs for network processing. In this section, we will discuss several SoC solutions used for network processing. Since in our opinion the term SoC is too broad to describe the different architectural approaches, we use the following categorization for SoC on programmable network devices:

All three SoC categories can be equipped with hard implemented accelerators, e.g., for cryptography.

The industry often uses other terms such as NPU, DPU, and Infrastructure Processing Unit (IPU). According to the SmartNIC Summit definition [29] and similar definitions from NVIDIA [31], a DPU should have the following characteristics:

While this definition generally fits the MuSoC well, we can fit most industry proposed DPUs into this category. However, other proposals, such as the DPU on the AMD/Pensando DSC, fit better into the MaSoC category in our opinion.

According to the definition provided by the SmartNIC Summit [29] and Intel, we see a strong overlap between DPU and IPU. The only difference is that the DPU definition emphasizes that industry-standard CPUs are used, while the IPU definition focuses more on the fact that the accelerators are hardened and tightly coupled to the dedicated programmable cores. Looking at the devices offered by Intel under the umbrella of this term, we can see that it was mainly defined in the context of the Mount Evans architecture [112]. However, other products offered by Intel under this term do not fit this definition. Intel also uses this term for the F2000X-PL (Intel Agilex 7 F) and C5000X-PL (Stratix 10 DX) boards. Both boards are SmartNICs with an Intel Xeon-D connected via an on-board PCIe interface to an FPGA-based SoC with a 64-bit quad-core Arm Cortex-A53. This contradicts the definition of an IPU, which requires the accelerators to be hardened. Table 1 shows the classification for some upcoming and already available SoC based on our categorization.

To the best of our knowledge, CGRAs in the context of programmable network devices have not yet been realized as own chip design and are not widely discussed in this context. The only exception is [113], which proposed a CGRA-based prototype on an FPGA for the data plane of programmable switches (details are discussed in Section 9.3). Nevertheless, we want to discuss this architecture, which has gained recognition as an alternative design for reconfigurable hardware compared to FPGAs for some domains, such as ML.

Research on CGRAs has been ongoing for about 30 years, dating back to the early 1990s [48, 66]. Unlike the fine-grained architecture of FPGAs, CGRAs are, as the name implies, a coarse-grained architecture. The higher granularity allows the construction of optimized Processing Elements (PEs) that require less area and power and can operate at a higher clock frequency. These PEs (also called Reconfigurable Cells or Functional Units (FUs) depending on the literature) are the smallest reconfigurable unit of a CGRA and are interconnected by programmable switches.

The higher degree of granularity allows reconfiguration times at a range of \(ns\) to \({\mu}\textrm{s}\) compared to FPGAs with reconfiguration times in the range of \(ms\) to \(s\). Therefore, temporal computation can be better exploited with CGRAs than with FPGAs [83]. However, the higher degree of granularity is accompanied by a loss of flexibility. Both CGRAs and FPGAs belong to the data-flow and configuration-driven architectures in the context of their execution model. Because CGRAs are less flexible, they are also considered part of a Domain Specific Accelerator classification [83], while FPGAs provide general flexibility. Even though CGRAs seem to be a good compromise between ASICs with fixed functionality and little to no flexibility and FPGAs with a high degree of flexibility, it is still an immature architecture that has to deal with a variety of problems. First, there are no high-level compilers that are well optimized for this architecture. The gap between manually and compiler optimized code is large. Liu et al. [83] suggest that it would be more efficient to use a lower-level programming model that exposes more architectural details to the programmer rather than relying on compiler optimization. However, this approach is not practical for many real-world (commercial) scenarios where time-to-market and low-cost development are more important than achieving the highest possible performance for the architecture. Therefore, we believe it is more important to develop compilers that are able to leverage the architectural features of CGRAs, for example, based on the Multi-Level Intermediate Representation (MLIR) compiler infrastructure [80] and providing libraries with optimized components, which is also discussed in [115].

Another approach that tries to take advantage of the fast reconfigurability of CGRAs is to use CGRA-like overlays for FPGAs. Jain et al. [72] proposed a coarse-grained overlay for JIT compilation on FPGAs by using the Clang compiler to transform OpenCL kernels into an LLVM IR to extract the Data Flow Graph (DFG). The nodes of the DFG are then mapped to the FUs of the overlay. This is used for FU netlist generation and Place and Route (PnR) on the overlay. Their proposed design improved PnR time by orders of magnitude, from several hundred seconds when using the FPGA fabric to a fraction of a second when using the overlay on a workstation. Even performing PnR on the ARM Cortex-A9 of the Zynq 7020 SoC took less than a second in most cases.

Zamacola et al. [130, 131] proposed a multi-grain overlay for FPGAs based on the CGRA-ME framework [49]. Their proposal builds on their previous work called IMPRESS [129], which supports multi-grain granularity for Xilinx 7 series FPGAs using DPR. In [130], they extend their work by integrating a modified version of CGRA-ME for their mapping tool. Like [72], they use the Clang compiler to transform the code into an LLVM IR to extract the DFG, which is then mapped onto the overlay. However, while [72] uses only a coarse-grained overlay, [130] supports two levels of granularity: a fine-grained level for configuring the FUs and a medium-grained overlay for assembling (or stitching, as they call it) the FUs.

As discussed in Section 5, intermediate nodes have limited computational capabilities. Therefore, they require hardware accelerators to provide the performance needed to offload applications to the network. While programmable ASICs like the Tofino can provide a high degree of performance, they lack the flexibility to meet the needs of many applications. Building a custom ASIC is time-consuming and costly, and can only provide a limited number of accelerators that must be clearly defined in advance, which is not feasible in most real-world scenarios. While highly flexible reconfigurability is one of the main strengths of FPGAs when it comes to creating your own custom design without going directly to an ASIC, it comes (in addition to high design times) with the disadvantage of high reconfiguration time (\(ms-s\)). CGRAs, on the other hand, have a reconfiguration time that is orders of magnitude (\(ns-{\mu}\textrm{s}\)) lower than FPGAs [83].

In order to provide acceleration for a wide range of applications in a dynamic network environment, we believe that fast accelerator exchange will be mandatory for future networks utilizing INC. Therefore, we believe that more research in the direction of implementing CGRAs as an actual integrated architecture for network devices, as well as a coarse-grained or multi-level grained overlay for FPGA-based network devices using technologies such as JITs, could be beneficial for accelerating algorithms in the network.

Lin et al. [81] present with PANIC a multi-tenancy SmartNIC design implemented on a ADM-PCIE-9V3 board equipped with a AMD/Xilinx Virtex UltraScale+ (XCVU3P-2) FPGA.

The incoming packet is received by a single-stage RMT module that is configured at design time. This means that, unlike the RMT stages of the PISA, it cannot be changed online. The module matches the IP address and port fields and maps them to a descriptor for further processing. The packet is then stored in the PB and the created descriptor is written to the scheduler’s Push-In First-Out (PIFO)-based priority queue [106].

The Central Scheduler supports both pull-based scheduling and push-based scheduling with chaining. The system also uses a credit-based system to keep track of the current workload of each CU. Depending on the workload, the scheduler operates in pull mode (high workload) or push mode (low workload). The push and pull mode used here uses the same concept established for push and pull mode in producer-consumer designs, such as those used in cloud messaging services. In pull mode, CUs actively request packets and must wait until the request is complete before they can start working. In push mode, the scheduler actively pushes packets to the CUs. Using chaining, a CU can push a packet directly to the next CU in the chain after processing if that CU’s buffer is not full, otherwise the packet is written back to the PB.

A CU can be a hardware accelerator or a processor core. The authors used two hardware accelerators (AES and SHA3) and a RISC-V softcore for their prototype. Both implemented hardware accelerators are based on IPs provided by OpenCores [13, 14]. For the softcore, the open source RISC-V core generator VexRiscv [26] was used. All components of PANIC are connected via a Crossbar build with the open source network-on-chip router RTL [40].

All components are implemented in Verilog and can run at a frequency of 250 MHz. The only exception is the PIFO which runs at 125 MHz. The reason for the lower frequency for the PIFO is, according to the authors, the poor scalability of the PIFO on FPGA. For the PIFO they could not use the BRAM of the FPGA and had to rely on Lookup Table (LUT) only, which has a strong impact on the available area utilization. With a data width of 512 bits, the crossbar can support a throughput of up to 128 Gbps per port, exceeding the 100 Gbps of the MAC. To transfer packets to the host memory, PANIC uses the DMA engine provided by Corundum [58]. They also use the Corundum NIC driver on the host side.

SuperNIC, proposed by Lin et al. [82], shares similarities with PANIC [81]. Both investigate task offloading for multi-tenants to CUs on a SmartNIC. In both designs, CUs are interconnected with each other and a credit-based central scheduler over a crossbar, and they support task chaining between CUs. However, there are several key differences.³

First, PANIC focused on prototyping a SmartNICs on FPGA with a general fixed architecture, potentially implementable as an ASIC. In contrast, SuperNIC consists of three components: (a) FPGA logic for user-offloaded network computation; (b) ASIC logic for fixed system tasks (packet transmission, scheduling, and reception); and (c) GP cores for executing control plane tasks in software. While the latter two components are common in many SmartNICs, Lin et al. [82] primarily focused on offloading network computation to the FPGA logic, in the form of NTs described by a Directed Acyclic Graph (DAG), including the reconfigurable nature of FPGAs in their consideration.

The second major difference relates to a key concept in SuperNIC’s design: the abstraction of NTs into Virtual NT chains. While PANIC provides a single CU for each task, leading to limited scalability as the number of tasks grows, SuperNIC maps a chain consisting of multiple connected tasks (physical NT chain) to a single reconfigurable CU, called NT region. A virtual chain is a subset of tasks from a DAG. However, while a virtual chain can contain tasks available in a CU others may not be part of it. To avoid frequent reconfiguration or overly fine granularity (which would lead back to PANIC’s scalability issues), the authors propose the concept of bypassing or “skipping” tasks in a chain. To support this mechanism, each task in a CU is augmented with a wrapper.

For better performance and utilization, tenants can also share a CU, i.e., if a task in a CU is not needed by one tenant, it can be used by another tenant at the same time. In addition, SuperNIC supports execution of multiple virtual chains belonging to a single DAG (DAG Parallelism) in parallel to reduce execution time and replication of CUs (Instance Parallelism) to increase throughput.

To ensure fairness, they provide the concepts of space sharing and time sharing. For space sharing, they determine the number of instances of an NT chain to start and the amount of onboard memory to allocate to each user, using the DRF approach [62]. For time sharing, the allocated resources from the space sharing step are considered fixed, and a virtual start and end time is assigned to each packet, allowing time sharing of CUs, ingress and egress bandwidth, and buffers.

Unlike NICs, switches are connected to a much larger number of network nodes, making them ideal targets for in-network application acceleration. However, adding compute logic can impact not only individual nodes but also overall network performance for traffic routed through the switch. Additionally, switches typically manage a larger number of tenants compared to NICs.

Stoyanov and Zilberman proposed Multi-Tenant Portable Switch Architecture (MTPSA) [109], an extension for the P4 PSA with OS-like roles and privileges for security isolation. The pipelines generally follow the PSA definition. However, the ingress and egress pipeline belongs to the superuser (typically the network operator), and the egress pipeline is extended by an encapsulated user (sub-)pipeline between parser and MAT stages of the superuser egress pipeline. Superuser header vector and metadata bypass the user pipeline. Encapsulated user packets (e.g., using VXLAN) are decapsulated by the superuser pipeline and processed by the user pipeline. This allows opaque processing of user programs without interference or knowledge by or from the superuser pipeline or other user programs. They extended P4C with a back-end compiler for BMv2 SW-target to generate superuser and user pipelines separately. Due to SDNet limitations, they adapted the design for NetFPGA, placing the user pipeline before the superuser egress pipeline.

Saquetti et al. [102] proposed an SoC consisting of ASIC and FPGA logic components based on their prior work P4VBox [101] for data plane virtualization. The core of their proposed architecture is an array of reconfigurable vSwitch realized on the FPGA logic. The input and output queues, buffers, interfaces, and vSwitch queues are implemented as ASIC. Additionally, the SoC provides a management interface to an external off-chip controller. In their evaluation, they achieved up to \(3.2\) Tbps for the ASIC implementation and \(129.61-132.63\) Gbps for a single vSwitch instance. However, they were constrained by the limited on-chip BRAM of the XCVU13P FPGA. This limitation allowed them to saturate the maximum bandwidth of \(3.2\) Tbps only for the vSwitch implementation of the L2-Switch. For the other evaluated use cases (Router, Firewall, INT), they achieved approximately \(1.43-2.23\) Tbps (about 45–70% of the maximum bandwidth) only.

Discussion. In most real-world scenarios, serving multiple tenants is necessary. Given that resources on SmartNICs or switches are much more limited compared to servers, it is crucial to investigate how to implement accelerators for specific tasks or domains. Additionally, it is important to ensure a high degree of fine-granularity and reusability in their design.

However, incorporating dedicated accelerators in a design increases the scheduling complexity when mapping tasks efficiently in a dynamic, multi-tenant environment. Therefore, it is necessary to investigate intelligent scheduling and placement methods, as demonstrated in [81] and [82]. The approach in [82] offers some advantages over [81], as it can map multiple tasks into a CU, potentially allowing better utilization of resources of a Pipeline Register (PR) and requiring less interaction with the scheduler. However, the authors of [82] currently only support manual mapping of physical chains to a CU, leaving automatic mapping as a topic for future research.

Works like PsPIN [53] and Flare [52] (based on [53]) have investigated the implementation of RISC-V-based MaSoC for SmartNICs and switches, respectively. These designs provide advantages such as flexibility, high performance through a high degree of parallelism, and easy scheduling due to the homogeneous design of the PEs (in contrast to [81, 82]). However, the scalability of such designs may be limited as the number of clusters and PEs in a cluster increases, due to interconnection complexity and limited access to shared memory. To address these limitations, such designs could be complemented by fine-granular hardware accelerators that can be shared within and/or between clusters. This approach, however, would potentially result in increased scheduling complexity again.

While both [102] and [82] discuss PR, [102] also includes a simple model regarding the effect of PR, during the initial (re-)configuration time of the CUs. PR provides the flexibility to change specific regions/CUs, but this typically comes at the cost of increased resource utilization, limiting the design space for (global) optimizations and potentially leaving partitions underutilized. Furthermore, it can lead to higher latencies when elements interact between partitions [102]. To address this, techniques like task chaining of offload to a single Reconfigurable Partition (RP), like investigated in [82], could provide an approach for better utilization of the RP. However, only a limited number of works, such as [82, 90, 102] have considered or investigated RP for SDN and INC. Furthermore, technologies like Nested Dynamic Function eXchange [32] still remain to be investigated in this context.

Brunella et al. [46] presented hXDP, a solution for accelerating eBPF/XDP programs on a NetFPGA. They implemented an IP consisting of a Programmable Input Queue and an Output Queue connected to a finite state machine called Active Packet Selector (APS) containing a read/packet and write buffer allowing parallel read and write. The APS is connected via a data bus to an implementation of the extended Instruction Set Architecture (ISA) [4] on a custom 4-stage, 4x superscalar soft-core Very Long Instruction Word (VLIW) processor on the FPGA. To take advantage of the data parallelism inherent in some functions, the VLIW processor is connected via an additional bus to the helper function module used to implement the accelerators. The interface of the helper function module follows the eBPF definition. The accelerators can read data directly from the registers containing the arguments of the function call (R1–R5) and write the results back to the return value register (R0). To support the functionality of eBPF maps⁴ in hardware, a configurator for the shared maps memory region on the FPGA was implemented, which creates the maps at program load time. To support direct access to individual map entries, the maps memory, like the APS, is connected directly to the VLIW processor via the data bus.

In order to run unmodified eBPF code, they provide a compiler that translates the compiled eBPF bytecode to the design’s hXDP ISA. The custom compiler for their hardware target optimizes the eBPF program by removing unnecessary instructions for the FPGA target, such as boundary checks and zeroing of memory areas for variable initialization. They also changed the ISA from a two-operand machine to a three-operand machine and added support for 6-byte alignment⁵ for load store instructions to fit the field size of the MAC address in a packet. Since the implementation of the VLIW processor on the FPGA is much simpler from a design perspective than that of a modern x86 CPU that supports, for example, out-of-order execution with branch prediction and speculative execution, they use static compiler analysis for instruction-level parallelism and scheduling of branches to different lanes for parallel execution, instead.

Their work showed that it is reasonable to offload XDP programs to a NIC. Not only can this lead to better performance for networking applications while being more power efficient, but it can also free up CPU cores from specialized networking tasks that they weren’t designed for and use them for other more general tasks instead. The latest version of hXDP has been integrated into Corundum [58] and now targets a Xilinx Alveo U50 [42].

A follow-up work by Bonola et al. [42] extended the in Corundum integrated design of hXDP [46] by a fused parser-MAT pipeline, called Warp Engine (WE). The idea is that simple parts of an eBPF/XDP program, such as header parsing, can be offloaded to simple MATs. In addition to the WE, they extend the hXDP compiler flow by adding their own custom compiler, called the Warp Optimizer, in front of this flow. The Warp Optimizer first reads the bytecode of the program and creates a Control Flow Graph (CFG) to analyze which parts can be offloaded. For offloading to the WE, the compiler, similar to a P4 compiler, creates match-action rules to configure the MAT. The WE also provides a feature called context restoration. This means that the WE is capable in case it cannot run the whole program it will execute only parts of it and create a correct initial state for hXDP which will then process the rest. Also, the WE is able to process the next packet while hXDP is still busy with the current packet. This ensures that the WE does not become the bottleneck in the system. Tasks such as simple packet forwarding can be handled without the hXDP components being involved at all.

In general, the concept behind this idea is the same as fast-path processing and slow-path offloading as already known from PISA architectures. The difference here is that while slow-path offloading usually uses GPP, the hXDP architecture can also provide accelerators for complex rules.

An alternative approach to [46] and [42] is eHDL [97]. Both [46] and [42] provide a predefined architecture that is mapped/prototyped on an FPGA and the eBPF bytecode is compiled for this fixed target architecture. On the other hand, eHDL [97] also takes the eBPF bytecode as input, but generates a hardware pipeline from it instead. They integrate the generated eBPF hardware pipeline into the Corundum NIC. Like the hXDP compiler, eHDL merges the two-operand eBPF instructions into three-operand instructions, and like in [42, 46] a CFG and Data Dependency Graph of the program are generated. eHDL provides templates for the hardware primitives and the instructions to them. To support (conditional) jump instructions, the pipeline creates disable signals to disable the operation for the following stages by simply forwarding the packet to the following stages until the offset (in pipeline stages) from the current stage is reached, while hXDP schedules all dependent instructions on the same lane to avoid data hazards and uses data forwarding per lane to avoid stalling the pipeline for these dependent instructions. For possible Write-after-Read data hazards, the design generates delay registers that work on a FIFO principle. In the case of Read-After-Write (RAW) data hazards, they distinguish between two cases, per-flow and global state cases. For the per-flow cases, they insert an additional block that contains the addresses of read operations and compares them with the address of a write operation. If at least one of the addresses matches, the pipeline is flushed. To avoid repeated write operations to earlier maps in case the RAW data hazard is detected at a later stage leading to a wrong system state, so-called elastic buffers are added between the pipeline stages. This allows only the pipeline stages starting with the hazard detection to be flushed, so that earlier stages do not have to be repeated. While this can theoretically improve throughput compared to a full pipeline flush, the authors’ evaluation showed that the actual probability of flushes caused by per-flow states and the resulting degradation of throughput is low. In the case of frequently used global states such as packet counters, eBPF provides an atomic operation to avoid concurrent access to the same map. To realize the same behavior in hardware, eHDL provides a block that performs in-place Key-Value (KV)-based lookups.

Discussion. The advantage of the approach presented in [97] is that it utilizes only the hardware components required for the given instructions. This avoids the overhead of instruction fetch and decode phases like in [42, 46] or the need for hardware to execute/accelerate instructions that might not be used in some cases. Both eHDL and SDNet implementations achieved line-rate performance compared to hXDP and BlueField-2. For the tested use cases, they delivered up to about \(164.4\) times higher throughput compared to hXDP and up to about \(23.4\) times higher throughput compared to BlueField-2. This demonstrates that dedicated hardware pipelines for packet processing can achieve significantly higher and more consistent performance compared to GP/MuSoC-based architectures, while being more power efficient.⁶ Additionally, eHDL requires less than \(20\%\) of the resources available on the FPGA (Figure 5), leaving room for scalability or providing multiple services simultaneously. However, a disadvantage exists: Adapting the architecture to a new program requires synthesis and FPGA reconfiguration. The authors of [97] acknowledge this and consider using DPR for future work. This would allow adding, replacing, or removing hardware pipelines at runtime depending on the specific services needed and/or the required throughput. While DPR might solve the downtime issue during reconfiguration, the synthesis process itself remains a significant overhead and is not suitable for fast redeployment. A potential solution to this problem, as discussed in Section 8.5, could be the use of overlays.

In the following we selected work conducting research on the domain of Domain Name Server (DNS), CPs, caching, ML, and security. For each domain we will first introduce the domain, present challenges, and discuss proposed works that address (some) of these challenges.

DNS. The DNS service is one of the most essential services on the Internet. It defines the mapping of human-readable domain names to machine-readable IP addresses. However, as the demand for scalability of network services such as DNS to handle an increasing number of queries while providing low latency increases, software-only solutions will no longer be sufficient.

Kiwi [104] is a HLS system to generate an RTL (Verilog) output from C# code that can be used to program FPGAs. Emu [111] is a standard library that supports the implementation of network functions with Kiwi. Part of the Emu library is to provide a simple DNS server (Emu DNS in Table 3) that can handle non-recursive queries and query resolution with names of at most 26 bytes for IPv4 addresses. However, according to the authors, these restrictions can be relaxed to handle longer names and IPv6 addresses.

Another approach to implementing a DNS service on FPGAs is P4DNS [124]. Unlike Emu, P4DNS is implemented in P4 using NetFPGA. P4DNS provides a simple low-level name server that supports recursive queries. The implementation uses features from both the data and control planes to achieve high performance. The data plane implementation is based on the RMT pipeline with some modifications responsible for the performance critical parts of the design.

Discussion. When investigating basic network services such as DNS with modern network processing models/languages such as PISA/P4, problems became apparent. First, the generality of the FSM for the parser produced by P4C would lead to increased compile time and resource usage because it would produce even hard the headers (Ethernet, IPv4, UDP, and DNS) that could be compressed into a single state. In addition, the P4C does not support variable-length fields, but DNS headers can have different lengths, the parser implementation supports multiple lengths, each with its own state [124]. Although P4 provides a vendor-independent model, the restrictions and limitations create strong hurdles that may make implementation for INC inadequate or even impossible. Therefore, further research is needed in the direction of providing/improving high-level abstraction models and (automatic) tool flows capable of efficiently translating and mapping such a model to reconfigurable hardware. In order to realize such tool flows, the investigation of improved HLS compilers for C# [104], C or C++ (e.g., from AMD/Xilinx, Intel), automatic integration on FPGA equipped SmartNIC and switches, and libraries such as Emu [111] will be of importance to ease the hurdles for implementations of reconfigurable hardware accelerators targeting INC.

CP. A fundamental problem in distributed computing is how to ensure the reliability of distributed systems while processes may be faulty. Therefore, the goal of CPs is to find consensus among processes about states or data needed for computation. The types of process failures in a distributed system can be divided into two types: Crash failures and Byzantine failures. Crash failures describe scenarios in which a process abnormally terminates its execution and cannot be resumed. Byzantine failures, on the other hand, describe arbitrary process failures. These can include cases such as crashes, incorrect states, failure to send and receive messages, sending messages with incorrect or even malicious content, and so on. Byzantine failures can disrupt other processes or cause failures that are unintentional or caused by a malicious attacker. However, they cannot control the network. Any process can uniquely identify the sender of a message [61]. Crash failures are handled by Crash Fault Tolerance CPs, and Byzantine failures are handled by Byzantine Fault Tolerant (BFT) CPs. However, to meet the demands for scalability, high bandwidth, and low latency, BFT CPs implemented in software would become a bottleneck.

Sit et al. [105] implemented the Practical Byzantine Fault Tolerance (PBFT) algorithm [47] to study different configurations of it. In their work, they show that it is unlikely that networks with more than 10 Gbps can be saturated without the use of hardware accelerators. Therefore, they investigated offloading the processing to SmartNICs and standalone FPGAs.

Discussion. One major challenge they identified for offloading PBFT algorithms is that the iterative computation of cryptographic algorithms such as RSA is difficult to realize on FPGAs because of low frequency and would account for a large portion of the logic utilization for a PBFT implementation. Nevertheless, their work shows that with SmartNICs, fine-grained batching with strict latency guarantees is a reasonable option for acceleration, and that mid-range FPGAs such as the Xilinx Virtex-7 690T are already a good target platform for PBFT implementations. In general, FPGAs perform best for high-parallel algorithms, but are not the best choice for iterative algorithms where higher frequencies are beneficial. However, we are seeing a trend where major FPGA manufacturers are offering PS tightly coupled to the FPGA’s PL on a single SoC that can also be used to implement the iterative parts of an algorithm, providing the benefits of both worlds.

Intrusion Detection and Prevention Systems(IDS/IPS). IDS/IPS are critical components of the network infrastructure to protect systems from attacks. Both systems scan network packets and compare their contents to a database of known threats. However, an IDS is only responsible for detection and monitoring and does not intervene on its own, while IPSs accept or reject packets based on a set of rules. Network operators are faced with the challenge of securing networks with hundreds of thousands of concurrent connections with tens of thousands of rules to be checked by IDS/IPS. To ensure future scalability, software-only solutions are insufficient and PISA designs such as the Tofino switch cannot be used to implement a full IDS/IPS.

Therefore, Zhao et al. proposed an FPGA-based solution for SmartNICs, called Pigasus [132]. In general, the concept of IDS/IPS is based on solving pattern matching problems (header matches, string matches, and regular expressions [132]). The proposed design consists of an FPGA connected via Ethernet and CPUs connected via PCIe. The parser, reassembler, and Multi-String Pattern Matcher (MSPM) (and its extensions Non-Fast/Fast Pattern String Matching) are implemented on the FPGA. The Regular Expression and Full Match stages are implemented on the CPUs. However, they only interact with about 5% of the packets. A core component of the design is the reassembler, which takes the forwarded packets from the parser, sorts the TCP packets, and records the last bytes of the previous packets to allow contiguous and cross-packet searches for the MSPM. To fit the Snort register ruleset [25, 98] into the available BRAM of the Intel Stratix 10 MX FPGA, the proposed design uses Hyperscan-inspired hash table lookups [123] for the MSPM instead of state machines for exact matching. For the full matcher on the CPU side, an adapted version of Snort 3 is used to receive Protocol Data Units and rule IDs. With their FPGA-based SmartNIC design combined with a CPU system, they were able to develop a single-server solution for IDS/IPS.

Discussion. The authors showed that it is generally possible to implement a complete IDS/IPS on a single FPGA. Only the full match stage was still running on a CPU because it is not the bottleneck for most packages and an implementation would have required too much BRAM (\(\sim\)24 MB) for only about 5 Gbps of traffic. Also related to the limited amount of on-chip memory, the authors faced the problem that there is a design tradeoff between the number of rules, concurrent flows, and the number of pipeline replications. While they could achieve 200 Gbps throughput with two pipelines, they were quickly limited by the number of rules they could store in the BRAM. However, since newer FPGAs can provide more on-chip memory (up to several hundred Mb), for example, in the form of Ultra-RAM (URAM) or up to several tenths of GB of HBM in a single package, implementing a complete IDS/IPS on a single FPGA may be possible with modern FPGAs.

Caching/KVS. Local storage/caching of data as close to a PE as possible is a proven technique for achieving low latency, not only for GPPs, but the concept is also used in large systems such as data centers. The reason for this is that it is not practical for data centers or other distributed systems to store all data locally at each node, which increases storage costs or requires additional synchronization to update all data over the network. Cloud server systems also use caching techniques to store frequently requested data on cloudlet servers, while requesting data from cloud servers only when it is missing from the cache to exploit data locality [37]. To meet the increasing demands for low latency, even with a large number of end-user devices, data must be stored even closer to or directly at the edge.

LaKe. LaKe [117] is a hardware solution implemented in Verilog based on the Memcached design [9, 57]. LaKe is implemented on a NetFPGA-SUME [136]. To hide access latency, the design provides two cache layers. The first layer is a shared cache consisting of fast but small on-chip BRAM, and the second layer consists of slower but larger on-board DRAM. The DRAM is used to store hash table buckets and data chunks, while the BRAM is used as a shared cache between PEs to store frequent KV pairs. In addition, a slab allocator also used in Memcached is implemented in hardware, using the available on-board SRAM to store the addresses of unused chunks, and a FIFO to preload the next available address to hide access latency.

Discussion. Their research showed a significant improvement in throughput, power efficiency, and especially latency compared to a host-based KV system. While the authors evaluated their design on a SmartNIC, they noted that such a design could also be implemented on switches. With the growing number of participants, such as IoT devices, the need to keep data close to the end devices within the network becomes apparent. This not only minimizes latency and offloads resources from the host but also relieves transmission pressure within the network, further reducing latency, when deployed on switches. In addition, conformance and seamless integration with existing technologies such as Memcached is of great importance to keep the hurdles for developers as low as possible, making it feasible to integrate INC designs into new and existing infrastructure.

ML. To efficiently train ML models, a large amount of data is required. Scaling, i.e., upgrading a device/server with more powerful hardware, is limited and economically unfeasible beyond a certain point. Therefore, a common approach to scaling large models and data volumes is to exploit data parallelism, not only on a single device, but by partitioning and distributing the data across multiple worker nodes. Training data in parallel over the network and synchronizing updates is supported by popular ML frameworks such as PyTorch [22] (Distributed Data-Parallel training) and TensorFlow [28] (tf.distribute.Strategy). Supervised ML models use iterative algorithms such as Stochastic Gradient Descent, where the sum of model updates must be computed frequently. This leads to periods of high network load for transmitting the updates, which results in the need to pause training until the updates are complete.

Liu et al. [85] proposed an in-network aggregation approach based on Remote Direct Memory Access (RDMA) [86] called NetReduce. They connected a Virtex Ultrascale FPGA [30] to a standard (nonprogrammable) switch. In their proposed design, updates sent by worker nodes are aggregated in the network and only the resulting aggregated gradient values are forwarded. They compare their proposed design to SwitchML [100], which is similar to NetReduce. However, SwitchML has been implemented for programmable ASIC switches (Tofino). To interact with the proposed switch design from the host side, they reuse the RDMA over Converged Ethernet protocol for their NetReduce protocol and the NVIDIA Collective Communication Library (NCCL) for optimized communication with the NVIDIA GPUs in their testbeds. They used the NetReduce protocol as a primitive in NCCL and used it in PyTorch and the Horovod framework to support TensorFlow.

Taurus [113] presents a modified RMT pipeline for the data plane by introducing a MapReduce control block based on CGRA Plasticine [95]. Their design targets line-rate inference for ML models such as Support Vector Machine, Deep Neural Network, k-means, and Long Short-Term Memory. However, in the absence of a CGRA-based programmable switch, the authors build a prototype using a Tofino switch connected over a 100 Gbps Ethernet link to a Xilinx Alveo U250 to emulate the MapReduce block and evaluate their design. In their follow-up work, Homunculus [114], they present a framework that supports both Taurus switches and programmable ASIC switches. Homunculus provides functions in Python that can be used, for example, with the TensorFlow function. The Python front-end generates JSON configuration files that are optimized by HyperMapper [92]. The back-end takes the optimized configuration to generate P4 or Spatial [78] code, depending on the target switch. To generate the P4 code targeting Tofino switches, they use the IIsy framework [133].

Discussion. The standard P4 programmable PISA architecture is not sufficient to accelerate ML in the data plane, according to [85] and [113]. In addition to the lack of features to implement ML models in PISA architectures [113], storing the results of the ML model [113] or the RDMA link [85] as flow rules would lead to frequent interaction with the control plane in modern dynamic network environments [85, 113], which should be avoided as it leads to additional delays in message processing. For example, in the context of inference in the network, the authors of Taurus [113] evaluated that it is not ML but rather table rule installation and packet collection that becomes the bottleneck when telemetry packets are sent to the control plane. In their study, they showed that this high delay causes the design to miss most anomalies, resulting in a low F1 score. When inference is performed on the data plane using the CGRA instead, they achieve a high F1 score regardless of the sampling rate (Table 3). However, for online training, they evaluated that using the control plane to optimize the global metric and update the weights of the Taurus ML model, that higher sampling rates and epochs with small batch size lead to faster convergence of the F1 score (Table 3). Their work showed that using ML models to optimize network functions such as anomaly detection can be realized in the network, but also that current PISA-based switches lack the necessary features and CPU-based do not provide the necessary performance to run ML models efficiently [113], which confirms the study conducted by Cooke and Fahmy [51].

We reviewed research proposing different hardware implementation solutions for different domains. We found that many challenges arise from the limitations of existing PISA-based architectures, but also that there is no one-size-fits-all solution. While some problems, such as variable field lengths, could be solved using the extern calls introduced with P4\({}_{16}\), P4 implementations would lose their generality/portability [124]. In addition, many PISA design solutions would require frequent interactions with the control plane to offload processing to the network [79, 85, 113] or lack necessary features [85]. Proposals such as PsPIN [53] can provide high performance despite a high degree of flexibility and parallelism, but there are limitations as soon as domain-specific acceleration is required. While [109] and [102] investigated virtualization for multi-tenant program offloading on reconfigurable switches, including security aspects [109] and a discussion about challenges regarding PR [102], their investigated scope was limited to basic network applications like L2-Switches and INT. Works such as [81, 82] have included the investigation of hardware scheduling techniques for heterogeneous SmartNIC devices. However, several areas remain open for future work, including automatization of finding the right partition size, and task mapping mechanism to decide which task or task chain to offload to a specific PR and when to perform PR.

As we have seen in the study by Sit et al. [105], while it is possible to implement entire services on FPGAs, implementations, for example, iterative algorithms may suffer from low achievable frequencies or some parts as in the work of Zhao et al. [132] are not easy to implement on hardware accelerators and the performance gain to offload such parts is not in reasonable proportion to the effort. Liu et al. [85] conclude that offloading a task such as in-network aggregation to a SmartNIC is generally only useful to free up the host CPU, since implementing a hardware accelerator requires a lot of development effort, and MuSoC-based SmartNIC such as NVIDIA’s BlueField DPUs may suffer from similar latency and throughput issues as existing in-network aggregation solutions.

Designs such as hXDP [46] have proposed combining an eBPF-compliant VLIW processor with tightly integrated accelerators and WE [42]. However, as the accelerators in hXDP are fixed at runtime, they lack like state-of-the-art MuSoC and MaSoC in the flexibility to exchange hardware accelerator online. eHDL [97] on the other hand provided an automatic eBPF \(\rightarrow\) FPGA workflow to create a whole pipeline instead. They were able to improve throughput compared to hXDP and BlueField-2 [42, 46], with similar latency compared to hXDP (Table 3). However, like hXDP [46], they do not provide online reconfigurability. On the contrary, hXDP even has the advantage that it can be reprogrammed and the WE can be repopulated at runtime [46]. Therefore, realizing online reconfigurability to support offloading and exchanging of eBPF programs (or programs in general) without reconfiguring the whole FPGA and interfering with other eBPF programs remains an open challenge for research.

The possible range of applications that can be covered with INC is wide. It includes domains like network services, security, and ML, which we have discussed in this work, but also extends to areas such as Computer Vision and robotics. To make the utilization of reconfigurable hardware accelerators attractive, not only the network community but also from other domains have to be convinced. Programming diverse network devices requires architecture-specific knowledge, even with tools like SDNet/VNP4 for FPGAs. While [42, 46, 97] investigated the offloading of eBPF/XDP to hardware accelerators, their work required implementing custom compilers for specific targets and hardware knowledge to add new features. DSLs and libraries that abstract target-specific functionalities and optimizations could potentially help to lower this burden. Additionally, compiler infrastructures like MLIR [80] can provide reusability and extensibility for rapid deployment of domain-specific compilers or feature extensions. Therefore, the investigation of design choices to support hardware acceleration for various applications without burdening application developers [135], including the seamless integration of reconfigurable hardware accelerators into existing network and application infrastructures and paradigms, is a potential topic that few have addressed. To determine the benefits and identify potential challenges of INC hardware acceleration, further research under different network traffic conditions and consideration of custom protocols [108] is needed. Additional research challenges such as programmability and adaptability are discussed in the following section.