More Web Proxy on the site http://driver.im/

research-article

revisiting the open vSwitch dataplane ten years later

Authors:

Gianni Antichi,

Ben PfaffAuthors Info & Claims

SIGCOMM '21: Proceedings of the 2021 ACM SIGCOMM 2021 Conference

Pages 245 - 257

https://doi.org/10.1145/3452296.3472914

Published: 09 August 2021 Publication History

Abstract

This paper shares our experience in supporting and running the Open vSwitch (OVS) software switch, as part of the NSX product for enterprise data center virtualization used by thousands of VMware customers. Starting in 2009, the OVS design split its code between tightly coupled kernel and userspace components. This split was necessary at the time for performance, but it caused maintainability problems that persist today. In addition, in-kernel packet processing is now much slower than newer options.

To solve the problems caused by the user/kernel split, OVS must adopt a new architecture. We describe two possibilities that we explored, but did not adopt, one because it gives up compatibility with drivers and tools that are important to virtual data center operators, the other because it performs poorly. Instead, we endorse a third approach, based on a new Linux socket type called AF\_XDP, which solves the maintainability problem in a compatible, performant way. The new code is already merged into the mainstream OVS repository. We include a thorough performance evaluation and a collection of lessons learned.

Supplementary Material

ganjali-public-review (300-public-review.pdf)

Revisiting the Open vSwitch Dataplane Ten Years Later: Public Review

Download
68.07 KB

MP4 File (video-presentation.mp4)

Conference Presentation Video

Download
18.93 MB

MP4 File (video-long.mp4)

Long Version Video

Download
221.94 MB

References

[1]

6Wind. 2020. 6Wind Datacenter Networking. https://www.6wind.com/products/solutions/data-center-networking/.

[2]

PREVAIL: a Polynomial-Runtime EBPF Verifier using an Abstract Interpretation Layer. 2021. uBPF. https://github.com/vbpf/ebpf-verifier.

[3]

Ariel Adam and Amnon Ilan. 2019. How vhost-user came into being: Virtio-networking and DPDK. https://www.redhat.com/en/blog/how-vhost-user-came-being-virtio-networking-and-dpdk.

[4]

David Barach, Leonardo Linguaglossa, Damjan Marion, Pierre Pfister, Salvatore Pontarelli, and Dario Rossi. 2018. High-Speed Software Data Plane via Vectorized Packet Processing. In Communications Magazine, Volume: 56, Issue: 12. IEEE.

[5]

Lorenzo Bianconi. 2020. Introduce support for XDP programs in CPUMAP. https://lwn.net/Articles/826114/.

[6]

Thomas Bittman, Philip Dawson, and Michael Warrilow. 2016. Gartner Magic Quadrant for x86 Server Virtualization Infrastructure. In Gartner Research.

[7]

Brenden Blanco, Yonghong Song, et al. 2016. BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more. https://github.com/iovisor/bcc/.

[8]

Jesper Dangaard Brouer. 2016. XDP -- eXpress Data Path, Intro and future use-cases. http://people.netfilter.org/hawk/presentations/xdp2016/xdp_intro_and_use_cases_sep2016.pdf.

[9]

Mihai Budiu. 2015. Compiling P4 to eBPF. https://github.com/iovisor/bcc/tree/master/src/cc/frontends/p4.

[10]

Paul Chaignon, Kahina Lazri, Jérôme François, Thibault Delmas, and Olivier Festor. 2018. Oko: Extending Open vSwitch with stateful filters. In Symposium on SDN Research (SOSR). ACM.

Digital Library

[11]

Cisco Inc. [n.d.]. TRex: Realistic Traffic Generator. https://trex-tgn.cisco.com/.

[12]

Jonathan Corbet. 2015. Improving Linux networking performance. https://lwn.net/Articles/629155/.

[13]

Jonathan Corbet. 2018. Accelerating networking with AFXDP. https://lwn.net/Articles/750845/.

[14]

Jonathan Corbet. 2020. A medley of performance-related BPF patches. https://lwn.net/Articles/808503/.

[15]

Linux Networking Documentation. 2020. AFXDP: XDPSKB and XDPDRV Modes. https://www.kernel.org/doc/html/v4.18/networking/af_xdp.html.

[16]

DPDK. 2020. AFXDP Poll Mode Driver. https://doc.dpdk.org/guides-20.05/nics/af_xdp.html.

[17]

DPDK Community. 2018. DPDK - DataPlane Development Kit. http://www.dpdk.org/.

[18]

DPDK Guide. [n.d.]a. DPDK Tools User Guides. https://doc.dpdk.org/guides/testpmd_app_ug/.

[19]

DPDK Guide. [n.d.]b. Testpmd Application User Guide. http://fast.dpdk.org/doc/pdf-guides-18.08/tools-18.08.pdf.

[20]

DPDK Guide. 2018. Virtio for Container Networking. https://doc.dpdk.org/guides/howto/virtio_user_for_container_networking.html.

[21]

Ericsson. 2020. Cloud SDN. https://www.ericsson.com/en/portfolio/digital-services/cloud-infrastructure/cloud-sdn.

[22]

Julia Evans. 2019. Writing eBPF tracing tools in Rust. https://jvns.ca/blog/2018/02/05/rust-bcc/.

[23]

F-Stack. 2019. User Space Network Development Kit. http://www.f-stack.org/.

[24]

Daniel Firestone. 2017. VFP: A Virtual Switch Platform for Host SDN in the Public Cloud. In Networked Systems Design and Implementation (NSDI). USENIX.

[25]

Matt Fleming. 2017. A thorough introduction to eBPF. https://lwn.net/Articles/740157/.

[26]

Linux Foundation. 2020. Open vSwitch upstream repository. https://github.com/openvswitch/ovs.

[27]

Sameh Gobriel and Charlie Tai. 2017. OVS-CD: Optimizing Flow Classification for OVS using the DPDK Membership Library. https://www.openvswitch.org/support/ovscon2017/.

[28]

DPDK Guide. 2019. Mbuf Library. https://static.sched.com/hosted_files/envoyconna18/e0/envoycon_dpdkvn.pdf.

[29]

Stefan Hajnoczi. 2011. QEMU Internals: vhost architecture. http://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html.

[30]

Sangjin Han, Keon Jang, Aurojit Panda, Shoumik Palkar, Dongsu Han, and Sylvia Ratnasamy. 2015. SoftNIC: A software NIC to augment hardware. EECS Department, University of California, Berkeley, Tech. Rep. UCB/EECS-2015-155 (2015).

[31]

Mark Haranas. 2019. Pat Gelsinger: Cisco ACI `Bicycle' Will Never Match VMware NSX `Lamborghini'. https://www.crn.com/news/networking/pat-gelsinger-cisco-aci-bicycle-will-never-match-vmware-nsx-lamborghini-.

[32]

Toke Høiland-Jørgensen, Jesper Dangaard Brouer, Daniel Borkmann, John Fastabend, Tom Herbert, David Ahern, and David Miller. 2018. The EXpress Data Path: Fast Programmable Packet Processing in the Operating System Kernel. In Conference on emerging Networking EXperiments and Technologies (CoNEXT). ACM.

Digital Library

[33]

Magnus Karlsson and Björn Töpel. 2018. The path to DPDK speeds for AF XDP. In Linux Plumbers Conference.

[34]

Rich Lane, Paul Chaignon, et al. 2021. uBPF. https://github.com/iovisor/ubpf.

[35]

LF Projects. 2018. Tungsten Fabric Architecture. https://tungstenfabric.github.io/website/Tungsten-Fabric-Architecture.html.

[36]

Linux bridge 2021. Linux bridge. https://wiki.linuxfoundation.org/networking/bridge.

[37]

Hangbin Liu. 2020. xdp: add dev map multicast support. https://lwn.net/Articles/817582/.

[38]

Alan Maguire. 2019. OVS iptunnel bug. https://github.com/openvswitch/ovs/commit/902c5ffd3360b05ad344f7f4f5ee0301ae331a5.

[39]

Saeed Mahameed. 2020. XDP meta-data Acceleration. https://netdevconf.info/0x14/session.html?workshop-XDP.

[40]

Saeed Mahameed and et al. 2020. Linux NetDev 0x14: XDP Workshop. https://netdevconf.info/0x14/session.html?workshop-XDP.

[41]

Toshiaki Makita and William Tu. 2020. Linux NetDev 0x14: Fast OVS Datapath with XDP. https://netdevconf.info/0x14/session.html?talk-fast-OVS-data-path-with-XDP.

[42]

Michael Marty, Marc de Kruijf, Jacob Adriaens, Christopher Alfeld, Sean Bauer, Carlo Contavalli, Michael Dalton, Nandita Dukkipati, William C Evans, Steve Gribble, et al. 2019. Snap: a microkernel approach to host networking. In Proceedings of the 27th ACM Symposium on Operating Systems Principles. 399--413.

Digital Library

[43]

John McNamara. 2017. API ABI Stability and LTS: Current state and Future. In DPDK Summit Userspace.

[44]

Microsoft. 2020. Hyper-V Technology Overview. https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/hyper-v-technology-overview.

[45]

Microsoft. 2021. eBPF for Windows. https://github.com/Microsoft/ebpf-for-windows.

[46]

David Miller. 2015. net: Add STT support. https://marc.info/?l=linux-netdev\&m=142275484322111\&w=2.

[47]

David S. Miller. 2012. Removing the Linux Routing Cache. http://vger.kernel.org/ davem/columbia2012.pdf.

[48]

David S. Miller. 2016. Fast Programmable Networks & Encapsulated Protocols. https://netdevconf.info/1.2/session.html?david-miller-keynote.

[49]

Paul Moore. 2017. Generic Netlink. https://lwn.net/Articles/208755/.

[50]

Digital Ocean. 2020. OVS in the Cloud. http://www.openvswitch.org/support/ovscon2019/.

[51]

Open Source Security, Inc. 2020. grsecurity. https://grsecurity.net/.

[52]

OVS. 2020. Releases: Q: Are all features available with all datapaths? https://docs.openvswitch.org/en/latest/faq/releases/.

[53]

OVS Community. 2018. Open vSwitch with DPDK. http://docs.openvswitch.org/en/latest/intro/install/dpdk/.

[54]

Justin Pettit, Ben Pfaff, Joe Stringer, Cheng-Chun Tu, Brenden Blanco, and Alex Tessmer. 2018. Bringing Platform Harmony to VMware NSX. In SIGOPS Operating Systems Review, Volume: 52, Issue: 1. ACM.

[55]

Ben Pfaff. 2011. Open vSwitch datapath developer documentation. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/openvswitch.rst.

[56]

Ben Pfaff, Justin Pettit, Teemu Koponen, Ethan J. Jackson, Andy Zhou, Jarno Rajahalme, Jesse Gross, Alex Wang, Jonathan Stringer, Pravin Shelar, Keith Amidon, and Martín Casado. 2015. The Design and Implementation of Open VSwitch. In Networked Systems Design and Implementation (NSDI). USENIX.

[57]

Greg Rose. 2018. compat: Add ipv6 GRE and IPV6 Tunneling. https://github.com/openvswitch/ovs/commit/c387d8177f20.

[58]

Gregory Rose. 2020. Question about supporting the OVS out-of-tree kernel drivers. https://mail.openvswitch.org/pipermail/ovs-dev/2020-December/378831.html.

[59]

Muhammad Shahbaz, Sean Choi, Ben Pfaff, Changhoon Kim, Nick Feamster, Nick McKeown, and Jennifer Rexford. 2016. PISCES: A Programmable, Protocol-Independent Software Switch. In Special Interest Group on Data Communication (SIGCOMM). ACM.

Digital Library

[60]

Pravin Shelar. 2015. Open vSwitch STT support. https://mail.openvswitch.org/pipermail/ovs-dev/2015-February/294420.html.

[61]

Pravin B Shelar. 2014. openvswitch: Introduce flow mask cache. https://patchwork.ozlabs.org/project/netdev/patch/[email protected]/.

[62]

Jianfeng Tan, Cunming Liang, Huawei Xie, Qian Xu, Jiayu Hu, Heqing Zhu, and Yuanhan Liu. 2017. VIRTIO-USER: A new versatile channel for kernel-bypass networks. In Proceedings of the Workshop on Kernel-Bypass Networks. 13--18.

Digital Library

[63]

Björn Töpel. 2020a. Introduce AFXDP buffer allocation API. https://lwn.net/Articles/821115/.

[64]

Björn Töpel. 2020b. Introduce preferred busy-polling. https://lwn.net/Articles/837010/.

[65]

Linus Torvalds. 2005. GCC versus kernel stability. https://yarchive.net/comp/linux/gcc_vs_kernel_stability.html.

[66]

Cheng-Chun Tu, Joe Stringer, and Justin Pettit. 2017. Building an extensible Open vSwitch datapath. In SIGOPS Operating Systems Review, Volume: 51, Issue: 1. ACM.

[67]

William Tu. 2018a. AFXDP support for veth. https://patchwork.ozlabs.org/cover/1015775/.

[68]

William Tu. 2018b. openvswitch: add erspan version I and II support. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fc1372f89ffe.

[69]

William Tu and Greg Rose. 2018. ERSPAN Support for Linux, Linux Plumber Conference. http://vger.kernel.org/lpc_net2018_talks/erspan-linux.pdf.

[70]

William Tu, Fabian Ruffy, and Mihai Budiu. 2018a. Linux Network Programming with P4. In Linux Plumber Conference.

[71]

William Tu and Alexei Starovoitov. 2018. [RFC PATCH 00/11] OVS eBPF datapath. https://lists.iovisor.org/g/iovisor-dev/topic/22656941.

[72]

William Tu, Joe Stringer, Yifeng Sun Sun, and Yi-Hung Wei. 2018b. Bringing the Power of eBPF to Open vSwitch. In Linux Plumber Conference.

[73]

William Tu, Yi-Hung Wei, and Ilya Maximets. 2020. Open vSwitch with AFXDP. https://docs.openvswitch.org/en/latest/intro/install/afxdp/.

[74]

Asanga Udugama. 2006. Manipulating the networking environment using RTNETLINK. Linux Journal 145 (2006).

[75]

Harry van Haaren. 2018. Applying SIMD Optimizations to the OVS Datapath. https://www.openvswitch.org/support/ovscon2018/5/1355-van-haaren.pdf.

[76]

VMware. 2020a. Update to VMware's per-CPU Pricing Model. https://www.vmware.com/company/news/updates/cpu-pricing-model-update-feb-2020.html.

[77]

VMware. 2020b. VMware NSX Data Center. https://www.vmware.com/products/nsx.html.

[78]

Peter P. Waskiewicz Jr. and Neerav Parikh. 2018. Accelerating XDP Programs Using HW-based Hints. http://vger.kernel.org/lpc_net2018_talks/xdp-plumbers-2018.pdf.

[79]

Yi-Hung Wei. 2018a. datapath: compat: Backports nfconncount. https://github.com/openvswitch/ovs/commit/744964326f6c.

[80]

Yi-Hung Wei. 2018b. datapath: compat: Fix compilation issue with grsecurity patch. https://github.com/openvswitch/ovs/commit/1556fcca6766.

[81]

Yi Yang. 2019. OVS DPDK issues in Openstack and Kubernetes and Solutions. http://www.openvswitch.org/support/ovscon2019/.

Cited By

Li XJiang XYang YChen LWang YWang CXu CLv YYang BWu TGao HChen ZQiao YDing HDong YYang HSong JLu JZhang PWei CZhang ZChen WHe QZhu SSekar VYu MSeneviratne AVeitch D(2024)Triton: A Flexible Hardware Offloading Architecture for Accelerating Apsara vSwitch in Alibaba CloudProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672224(750-763)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672224
Liu YGuo JJiang BZhang PSun XSong YRen WHou ZLyu BWen RZhu SWang XVallina-Rodríguez NSuarez-Tángil GLevin DPelsser C(2024)Understanding Network Startup for Secure Containers in Multi-Tenant Clouds: Performance, Bottleneck and OptimizationProceedings of the 2024 ACM on Internet Measurement Conference10.1145/3646547.3688436(635-650)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3646547.3688436
Chen ZLi HOta KDong M(2024)Profit-Maximizing Service Function Chain Embedding in NFV-Based 5G Core NetworksIEEE Transactions on Network Science and Engineering10.1109/TNSE.2024.345475911:6(6105-6117)Online publication date: Nov-2024
https://doi.org/10.1109/TNSE.2024.3454759
Show More Cited By

Index Terms

revisiting the open vSwitch dataplane ten years later
1. Networks
  1. Network services
    1. Programmable networks

Recommendations

RFC 7047: The Open vSwitch Database Management Protocol
Accelerating Open vSwitch with Integrated GPU
KBNets '17: Proceedings of the Workshop on Kernel-Bypass Networks

With the fast development of Software Defined Networking (SDN) and network virtualization, software-based network virtual switches have emerged as a critical component to provide network services to VMs. Among virtual switches, Open vSwitch (OvS) is an ...
Characterizing the Performance of Concurrent Virtualized Network Functions with OVS-DPDK, FD.IO VPP and SR-IOV
ICPE '18: Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering

The virtualization of network functions is promising significant cost reductions for network operators. Running multiple network functions on a standard x86 server instead of dedicated appliances can increase the utilization of the underlying hardware,...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCOMM '21: Proceedings of the 2021 ACM SIGCOMM 2021 Conference

August 2021

868 pages

ISBN:9781450383837

DOI:10.1145/3452296

General Chairs:
Fernando Kuipers
Delft University of Technology
,
Matthew Caesar
University of Illinois at Urbana-Champaign

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 August 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Conference

SIGCOMM '21

Sponsor:

SIGCOMM

SIGCOMM '21: ACM SIGCOMM 2021 Conference

August 23 - 27, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 462 of 3,389 submissions, 14%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
1,860
Total Downloads

Downloads (Last 12 months)205
Downloads (Last 6 weeks)24

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li XJiang XYang YChen LWang YWang CXu CLv YYang BWu TGao HChen ZQiao YDing HDong YYang HSong JLu JZhang PWei CZhang ZChen WHe QZhu SSekar VYu MSeneviratne AVeitch D(2024)Triton: A Flexible Hardware Offloading Architecture for Accelerating Apsara vSwitch in Alibaba CloudProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672224(750-763)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672224
Liu YGuo JJiang BZhang PSun XSong YRen WHou ZLyu BWen RZhu SWang XVallina-Rodríguez NSuarez-Tángil GLevin DPelsser C(2024)Understanding Network Startup for Secure Containers in Multi-Tenant Clouds: Performance, Bottleneck and OptimizationProceedings of the 2024 ACM on Internet Measurement Conference10.1145/3646547.3688436(635-650)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3646547.3688436
Chen ZLi HOta KDong M(2024)Profit-Maximizing Service Function Chain Embedding in NFV-Based 5G Core NetworksIEEE Transactions on Network Science and Engineering10.1109/TNSE.2024.345475911:6(6105-6117)Online publication date: Nov-2024
https://doi.org/10.1109/TNSE.2024.3454759
Qi SMonis LZeng ZWang IRamakrishnan K(2024)SPRIGHT: High-Performance eBPF-Based Event-Driven, Shared-Memory Processing for Serverless ComputingIEEE/ACM Transactions on Networking10.1109/TNET.2024.336656132:3(2539-2554)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3366561
Xin YJia CLi WRottenstreich OXu YXie GTian ZLi J(2024)A Heterogeneous and Adaptive Architecture for Decision-tree-based ACL Engine on FPGAIEEE Transactions on Computers10.1109/TC.2024.3477955(1-14)Online publication date: 2024
https://doi.org/10.1109/TC.2024.3477955
Chen XSun XZhang WYao XWang ZLiu HHuang QPan GLiu XZhou HWu C(2024)Accelerating Sketch-based End-Host Traffic Measurement with Automatic DPU OffloadingIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621293(171-180)Online publication date: 20-May-2024
https://doi.org/10.1109/INFOCOM52122.2024.10621293
Charoenphol AReed M(2024)Link Fabrication Attack Detection for Software Defined Networks2024 International Conference on Intelligent Computing, Communication, Networking and Services (ICCNS)10.1109/ICCNS62192.2024.10776339(1-8)Online publication date: 24-Sep-2024
https://doi.org/10.1109/ICCNS62192.2024.10776339
du Perron KPacheco DHuet F(2024)Understanding Delays in AF_XDP-based ApplicationsICC 2024 - IEEE International Conference on Communications10.1109/ICC51166.2024.10622351(5497-5502)Online publication date: 9-Jun-2024
https://doi.org/10.1109/ICC51166.2024.10622351
Caldare ALeonardi LMiano SProcissi GAntichi GLettieri G(2024)Rethinking Cloud Network Stacks with Switch Bypass2024 IEEE 25th International Conference on High Performance Switching and Routing (HPSR)10.1109/HPSR62440.2024.10635954(55-60)Online publication date: 22-Jul-2024
https://doi.org/10.1109/HPSR62440.2024.10635954
Kfoury EChoueiri SMazloum AAlSabeh AGomez JCrichigno J(2024)A Comprehensive Survey on SmartNICs: Architectures, Development Models, Applications, and Research DirectionsIEEE Access10.1109/ACCESS.2024.343720312(107297-107336)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3437203
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents