More Web Proxy on the site http://driver.im/

research-article

vSocket: virtual socket interface for RDMA in public clouds

Authors:

Bei HuaAuthors Info & Claims

VEE 2019: Proceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

Pages 179 - 192

https://doi.org/10.1145/3313808.3313813

Published: 14 April 2019 Publication History

Abstract

RDMA has been widely adopted as a promising solution for high performance networks, but is still unavailable for a large number of socket-based applications running in public clouds due to the following reasons. There is no available virtualization technique of RDMA that can meet the cloud's requirements. Moreover, it is cost prohibitive to rewrite the socket-based applications with the Verbs API. To address the above problems, we present vSocket, a software-based RDMA virtualization framework for socket-based applications in public clouds. vSocket takes into account the demands of clouds such as security rules and network isolation, so it can be deployed in the current public clouds. Furthermore, vSocket provides native socket API so that socket-based applications can use it without any modifications. Finally, to validate the performance gains, we implemented a prototype and compared it with current virtual network solutions against 1) basic network benchmarks and 2) the Redis, a typical I/O intensive application. Experimental results show that the latency of basic benchmarks can be reduced by 88% and the throughput of Redis is improved by 4 times.

References

[1]

2018. Cloudius Systems. Seastar. http://seastar.io/ [Online; accessed 8-December-2018].

[2]

2018. Mellanox ConnectX-4 adaptor. http://www.mellanox.com/page/products_dyn?product_family=201&mtag=connectx_4_vpi_card. [Online; accessed 19-September-2018].

[3]

2018. Netperf benchmark. https://hewle_packard.github.io/netperf/ [Online; accessed 19-September-2018].

[4]

2018. Network ACLs. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html. [Online; accessed 26-February-2019].

[5]

2018. The official benchmark of Redis. https://redis.io/topics/benchmarks [Online; accessed 19-September-2018].

[6]

2018. Perf Wiki. https://perf.wiki.kernel.org/index.php/Main_Page. [Online; accessed 19-September-2018].

[7]

2018. The RDMA Verbs Specification. http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs-v1.0-RDMAC.pdf. [Online; accessed 19-September-2018].

[8]

2018. Redis, an open source (BSD licensed), in-memory data structure store. https://redis.io/ [Online; accessed 19-September-2018].

[9]

2018. Security Groups for Your VPC. https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html. [Online; accessed 8-December-2018].

[10]

2018. Single root I/O virtualization. http://pcisig.com/specifications/iov/single_root/ [Online; accessed 8-December-2018].

[11]

Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, Savannah, GA, 265-283.

Digital Library

[12]

Inho Cho, Keon Jang, and Dongsu Han. 2017. Credit-Scheduled Delay-Bounded Congestion Control for Datacenters. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM '17). ACM, New York, NY, USA, 239-252.

Digital Library

[13]

Aleksandar Dragojevic, Dushyanth Narayanan, Orion Hodson, and Miguel Castro. 2014. FaRM: Fast Remote Memory. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (NSDI'14). USENIX Association, Berkeley, CA, USA, 401-414. http://dl.acm.org/citation.cfm?id=2616448.2616486

Digital Library

[14]

Adam Dunkels. 2001. Design and Implementation of the lwIP TCP/IP Stack. In Swedish Institute of Computer Science.

[15]

M. Ferdman, P. Lotfi-Kamran, K. Balet, and B. Falsafi. 2011. Cuckoo directory: A scalable directory for many-core systems. In 2011 IEEE 17th International Symposium on High Performance Computer Architecture. 169-180.

Digital Library

[16]

Daniel Firestone, Andrew Putnam, Sambhrama Mundkur, Derek Chiou, Alireza Dabagh, Mike Andrewartha, Hari Angepat, Vivek Bhanu, Adrian Caulfield, Eric Chung, et al. 2018. Azure Accelerated Networking: SmartNICs in the Public Cloud. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18), Renton, WA.

Digital Library

[17]

D. Goldenberg, M. Kagan, R. Ravid, and M. S. Tsirkin. 2005. Zero copy sockets direct protocol over infiniband-preliminary implementation and performance analysis. In 13th Symposium on High Performance Interconnects (HOTI'05). 128-137.

Digital Library

[18]

Stefan Hajnoczi. 2011. QEMU Internals: vhost architecture. http://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html. [Online].

[19]

Mark Handley, Costin Raiciu, Alexandru Agache, Andrei Voinescu, Andrew W. Moore, Gianni Antichi, and Marcin Wojcik. 2017. Rearchitecting Datacenter Networks and Stacks for Low Latency and High Performance. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM '17). ACM, New York, NY, USA, 29-42.

Digital Library

[20]

SeanHefty. 2012. Rsockets. In 2012 OpenFabris International Workshop, Monterey, CA, USA.

[21]

Jinho Hwang, K K Ramakrishnan, and Timothy Wood. 2015. NetVM: high performance and fiexible networking using virtualization on commodity platforms. IEEE Transactions on Network and Service Management 12, 1 (2015), 34-47.

[22]

DPDK Intel. 2018. Data plane development kit.

[23]

Manish Jain, Ravi S Prasad, and Constantinos Dovrolis. 2003. The TCP bandwidth-delay product revisited: network buffering, cross traffic, and socket buffer auto-sizing. Technical Report. Georgia Institute of Technology.

[24]

EunYoung Jeong, Shinae Woo, Muhammad Asim Jamshed, Haewon Jeong, Sunghwan Ihm, Dongsu Han, and KyoungSoo Park. 2014. mTCP: a Highly Scalable User-level TCP Stack for Multicore Systems. In NSDI, Vol. 14. 489-502.

Digital Library

[25]

Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2014. Using RDMA Efficiently for Key-value Services. In Proceedings of the 2014 ACM Conference on SIGCOMM (SIGCOMM '14). ACM, New York, NY, USA, 295-306.

Digital Library

[26]

Patrick Kelsey. [n. d.]. Libuinet. https://github.com/pkelsey/libuinet [Online].

[27]

M. Mahalingam, K. Duda, P. Agarwal, L. Kreeger, T. Sridhar, M. Bursell, and C. Wright. 2018. Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks. https://www.rfc-editor.org/info/rfc7348 [Online].

[28]

Mellanox. [n. d.]. Mellanox Messaging Accelerator. http://www.mellanox.com/page/so_ware_vma [Online].

[29]

Ben Pfaff, Justin Pettit, Teemu Koponen, Ethan J Jackson, Andy Zhou, Jarno Rajahalme, Jesse Gross, Alex Wang, Joe Stringer, Pravin Shelar, et al. 2015. The Design and Implementation of Open v Switch. In NSDI, Vol. 15. 117-130.

Digital Library

[30]

Jonas Pfefferle et al. 2015. A Hybrid I/O Virtualization Framework for RDMA-capable Network Interfaces. In Proceedings of the VEE (VEE '15). 17-30.

Digital Library

[31]

Adit Ranadive et al. 2012. Toward a Paravirtual v RDMA Device for VMware ESXi Guests. VMware Technical Journal. Georgia Institute of Technology, VMWare Inc.

[32]

Rusty Russell. 2008. virtio: towards a de-facto standard for virtual I/O devices. ACM SIGOPS Operating Systems Review 42, 5 (2008), 95-103.

Digital Library

[33]

Michael Scharf and Sebastian Kiesel. 2006. Head-of-line Blocking in TCP and SCTP: Analysis and Measurements. In GLOBECOM, Vol. 6. 1-5.

[34]

P. Shivam, P. Wyckoff, and D. Panda. 2001. EMP: Zero-Copy OS-Bypass NIC-Driven Gigabit Ethernet Message Passing. In SC '01: Proceedings of the 2001 ACM/IEEE Conference on Supercomputing. 49-49.

Digital Library

[35]

Jia Song and Jim Alves-Foss. 2012. Performance review of zero copy techniques. International Journal of Computer Science and Security (IJCSS) 6, 4 (2012), 256.

[36]

Wikipedia. 2018. Token bucket. https://en.wikipedia.org/wiki/Token_bucket [Online; accessed 8-December-2018].

[37]

Wikipedia. 2018. Traffic shaping. https://en.wikipedia.org/wiki/Traffic_shaping [Online; accessed 8-December-2018].

[38]

Bairen Yi, Jiacheng Xia, Li Chen, and Kai Chen. 2017. Towards Zero Copy Dataflows Using RDMA. In Proceedings of the SIGCOMM Posters and Demos (SIGCOMM Posters and Demos '17). ACM, New York, NY, USA, 28-30.

Digital Library

[39]

Tianlong Yu, Shadi Abdollahian Noghabi, Shachar Raindel, Hongqiang Liu, Jitu Padhye, and Vyas Sekar. 2016. FreeFlow: High Performance Container Networking. In Proceedings of the 15th ACM Workshop on Hot Topics in Networks (HotNets '16). ACM, New York, NY, USA, 43-49.

Digital Library

[40]

Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. 2015. Congestion Control for Large-Scale RDMA Deployments. SIGCOMM Comput. Commun. Rev. 45, 4 (Aug. 2015), 523-536.

Digital Library

Cited By

Xu ZXu GSong X(2024)Transparent Dual-Node Hot Standby Scheme Based on PCIe Synchronization2024 IEEE 6th International Conference on Power, Intelligent Computing and Systems (ICPICS)10.1109/ICPICS62053.2024.10796769(1732-1735)Online publication date: 26-Jul-2024
https://doi.org/10.1109/ICPICS62053.2024.10796769
Rosa LFoschini LCorradi A(2024)Empowering Cloud Computing With Network Acceleration: A SurveyIEEE Communications Surveys & Tutorials10.1109/COMST.2024.337753126:4(2729-2768)Online publication date: Dec-2025
https://doi.org/10.1109/COMST.2024.3377531
Liu HLi WPang YPei RHu YLiu YSuo LLi K(2023)Accelerating Data Delivery of Latency-Sensitive Applications in Container Overlay NetworkIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.330074534:12(3046-3058)Online publication date: Dec-2023
https://doi.org/10.1109/TPDS.2023.3300745
Show More Cited By

Index Terms

vSocket: virtual socket interface for RDMA in public clouds
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Software infrastructure
        Virtual machines

Recommendations

A Hybrid I/O Virtualization Framework for RDMA-capable Network Interfaces
VEE '15: Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

DMA-capable interconnects, providing ultra-low latency and high bandwidth, are increasingly being used in the context of distributed storage and data processing systems. However, the deployment of such systems in virtualized data centers is currently ...
A Hybrid I/O Virtualization Framework for RDMA-capable Network Interfaces
VEE '15

DMA-capable interconnects, providing ultra-low latency and high bandwidth, are increasingly being used in the context of distributed storage and data processing systems. However, the deployment of such systems in virtualized data centers is currently ...
Software-based Live Migration for Containerized RDMA
APNet '24: Proceedings of the 8th Asia-Pacific Workshop on Networking

Container live migration is critical to ensure services are not interrupted during host maintenance in data centers. On the other hand, RDMA containerization has attracted both academia and industry for years. However, live migration for containerized ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

VEE 2019: Proceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

April 2019

206 pages

ISBN:9781450360203

DOI:10.1145/3313808

General Chair:
Jennifer Sartor
Ghent University, Belgium / Vrije Universiteit Brussel, Belgium
,
Program Chairs:
Mayur Naik
University of Pennsylvania, USA
,
Chris Rossbach
University of Texas at Austin, USA / VMware, USA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 April 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

VEE '19

Sponsor:

VEE '19: 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

April 14, 2019

RI, Providence, USA

Acceptance Rates

Overall Acceptance Rate 80 of 235 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
705
Total Downloads

Downloads (Last 12 months)59
Downloads (Last 6 weeks)7

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xu ZXu GSong X(2024)Transparent Dual-Node Hot Standby Scheme Based on PCIe Synchronization2024 IEEE 6th International Conference on Power, Intelligent Computing and Systems (ICPICS)10.1109/ICPICS62053.2024.10796769(1732-1735)Online publication date: 26-Jul-2024
https://doi.org/10.1109/ICPICS62053.2024.10796769
Rosa LFoschini LCorradi A(2024)Empowering Cloud Computing With Network Acceleration: A SurveyIEEE Communications Surveys & Tutorials10.1109/COMST.2024.337753126:4(2729-2768)Online publication date: Dec-2025
https://doi.org/10.1109/COMST.2024.3377531
Liu HLi WPang YPei RHu YLiu YSuo LLi K(2023)Accelerating Data Delivery of Latency-Sensitive Applications in Container Overlay NetworkIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.330074534:12(3046-3058)Online publication date: Dec-2023
https://doi.org/10.1109/TPDS.2023.3300745
Xu ZYin JLi W(2023)A Load-balancing method for high performance cluster computing system2023 IEEE 3rd International Conference on Electronic Technology, Communication and Information (ICETCI)10.1109/ICETCI57876.2023.10176628(1775-1779)Online publication date: 26-May-2023
https://doi.org/10.1109/ICETCI57876.2023.10176628
Kashyap ALu XWeissman JChandra AGavrilovska ATiwari D(2022)NVMe-oAFProceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing10.1145/3502181.3531476(56-70)Online publication date: 27-Jun-2022
https://dl.acm.org/doi/10.1145/3502181.3531476
Ma SMa TChen KWu Y(2022)A Survey of Storage Systems in the RDMA EraIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2022.318865633:12(4395-4409)Online publication date: 1-Dec-2022
https://doi.org/10.1109/TPDS.2022.3188656
Kawashima R(2021)Software Physical/Virtual Rx Queue Mapping Toward High-Performance Containerized NetworkingIEEE Transactions on Network and Service Management10.1109/TNSM.2020.304905318:1(687-700)Online publication date: Mar-2021
https://doi.org/10.1109/TNSM.2020.3049053
Liang YDai H(2021)Application Virtualization: An Agent Encapsulation of Software in Virtual Machines to Archive the Execution Performance in Hosts2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom)10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00090(618-625)Online publication date: Sep-2021
https://doi.org/10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00090
Xu ZLiu HLiu Y(2021)PCIe P2P Communication for the High Performance Heterogeneous Computing System2021 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA)10.1109/ICAICA52286.2021.9498081(284-288)Online publication date: 28-Jun-2021
https://doi.org/10.1109/ICAICA52286.2021.9498081
Kawashima R(2021)A Vision to Software-Centric Cloud Native Network Functions: Achievements and Challenges2021 IEEE 22nd International Conference on High Performance Switching and Routing (HPSR)10.1109/HPSR52026.2021.9481848(1-7)Online publication date: 7-Jun-2021
https://doi.org/10.1109/HPSR52026.2021.9481848
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents