Article

SR-IOV networking in Xen: architecture, design and implementation

Authors:

Yaozu Dong,

Zhao Yu,

Greg RoseAuthors Info & Claims

WIOV'08: Proceedings of the First conference on I/O virtualization

Page 10

Published: 10 December 2008 Publication History

Publisher Site

Abstract

SR-IOV capable network devices offer the benefits of direct I/O throughput and reduced CPU utilization while greatly increasing the scalability and sharing capabilities of the device. SR-IOV allows the benefits of the paravirtualized driver's throughput increase and additional CPU usage reductions in HVMs (Hardware Virtual Machines). SR-IOV uses direct I/O assignment of a network device to multiple VMs, maximizing the potential for using the full bandwidth capabilities of the network device, as well as enabling unmodified guest OS based device drivers which will work for different underlying VMMs.

Drawing on our recent experience in developing an SR-IOV capable networking solution for the Xen hypervisor we discuss the system level requirements and techniques for SR-IOV enablement on the platform. We discuss PCI configuration considerations, direct MMIO, interrupt handling and DMA into an HVM using an IOMMU (I/O Memory Management Unit). We then explain the architectural, design and implementation considerations for SR-IOV networking in Xen in which the Physical Function has a driver running in the driver domain that serves as a "master" and each Virtual Function exposed to a guest VM has its own virtual driver.

References

[1]

Getting 10 Gb/s from Xen: Safe and Fast Device Access from Unprivileged Domains. http://www.solarflare.com/technology/documents/Getting_10Gbps_from_Xen_cognidox.pdf

Google Scholar

[2]

PCI SIG: PCI-SIG Single Root I/O Virtualization. http://www.pcisig.com/specifications/iov/single_root/

Google Scholar

[3]

Intel Virtualization Technology. http://www.intel.com/technology/itj/2006/v10i3/1-hardware/6-vt-x-vt-isolutions. htm

Google Scholar

[4]

Intel Corporation: Intel Virtualization Technology for Directed I/O. http://download.intel.com/technology/itj/2006/v10i3/v10-i3-art02.pdf

Google Scholar

[5]

FRASER, K., HAND, S., NEUGEBAUER, R., PRATT, I., WARFIELD, A., AND WILLIAMS, M. Safe hardware access with the Xen virtual machine monitor. In 1st Workshop on Operating System and Architectural Support for the on demand IT InfraStructure (OASIS) (October 2004). http://www.cl.cam.ac.uk/research/srg/netos/papers/2004-oasis-ngio.pdf

Google Scholar

[6]

Paul Willmann, Scott Rixner, and Alan L. Cox, Protection Strategies for Direct Access to Virtualized I/O Devices. In USENIX Annual Technical Conference (USENIX 08), Boston, MA, June 2008. http://www.usenix.org/events/usenix08/tech/full_papers/willmann/willmann_html/index.html

Crossref

Google Scholar

[7]

PCI SIG: PCI-SIG Local Bus Specification 3.0. http://www.pcisig.com/members/downloads/specifications/conventional/PCI_LB3.0-2-6-04.pdf

Google Scholar

[8]

Xen.org. http://www.xen.org/

Google Scholar

[9]

Intel 82576 Gigabit Ethernet Controller. http://download.intel.com/design/network/ProdBrf/320025.pdf

Google Scholar

[10]

I. Pratt, K. Fraser, S.Hand, C. Limpach, A. Warfield, Dan. J Magenheimer, Jun Nakajima, and Asit Mallick. Xen 3.0 and the art of virtualization. In Proceedings of the 2005 Ottawa Linux Symposium, pages 65-77, July 2005.

Google Scholar

[11]

DONG, Y., LI, S., MALLICK, A., NAKAJIMA, J., TIAN, K., XU, X., YANG, F., AND YU, W. Extending Xen with Intel Virtualization Technology. In Intel Technology Journal (August 2006), vol. 10.

Google Scholar

[12]

Jose Renato Santos, Yoshio Turner, G.(John) Janakiraman, Ian Pratt. Bridging the Gap between Software and Hardware Techniques for I/O Virtualization, In USENIX Annual Technical Conference (USENIX 08), Boston, MA, June 2008.

Crossref

Google Scholar

[13]

Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. Xen and the art of virtualization. In Proceedings of the nineteenth ACM symposium on Operating Systems Principles (SOSP19), pages 164-177. ACM Press, 2003

Crossref

Google Scholar

[14]

The AMD-V Story http://www.amd.com/us-en/0,3715_15781_15785,00.html

Google Scholar

Cited By

View all

Jain VChu HQi SLee CChang HHsieh CRamakrishnan KChen JKuipers FOrda A(2022)L25GCProceedings of the ACM SIGCOMM 2022 Conference10.1145/3544216.3544267(143-157)Online publication date: 22-Aug-2022
https://dl.acm.org/doi/10.1145/3544216.3544267
Cheng KDoddamani SChiueh TLi YGopalan KNagarakatte SBaumann AKasikci B(2020)DirectvisorProceedings of the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3381052.3381317(45-58)Online publication date: 17-Mar-2020
https://dl.acm.org/doi/10.1145/3381052.3381317
Doddamani SSinha PLu HCheng TBagdi HGopalan KSartor JNaik MRossbach C(2019)Fast and live hypervisor replacementProceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3313808.3313821(45-58)Online publication date: 14-Apr-2019
https://dl.acm.org/doi/10.1145/3313808.3313821
Show More Cited By

SR-IOV networking in Xen: architecture, design and implementation
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains

Recommendations

Early experiences with live migration of SR-IOV enabled InfiniBand

Virtualization is the key to efficient resource utilization and elastic resource allocation in cloud computing. It enables consolidation, the on-demand provisioning of resources, and elasticity through live migration. Live migration makes it possible to ...
SR-IOV: Performance Benefits for Virtualized Interconnects
XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment

The demand for virtualization within high-performance computing is rapidly growing as new communities, driven by both new application stacks and new computing modalities, continue to grow and expand. While virtualization has traditionally come with ...
A Hypervisor Approach to Enable Live Migration with Passthrough SR-IOV Network Devices
Special Topics

Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (physical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

WIOV'08: Proceedings of the First conference on I/O virtualization

December 2008

10 pages

Publisher

USENIX Association

United States

Publication History

Published: 10 December 2008

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

44
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Jain VChu HQi SLee CChang HHsieh CRamakrishnan KChen JKuipers FOrda A(2022)L25GCProceedings of the ACM SIGCOMM 2022 Conference10.1145/3544216.3544267(143-157)Online publication date: 22-Aug-2022
https://dl.acm.org/doi/10.1145/3544216.3544267
Cheng KDoddamani SChiueh TLi YGopalan KNagarakatte SBaumann AKasikci B(2020)DirectvisorProceedings of the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3381052.3381317(45-58)Online publication date: 17-Mar-2020
https://dl.acm.org/doi/10.1145/3381052.3381317
Doddamani SSinha PLu HCheng TBagdi HGopalan KSartor JNaik MRossbach C(2019)Fast and live hypervisor replacementProceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3313808.3313821(45-58)Online publication date: 14-Apr-2019
https://dl.acm.org/doi/10.1145/3313808.3313821
Liu LWang HWang AXiao MCheng YChen SSartor JNaik MRossbach C(2019)vCPU as a container: towards accurate CPU allocation for VMsProceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3313808.3313814(193-206)Online publication date: 14-Apr-2019
https://dl.acm.org/doi/10.1145/3313808.3313814
Jiang ZAudsley NDong P(2019)BlueIOACM Transactions on Embedded Computing Systems10.1145/330976518:3(1-25)Online publication date: 9-Apr-2019
https://dl.acm.org/doi/10.1145/3309765
Zhou ZLi ZZhang KAhn GPretschner AGhinita G(2017)All Your VMs are DisconnectedProceedings of the Seventh ACM on Conference on Data and Application Security and Privacy10.1145/3029806.3029810(249-260)Online publication date: 22-Mar-2017
https://dl.acm.org/doi/10.1145/3029806.3029810
Kontodimas KKokkinos PKuperman YHoubavlis AVarvarigos E(2017)Analysis and Evaluation of Scheduling Policies for Consolidated I/O OperationsJournal of Grid Computing10.1007/s10723-017-9392-415:1(107-125)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1007/s10723-017-9392-4
Kim JAhn SLa KChang WOssowski S(2016)Improving I/O performance of NVMe SSD on virtual machinesProceedings of the 31st Annual ACM Symposium on Applied Computing10.1145/2851613.2851739(1852-1857)Online publication date: 4-Apr-2016
https://dl.acm.org/doi/10.1145/2851613.2851739
Teabe BTchana AHagimont DVarela CCastro HBarrios C(2016)Billing system CPU time on individual VMProceedings of the 16th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2016.76(493-496)Online publication date: 16-May-2016
https://dl.acm.org/doi/10.1109/CCGrid.2016.76
Kontodimas KKokkinos PKuperman YVarvarigos EAnjum APapadopoulos G(2015)Analysis and evaluation of I/O hypervisor schedulingProceedings of the 8th International Conference on Utility and Cloud Computing10.5555/3233397.3233405(45-54)Online publication date: 7-Dec-2015
https://dl.acm.org/doi/10.5555/3233397.3233405
Show More Cited By

Abstract

References

Cited By

Recommendations

Early experiences with live migration of SR-IOV enabled InfiniBand

SR-IOV: Performance Benefits for Virtualized Interconnects

A Hypervisor Approach to Enable Live Migration with Passthrough SR-IOV Network Devices

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations