More Web Proxy on the site http://driver.im/

tutorial

Offloading communication control logic in GPU accelerated applications

Authors:

Elena Agostini,

Davide Rossetti,

Sreeram PotluriAuthors Info & Claims

CCGrid '17: Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

Pages 248 - 257

https://doi.org/10.1109/CCGRID.2017.29

Published: 14 May 2017 Publication History

Abstract

NVIDIA GPUDirect is a family of technologies aimed at optimizing data movement among GPUs (P2P) or between GPUs and third-party devices (RDMA). GPUDirect Async, introduced in CUDA 8.0, is a new addition which allows direct synchronization between GPU and third party devices. For example, Async allows an NVIDIA GPU to directly trigger and poll for completion of communication operations queued to an InfiniBand Connect-IB network adapter, removing CPU involvement from the critical path in GPU accelerated applications. In this paper, we present the building blocks of GPUDirect Async and explain the supported usage models of this new technology. We also present a performance evaluation using a micro-benchmark and a synthetic stencil benchmark. Finally, we demonstrate the use of Async in a few multi-GPU MPI applications: HPGMG-FV (geometric multi-grid), achieving up to 25% improvement in total execution time; CoMD-CUDA (classical molecular dynamics), reducing communications times up to 30%; LULESH2-CUDA, achieving an average performance improvement of 13% in the total execution time.

References

[1]

GPUDirect family https://developer.nvidia.com/gpudirect

[2]

Mellanox GDR http://www.mellanox.com/page/products_dyn?product_family=116

[3]

IB Verbs RDMA programming guide. http://www.mellanox.com/relateddocs/prod_software/RDMA_Aware_Programming_user_manual.pdf

[4]

CUDA-Aware MPI. https://devblogs.nvidia.com/parallelforall/introductioncuda-aware-mpi

[5]

S. Kim, S. Huh, Y. Hu, X. Zhang, E. Witchel GPUnet: Networking Abstractions for GPU Programs, Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, October, 2014.

Digital Library

[6]

F. Daoud, A. Watad, M. Silberstein GPUrdma: GPU-side library for high performance networking from GPU kernels, Proceedings of the 6th International Workshop on Runtime and Operating Systems for Supercomputers, Article No. 6.

Digital Library

[7]

L. Oden, H. Frning, F. Pfreundt Infiniband-Verbs on GPU: A case study of controlling an Infiniband network device from the GPU, Parallel & Distributed Processing Symposium Workshops (IPDPSW), 2014 IEEE International.

Digital Library

[8]

S. Potluri, K. Hamidouche, A. Venkatesh, D. Bureddy, D. K. Panda, Efficient Inter-node MPI Communication Using GPUDirect RDMA for InfiniBand Clusters with NVIDIA GPUs.

[9]

Infiniband. http://www.mellanox.com/pdf/whitepapers/IB_Intro_WP_190.pdf

[10]

HPGMG https://hpgmg.org

[11]

N. Sakharnykh High-Performance Geometric Multi-Grid with GPU Acceleration. https://devblogs.nvidia.com/parallelforall/high-performance-geometric-multi-grid-gpu-acceleration

[12]

Finite Volume method. https://en.wikipedia.org/wiki/Finite_volume_method

[13]

Full MultiGrid method. https://en.wikipedia.org/wiki/Multigrid_method

[14]

Unified Memory. https://devblogs.nvidia.com/parallelforall/unified-memory-in-cuda-6

[15]

ORNL Titan supercomputer. https://www.olcf.ornl.gov/titan

[16]

Wilkes cluster Cambridge, UK. www.hpc.cam.ac.uk

[17]

ExMaTex. http://www.exmatex.org/comd.html

[18]

CoMD Code. https://github.com/exmatex/CoMD

[19]

CoMD-CUDA Code. https://github.com/NVIDIA/CoMD-CUDA

[20]

Lulesh Website. https://codesign.llnl.gov/lulesh.php

[21]

I. Karlin, J. Keasler, R. Neely LULESH 2.0 Updates and Changes. https://codesign.llnl.gov/lulesh.php, August 2013, pages 1--9

[22]

GPUDirect libmlx5. https://github.com/gpudirect/libmlx5

[23]

GPUDirect libgdsync. https://github.com/gpudirect/libgdsync

[24]

GPUDirect RDMA Considerations. http://docs.nvidia.com/cuda/gpudirectrdma/#design-considerations

Cited By

Issa MSasongko MTurimbetov IBaydamirli JSağbili DUnat D(2024)Snoopie: A Multi-GPU Communication Profiler and VisualizerProceedings of the 38th ACM International Conference on Supercomputing10.1145/3650200.3656597(525-536)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3650200.3656597
Ismayilov IBaydamirli JSağbili DWahib MUnat DGallivan KNikolopoulos DBeivide RGallopoulos E(2023)Multi-GPU Communication Schemes for Iterative Solvers: When CPUs are Not in ChargeProceedings of the 37th International Conference on Supercomputing10.1145/3577193.3593713(192-202)Online publication date: 21-Jun-2023
https://dl.acm.org/doi/10.1145/3577193.3593713
Kwon DBoo JKim DKim JLu SHowell J(2020)FVMProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation10.5555/3488766.3488820(955-971)Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.5555/3488766.3488820
Show More Cited By

Recommendations

Analyzing GPU-controlled communication with dynamic parallelism in terms of performance and energy

Intra-GPU synchronization is a problem for GPU controlled communication.Options, based on dynamic parallelism provide on-device synchronization.GPU controlled communication have a lower performance than CPU assisted approaches.Relieving the CPU from the ...
Energy-efficient stencil computations on distributed GPUs using dynamic parallelism and GPU-controlled communication
E2SC '14: Proceedings of the 2nd International Workshop on Energy Efficient Supercomputing

GPUs are widely used in high performance computing, due to their high computational power and high performance per Watt. Still, one of the main bottlenecks of GPU-accelerated cluster computing is the data transfer between distributed GPUs. This not only ...
The development of Mellanox/NVIDIA GPUDirect over InfiniBand--a new model for GPU to GPU communications

The usage and adoption of General Purpose GPUs (GPGPU) in HPC systems is increasing due to the unparalleled performance advantage of the GPUs and the ability to fulfill the ever-increasing demands for floating points operations. While the GPU can ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CCGrid '17: Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

May 2017

1167 pages

ISBN:9781509066100

Sponsors

Publisher

IEEE Press

Publication History

Published: 14 May 2017

Check for updates

Author Tags

Qualifiers

Tutorial
Research
Refereed limited

Conference

CCGrid '17

Sponsor:

SIGARCH

CCGrid '17: 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

May 14 - 17, 2017

Madrid, Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
172
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Issa MSasongko MTurimbetov IBaydamirli JSağbili DUnat D(2024)Snoopie: A Multi-GPU Communication Profiler and VisualizerProceedings of the 38th ACM International Conference on Supercomputing10.1145/3650200.3656597(525-536)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3650200.3656597
Ismayilov IBaydamirli JSağbili DWahib MUnat DGallivan KNikolopoulos DBeivide RGallopoulos E(2023)Multi-GPU Communication Schemes for Iterative Solvers: When CPUs are Not in ChargeProceedings of the 37th International Conference on Supercomputing10.1145/3577193.3593713(192-202)Online publication date: 21-Jun-2023
https://dl.acm.org/doi/10.1145/3577193.3593713
Kwon DBoo JKim DKim JLu SHowell J(2020)FVMProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation10.5555/3488766.3488820(955-971)Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.5555/3488766.3488820
Ajdari MLee WPark PKim JKim J(2019)FIDRProceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture10.1145/3352460.3358303(239-252)Online publication date: 12-Oct-2019
https://dl.acm.org/doi/10.1145/3352460.3358303
LeBeane MHamidouche KBenton BBreternitz MReinhardt SJohn LEvripidou SStenström PO'Boyle M(2018)ComP-netProceedings of the 27th International Conference on Parallel Architectures and Compilation Techniques10.1145/3243176.3243179(1-13)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1145/3243176.3243179
Kwon DAhn JChae DAjdari MLee JBae SKim YKim J(2018)DCS-ctrlProceedings of the 45th Annual International Symposium on Computer Architecture10.1109/ISCA.2018.00048(491-504)Online publication date: 2-Jun-2018
https://dl.acm.org/doi/10.1109/ISCA.2018.00048
LeBeane MHamidouche KBenton BBreternitz MReinhardt SJohn LMohr BRaghavan P(2017)GPU triggered networking for intra-kernel communicationsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3126908.3126950(1-12)Online publication date: 12-Nov-2017
https://dl.acm.org/doi/10.1145/3126908.3126950

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents