Abstract
Govinderaju et al. [1] have shown that a hybrid CPU-GPU system is cost-performance effective at sorting large datasets on a single node, but thus far large clusters used on sorting benchmarks have been limited by network and storage performance, and such clusters have remained CPU-only. With network and storage bandwidths improving more rapidly than CPU throughput performance, the cost effectiveness of CPU-GPU clusters for large sorts should be re-examined. As a first step, we evaluate sort performance on a single GPU-accelerated node with initial and final data residing in system memory. Access to main memory is limited to two reads and two writes, while executing the partitioning and sort in GPU memory. On a dual-socket IBM POWER9 system with four NVlink-attached NVIDIA V100 GPUs a single-node sort of 64 GB 8-byte key, 8-byte value records completes in under 2.3 s corresponding to a sort rate of over 28 GB/s. On a small (4-node) cluster with the same amount of data per node, the cluster sort completes in under 4.5 s. Sort performance is enabled by high system memory bandwidth, managing system-memory NUMA affinities, high CPU-GPU bandwidth, an efficient GPU-based partitioner, and an optimized GPU sort implementation. A cluster version of the algorithm benefits from minimizing copy operations by using RDMA. Matching the throughput of an optimized partitioner for our system would require a 50-100 GB/s network, which is feasible with a dual-socket POWER9 system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Govindaraju, N., Gray, J., Kumar, R., Manocha, D.: Gputerasort: high performance graphics co-processor sorting for large database management. In: SIGMOD (2006)
Parallel Integer Sort/BigSort. https://asc.llnl.gov/coral-2-benchmarks/
Kruger, F.: CPU Bandwidth, the Worrisome 2020 Trend, March 2016. https://blog.westerndigital.com/cpu-bandwidth-the-worrisome-2020-trend/. Accessed Feb 2018
Jiang, J., et al.: Tencent Sort. http://www.sortbenchmark.org/TencentSort2016.pdf. Accessed Dec 2017
Satish, N., Harris, M., Garland, M.: Designing efficient sorting algorithms for manycore gpus. In: IEEE International Symposium on Parallel and Distributed Processing, IPDPS 2009. IEEE, pp. 1–10 (2009)
Arkhipov, D.I., Wu, D., Li, K., Regan, A.C.: Sorting with GPUs: a survey. arXiv.org 1709.02520, September 2017. (www.arxiv.org/pdf/1709.02520)
Stehle, E., Jacobsen, H.-A.: A memory bandwidth-efficient hybrid radix sort on GPUs. In: Proceedings of the 2017 ACM International Conference on Management of Data, pp. 417–432, ACM, New York (2017)
Hoberock, J., Bell, N.: Thrust: A Parallel Template Library (2016). https://thrust.github.io
Cho, M., Brand, D., Bordawekar, R., Finkler, U., Kulandaisamy, V., Puri, R.: Paradis: an efficient parallel algorithm for in-place radix sort. In: PVLDB (2015)
Acknowledgements
The authors would like to thank Mark Nutter for discussions early-on in the project. We thank Bruce D’Amora for his support and feedback, and we want to thank the anonymous reviewers of an earlier version of this paper for their extensive feedback that contributed to significant improvements in this version of the paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Fossum, G.C., Wang, T., Hofstee, H.P. (2018). A 64-GB Sort at 28 GB/s on a 4-GPU POWER9 Node for Uniformly-Distributed 16-Byte Records with 8-Byte Keys. In: Yokota, R., Weiland, M., Shalf, J., Alam, S. (eds) High Performance Computing. ISC High Performance 2018. Lecture Notes in Computer Science(), vol 11203. Springer, Cham. https://doi.org/10.1007/978-3-030-02465-9_25
Download citation
DOI: https://doi.org/10.1007/978-3-030-02465-9_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-02464-2
Online ISBN: 978-3-030-02465-9
eBook Packages: Computer ScienceComputer Science (R0)