More Web Proxy on the site http://driver.im/

Article

Performance Comparison of MPI Implementations over InfiniBand, Myrinet and Quadrics

Authors:

Balasubramanian Chandrasekaran,

Sushmitha Kini,

Darius Buntinas,

D K. PandaAuthors Info & Claims

SC '03: Proceedings of the 2003 ACM/IEEE conference on Supercomputing

Page 58

https://doi.org/10.1145/1048935.1050208

Published: 15 November 2003 Publication History

Abstract

In this paper, we present a comprehensive performance comparison of MPI implementations over Infini-Band, Myrinet and Quadrics. Our performance evaluation consists of two major parts. The first part consists of a set of MPI level micro-benchmarks that characterize different aspects of MPI implementations. The second part of the performance evaluation consists of application level benchmarks. We have used the NAS Parallel Benchmarks and the sweep3D benchmark. We not only present the overall performance results, but also relate application communication characteristics to the information we acquired from the micro-benchmarks. Our results show that the three MPI implementations all have their advantages and disadvantages. For our 8-node cluster, InfiniBand can offer significant performance improvements for a number of applications compared with Myrinet and Quadrics when using the PCI-X bus. Even with just the PCI bus, InfiniBand can still perform better if the applications are bandwidth-bound.

References

[1]

{1} C. Bell, D. Bonachea, Y. Cote, J. Duell, P. Hargrove, P. Husbands, C. Iancu, M. Welcome, and K. Yelick. An Evaluation of Current High-Performance Networks. In International Parallel and Distributed Processing Symposium (IPDPS'03), April 2003.

Digital Library

[2]

{2} B. Chandrasekaran, P. Wyckoff, and D. K. Panda. MIBA: A Micro-benchmark Suite for Evaluating InfiniBand Architecture Implementations. In Performance TOOLS 2003, September 2003.

[3]

{3} D. E. Culler, R. M. Karp, D. A. Patterson, A. Sahay, K. E. Schauser, E. Santos, R. Subramonian, and T. von Eicken. Logp: Towards a realistic model of parallel computation. In Principles Practice of Parallel Programming, pages 1-12, 1993.

Digital Library

[4]

{4} W. Gropp, E. Lusk, N. Doss, and A. Skjellum. A high-performance, portable implementation of the MPI message passing interface standard. Parallel Computing, 22(6):789- 828, 1996.

Digital Library

[5]

{5} A. Hoisie, O. Lubeck, H. Wasserman, F. Petrini, and H. Alme. A General Predictive Performance Model for Wavefront Algorithms on Cluster of SMPs. In ICPP 2000, 2000.

Digital Library

[6]

{6} J. Hsieh, T. Leng, V. Mashayekhi, and R. Rooholamini. Architectural and performance evaluation of giganet and myrinet interconnects on clusters of small-scale SMP servers. In Supercomputing, 2000.

Digital Library

[7]

{7} InfiniBand Trade Association. InfiniBand Architecture Specification, Release 1.0, October 24 2000.

[8]

{8} S. Kini, J. Liu, J. Wu, P. Wyckoff, and D. K. Panda. Fast and scalable barrier using rdma and multicast mechanisms for infiniband-based clusters. In EuroPVM/MPI 2003, September 2003.

[9]

{9} J. Liu, B. Chandrasekaran, W. Yu, J. Wu, D. Buntinas, S. P. Kini, P. Wyckoff, and D. K. Panda. Micro-benchmark level performance comparison of high-speed cluster interconnects. In Hot Interconnects 11, August 2003.

[10]

{10} J. Liu, J. Wu, S. P. Kini, P. Wyckoff, and D. K. Panda. High Performance RDMA-Based MPI Implementation over InfiniBand. In 17th Annual ACM International Conference on Supercomputing (ICS '03), June 2003.

Digital Library

[11]

{11} R. Martin, A. Vahdat, D. Culler, and T. Anderson. Effects of Communication Latency, Overhead, and Bandwidth in a Cluster Architecture. In Proceedings of the International Symposium on Computer Architecture, 1997.

Digital Library

[12]

{12} Mellanox Technologies. Mellanox InfiniBand InfiniHost Adapters, July 2002.

[13]

{13} Myricom. Myricom, Inc. http://www.myri.com.

[14]

{14} NASA. NAS Parallel Benchmarks. http://www.nas.nasa.gov/Software/NPB.

[15]

{15} Network-Based Computing Lab, The Ohio State University. MVAPICH: MPI for InfiniBand over VAPI Layer. http://nowlab.cis.ohio-state.edu/projects/mpi-iba/, June 2003.

[16]

{16} N. J. Boden, D. Cohen, R. E. Felderman, A. E. Kulawik, C. L. Seitz, J. N. Seizovic, and W. Su. Myrinet: A Gigabit-persecond Local Area Network. IEEE Micro, 15(1):29-36, February 1995.

Digital Library

[17]

{17} Pallas. Pallas MPI Benchmarks. http://www.pallas.com/e/products/pmb/.

[18]

{18} F. Petrini, W. Feng, A. Hoisie, S. Coll, and E. Frachtenberg. The Quadrics Network: High-Performance Clustering Technology. IEEE Micro, 22(1):46-57, 2002.

Digital Library

[19]

{19} Quadrics. Quadrics, Ltd. http://www.quadrics.com.

[20]

{20} M. Snir, S. Otto, S. Huss-Lederman, D. Walker, and J. Dongarra. MPI-The Complete Reference. Volume 1 - The MPI-1 Core, 2nd edition. The MIT Press, 1998.

Digital Library

[21]

{21} T. Tabe and Q. F. Stout. The use of the MPI communication library in the NAS parallel benchmarks. Technical Report CSE-TR-386-99, University of Michgan, 1999.

[22]

{22} H. Tezuka, F. O'Carroll, A. Hori, and Y. Ishikawa. Pin-down cache: A virtual memory management technique for zero-copy communication. In In 12th Int. Parallel Processing Symposium, pages 308-315, March 1998.

Digital Library

[23]

{23} Topspin. Topspin InfiniBand Products. http://www.topspin.com.

[24]

{24} J. S. Vetter and F. Mueller. Communication Characteristics of Large-Scale Scientific Applications for Contemporary Cluster Architectures. In IPDPS 02, April 2002.

Digital Library

[25]

{25} F. C. Wong, R. P. Martin, R. H. Arpaci-Dusseau, and D. E. Culler. Architectural requirements and scalability of the nas parallel benchmarks. In In the Proceedings of Supercomputing'99 , 1999.

Digital Library

[26]

{26} J. Wu, J. Liu, P. Wyckoff, and D. K. Panda. Impact of On-Demand Connection Management in MPI over VIA. In Proceedings of the IEEE International Conference on Cluster Computing, 2002.

Digital Library

Cited By

Zhuang JJacob DLin HLundgren EYantosca RGaya JSulprizio MEastham S(2020)Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry ModelJournal of Advances in Modeling Earth Systems10.1029/2020MS00206412:5Online publication date: 6-May-2020
https://doi.org/10.1029/2020MS002064
Jammer TIwainsky CBischof C(2020)A Comparison of the Scalability of OpenMP ImplementationsEuro-Par 2020: Parallel Processing10.1007/978-3-030-57675-2_6(83-97)Online publication date: 24-Aug-2020
https://dl.acm.org/doi/10.1007/978-3-030-57675-2_6
Stegmeier AFrieb MMische JUngerer TOuhammou YRidouard FGrolleau EJan MBehnam M(2018)Analysing Real-Time Behaviour of Collective Communication Patterns in MPIProceedings of the 26th International Conference on Real-Time Networks and Systems10.1145/3273905.3273906(137-147)Online publication date: 10-Oct-2018
https://dl.acm.org/doi/10.1145/3273905.3273906
Show More Cited By

Recommendations

High performance RDMA-based MPI implementation over InfiniBand
ICS '03: Proceedings of the 17th annual international conference on Supercomputing

Although InfiniBand Architecture is relatively new in the high performance computing area, it offers many features which help us to improve the performance of communication subsystems. One of these features is Remote Direct Memory Access (RDMA) ...
Using the Intel MPI benchmarks (IMB) to evaluate MPI implementations on an Infiniband Nehalem Linux cluster
SpringSim '10: Proceedings of the 2010 Spring Simulation Multiconference

According to Moore's Law, computer speeds are expected to double approximately every 2 years. But with the current challenges that computer manufacturers are facing to double speeds of individual processors, due to various reasons, i.e., processor ...
High performance RDMA-based MPI implementation over infiniBand
Special issue I: The 17th annual international conference on supercomputing (ICS'03)

Although InfiniBand Architecture is relatively new in the high performance computing area, it offers many features which help us to improve the performance of communication subsystems. One of these features is Remote Direct Memory Access (RDMA) ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '03: Proceedings of the 2003 ACM/IEEE conference on Supercomputing

November 2003

859 pages

ISBN:1581136951

DOI:10.1145/1048935

General Chair:
James R. McGraw

Copyright © 2003 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 November 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

SC '03

Sponsor:

SIGARCH
IEEE-CS

SC '03: International Conference for High Performance Computing, Networking, Storage and Analysis

November 15 - 21, 2003

AZ, Phoenix, USA

Acceptance Rates

SC '03 Paper Acceptance Rate 60 of 207 submissions, 29%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

82
Total Citations
View Citations
739
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhuang JJacob DLin HLundgren EYantosca RGaya JSulprizio MEastham S(2020)Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry ModelJournal of Advances in Modeling Earth Systems10.1029/2020MS00206412:5Online publication date: 6-May-2020
https://doi.org/10.1029/2020MS002064
Jammer TIwainsky CBischof C(2020)A Comparison of the Scalability of OpenMP ImplementationsEuro-Par 2020: Parallel Processing10.1007/978-3-030-57675-2_6(83-97)Online publication date: 24-Aug-2020
https://dl.acm.org/doi/10.1007/978-3-030-57675-2_6
Stegmeier AFrieb MMische JUngerer TOuhammou YRidouard FGrolleau EJan MBehnam M(2018)Analysing Real-Time Behaviour of Collective Communication Patterns in MPIProceedings of the 26th International Conference on Real-Time Networks and Systems10.1145/3273905.3273906(137-147)Online publication date: 10-Oct-2018
https://dl.acm.org/doi/10.1145/3273905.3273906
Bochenina KKesarev SBoukhanovsky A(2018)Scalable parallel simulation of dynamical processes on large stochastic Kronecker graphsFuture Generation Computer Systems10.1016/j.future.2017.07.02178(502-515)Online publication date: Jan-2018
https://doi.org/10.1016/j.future.2017.07.021
Gomez-Folgar FIndalecio GSeoane NPena TGarcia-Loureiro A(2018)MPI-Performance-Aware-ReallocationComputing10.1007/s00607-017-0573-6100:2(211-226)Online publication date: 1-Feb-2018
https://dl.acm.org/doi/10.1007/s00607-017-0573-6
Breuer AHeinecke ACui Y(2017)EDGE: Extreme Scale Fused Seismic Simulations with the Discontinuous Galerkin MethodHigh Performance Computing10.1007/978-3-319-58667-0_3(41-60)Online publication date: 18-Jun-2017
https://dl.acm.org/doi/10.1007/978-3-319-58667-0_3
Knight JFurber S(2016)Synapse-Centric Mapping of Cortical Models to the SpiNNaker Neuromorphic ArchitectureFrontiers in Neuroscience10.3389/fnins.2016.0042010Online publication date: 14-Sep-2016
https://doi.org/10.3389/fnins.2016.00420
Almeida EBauer MFazenda A(2016)Numerical weather model BRAMS evaluation on many-core architecturesInternational Journal of Computational Science and Engineering10.1504/IJCSE.2016.07694012:4(330-340)Online publication date: 1-Jan-2016
https://dl.acm.org/doi/10.1504/IJCSE.2016.076940
Dong BLi XXiao LRuan L(2014)Towards minimizing disk I/O contention: A partitioned file assignment approachFuture Generation Computer Systems10.1016/j.future.2013.12.02237(178-190)Online publication date: Jul-2014
https://doi.org/10.1016/j.future.2013.12.022
Al-Mulhem MAl-Shaikh R(2013)Performance modelling of parallel BLAST using Intel and PGI compilers on an infiniband-based HPC clusterInternational Journal of Bioinformatics Research and Applications10.1504/IJBRA.2013.0560869:5(534-546)Online publication date: 1-Aug-2013
https://dl.acm.org/doi/10.1504/IJBRA.2013.056086
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents