More Web Proxy on the site http://driver.im/

research-article

QMPI: a next generation MPI profiling interface for modern HPC platforms

Authors:

Martin SchulzAuthors Info & Claims

EuroMPI '19: Proceedings of the 26th European MPI Users' Group Meeting

Article No.: 4, Pages 1 - 10

https://doi.org/10.1145/3343211.3343215

Published: 11 September 2019 Publication History

Abstract

As we approach exascale and start planning for beyond, the rising complexity of systems and applications demands new monitoring, analysis, and optimization approaches. This requires close coordination with the parallel programming system used, which for HPC in most cases includes MPI, the Message Passing Interface. While MPI provides comprehensive tool support in the form of the MPI Profiling interface, PMPI, which has inspired a generation of tools, it is not sufficient for the new arising challenges. In particular, it does not support modern software design principles nor the composition of multiple monitoring solutions from multiple agents or sources. We approach these gaps and present QMPI, as a possible successor to PMPI. In this paper, we present the use cases and requirements that drive its development, offer a prototype design and implementation, and demonstrate its effectiveness and low overhead.

References

[1]

A. Netti, M. Müller, A. Auweter, C. Guillen, M. Ott, D. Tafani, and M. Schulz. 2019. From Facility to Application Sensor Data: Modular, Continuous and Holistic Monitoring with DCDB. (Nov 2019), 1--12.

[2]

Jack Dongarra, Michael A Heroux, and Piotr Luszczek. 2016. High-performance conjugate-gradient benchmark: A new metric for ranking high-performance computing systems. The International Journal of High Performance Computing Applications 30, 1 (2016), 3--10.

Digital Library

[3]

Jack Dongarra, Piotr Luszczek, and Michael A Heroux. 2013. HPCG Technical Specification. Sandia National Laboratories. https://prod.sandia.gov/techlib-noauth/access-control.cgi/2013/138752.pdf

[4]

Jonathan Eastep, Steve Sylvester, Christopher Cantalupo, Brad Geltz, Federico Ardanaz, Asma Al-Rawi, Kelly Livingston, Fuat Keceli, Matthias Maiterth, and Siddhartha Jana. 2017. Global Extensible Open Power Manager: A Vehicle for HPC Community Collaboration on Co-Designed Energy Management Solutions. In High Performance Computing, Julian M. Kunkel, Rio Yokota, Pavan Balaji, and David Keyes (Eds.). Springer International Publishing, Cham, 394--412.

[5]

Edgar Gabriel, Graham E. Fagg, George Bosilca, Thara Angskun, Jack J. Dongarra, Jeffrey M. Squyres, Vishal Sahay, Prabhanjan Kambadur, Brian Barrett, Andrew Lumsdaine, Ralph H. Castain, David J. Daniel, Richard L. Graham, and Timothy S. Woodall. 2004. Open MPI: Goals, Concept, and Design of a Next Generation MPI Implementation. In Proceedings, 11th European PVM/MPI Users' Group Meeting. Budapest, Hungary, 97--104.

[6]

Todd Gamblin. 2016. Measuring and Analyzing Entire HPC Centers: The Sonar Project at LLNL. Invited talk at the Tokyo Institute of Technology, Tokyo, Japan.

[7]

Marc-André Hermanns, Nathan T Hjlem, Michael Knobloch, Kathryn Mohror, and Martin Schulz. 2018. Enabling callback-driven runtime introspection via MPI_T. In Proceedings of the 25th European MPI Users' Group Meeting. ACM, 8.

Digital Library

[8]

HPC Advisory Council. 2014. HPCG Performance Benchmark and Profiling. http://www.hpcadvisorycouncil.com/pdf/HPCG_Analysis_and_Profiling.pdf

[9]

IBM Corporation. 2017. IBM Spectrum MPI V10.1 documentation. https://www.ibm.com/support/knowledgecenter/en/SSZTET_10.1/smpi_welcome/smpi_welcome.html. Accessed on 07.05.2019.

[10]

Tanzima Islam, Kathryn Mohror, and Martin Schulz. 2016. Exploring the MPI tool information interface: features and capabilities. The International Journal of High Performance Computing Applications 30, 2 (2016), 212--222.

Digital Library

[11]

Robert Mijaković, Antonio Pimenta Soto, Isaías A Comprés Ureña, Michael Gerndt, Anna Sikora, and Eduardo César. 2014. Specification of periscope tuning framework plugins. (2014), 123--132.

[12]

M. Schulz and B. R. De Supinski. 2006. A Flexible and Dynamic Infrastructure for MPI Tool Interoperability. (Aug 2006), 193--202.

Digital Library

[13]

M. Schulz and B. R. de Supinski. 2007. PNMPI tools: a whole lot greater than the sum of their parts. (Nov 2007), 1--10.

Digital Library

[14]

M. Schulz S.Rasmussen and K. Mohror. 2016. Allowing MPI tools builders to forget about Fortran. (2016), 208--211.

Digital Library

[15]

The Ohio State University. 2018. MVAPICH: MPI over InfiniBand, Omni-Path, Ethernet/iWARP, and RoCE. http://mvapich.cse.ohio-state.edu/benchmarks/. Accessed on 07.05.2019.

[16]

Ulrike Yang, Robert Falgout, and Jongsoo Park. 2017. Algebraic Multigrid Benchmark, Version 00. https://www.osti.gov//servlets/purl/1389816. Accessed on 07.05.2019.

Cited By

Tarraf ASchreiber MCascajo ABesnard JVef MHuber DHapp SBrinkmann ASingh DHoppe HMiranda APeña AMachado RGarcia-Gasulla MSchulz MCarpenter PPickartz SRotaru TIserte SLopez VEjarque JSirwani HCarretero JWolf F(2024)Malleability in Modern HPC Systems: Current Experiences, Challenges, and Future OpportunitiesIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2024.340676435:9(1551-1564)Online publication date: Sep-2024
https://doi.org/10.1109/TPDS.2024.3406764
Hammond JDalcin LSchnetter EPéRache MBesnard JBrown JGadeschi GByrne SSchuchart JZhou H(2023)MPI Application Binary Interface StandardizationProceedings of the 30th European MPI Users' Group Meeting10.1145/3615318.3615319(1-12)Online publication date: 11-Sep-2023
https://dl.acm.org/doi/10.1145/3615318.3615319
Kousha PSankarapandian Dayala Ganesh Ram KKedia MSubramoni HJain AShafi APanda DDockendorf TNa HTomko K(2021)INAM: Cross-stack Profiling and Analysis of Communication in MPI-based ApplicationsPractice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions10.1145/3437359.3465582(1-11)Online publication date: 17-Jul-2021
https://dl.acm.org/doi/10.1145/3437359.3465582
Show More Cited By

Index Terms

QMPI: a next generation MPI profiling interface for modern HPC platforms
1. Computing methodologies
  1. Parallel computing methodologies
    1. Parallel programming languages

Recommendations

Problems of Parallel Realization of Algorithms Calculation of Electronic Structure of Large Molecules
DEPCOS-RELCOMEX '08: Proceedings of the 2008 Third International Conference on Dependability of Computer Systems DepCoS-RELCOMEX

The problem of the organization of effective computations for the calculation of electronic structure of large molecules using self-consistent neglect differential overlap purification method is described. A mathematical model of problem is presented. ...
OpenMP for Networks of SMPs

In this paper, we present the first system that implements OpenMP on a network of shared-memory multiprocessors. This system enables the programmer to rely on a single, standard, shared-memory API for parallelization within a multiprocessor and between ...
SPMD OpenMP versus MPI on a IBM SMP for 3 Kernels of the NAS Benchmarks
ISHPC '02: Proceedings of the 4th International Symposium on High Performance Computing

Shared Memory Multiprocessors are becoming more popular since they are used to deploy large parallel computers. The current trend is to enlarge the number of processors inside such multiprocessor nodes. However a lot of existing applications are using ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

EuroMPI '19: Proceedings of the 26th European MPI Users' Group Meeting

September 2019

134 pages

ISBN:9781450371759

DOI:10.1145/3343211

Editors:
Torsten Hoefler
ETH, Zürich
,
Jesper Larsson Träff
TU Wien, Austria

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 September 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

German Federal Ministry for Education and Research

Conference

EuroMPI 2019

EuroMPI 2019: 26th European MPI Users' Group Meeting

September 11 - 13, 2019

Zürich, Switzerland

Acceptance Rates

EuroMPI '19 Paper Acceptance Rate 13 of 26 submissions, 50%;

Overall Acceptance Rate 66 of 139 submissions, 47%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
214
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)1

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tarraf ASchreiber MCascajo ABesnard JVef MHuber DHapp SBrinkmann ASingh DHoppe HMiranda APeña AMachado RGarcia-Gasulla MSchulz MCarpenter PPickartz SRotaru TIserte SLopez VEjarque JSirwani HCarretero JWolf F(2024)Malleability in Modern HPC Systems: Current Experiences, Challenges, and Future OpportunitiesIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2024.340676435:9(1551-1564)Online publication date: Sep-2024
https://doi.org/10.1109/TPDS.2024.3406764
Hammond JDalcin LSchnetter EPéRache MBesnard JBrown JGadeschi GByrne SSchuchart JZhou H(2023)MPI Application Binary Interface StandardizationProceedings of the 30th European MPI Users' Group Meeting10.1145/3615318.3615319(1-12)Online publication date: 11-Sep-2023
https://dl.acm.org/doi/10.1145/3615318.3615319
Kousha PSankarapandian Dayala Ganesh Ram KKedia MSubramoni HJain AShafi APanda DDockendorf TNa HTomko K(2021)INAM: Cross-stack Profiling and Analysis of Communication in MPI-based ApplicationsPractice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions10.1145/3437359.3465582(1-11)Online publication date: 17-Jul-2021
https://dl.acm.org/doi/10.1145/3437359.3465582
Lin FLiu YGuo YQian D(2020)ELS: Emulation system for debugging and tuning large-scale parallel programs on small clustersThe Journal of Supercomputing10.1007/s11227-020-03319-6Online publication date: 23-May-2020
https://doi.org/10.1007/s11227-020-03319-6

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents