research-article

Understanding I/O Bottlenecks and Tuning for High Performance I/O on Large HPC Systems: A Case Study

Authors:

Manu Shantharam,

Mahidhar Tatineni,

Dongju Choi,

Amitava MajumdarAuthors Info & Claims

PEARC '18: Proceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity

Article No.: 54, Pages 1 - 6

https://doi.org/10.1145/3219104.3219120

Published: 22 July 2018 Publication History

Get Access

Abstract

As we move towards peta-to-exascale machines, large-scale physics based simulations are expected to generate large amount of I/O traffic based on unprecedented growth in the volume and types of data. It is imperative to understand and characterize the I/O behavior of scientific applications, including complex checkpoint/restart options, on different hardware-software configurations including large shared parallel file systems, node local flash, and burst buffer technologies, to tune and improve the overall application performance. In this work, we study the I/O behavior of WRF, a widely used scientific application for atmospheric research and operational weather forecasting, on high performance computing systems. WRF provides a rich collection of I/O strategies such as using different parallel I/O libraries (PnetCDF, NetCDF) and I/O quilting options with these libraries, as well as configurable I/O "knobs" that can be used to modify the I/O frequency. We evaluate the effectiveness of using various I/O strategies within WRF in conjunction with parallel file system parameter tuning on Comet and Stampede2 HPC systems. We discuss the impact of using various parallel I/O strategies and further show the use of an I/O profiling tool to analyze an anomalous parallel I/O behavior. Overall, we provide a discussion on tuning and performance insights gained from our evaluations.

References

[1]

{n. d.}. WRF CONUS 2.5km benchmark.

Google Scholar

[2]

Tricia Balle and Pete Johnsen. 2016. Improving i/o performance of the weather research and forecast (WRF) model. In Cray User Group.

Google Scholar

[3]

Richard L. Moore, Chaitan Baru, Diane Baxter, Geoffrey C. Fox, Amit Majumdar, Phillip Papadopoulos, Wayne Pfeiffer, Robert S. Sinkovits, Shawn Strande, Mahidhar Tatineni, Richard P. Wagner, Nancy Wilkins-Diehr, and Michael L. Norman. 2014. Gateways to Discovery: Cyberinfrastructure for the Long Tail of Science. In Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment (XSEDE '14). ACM, New York, NY, USA, Article 39, 8 pages.

Digital Library

Google Scholar

[4]

Andrew Porter and M Ashworth. 2010. Configuring and optimizing the weather research and forecast model on the CRAY XT. In Cray User Group Proceedings.

Google Scholar

[5]

W. C. Skamarock, J. B. Klemp, J. Dudhia, D. O. Gill, D. M. Barker, M. G Duda, X.-Y. Huang, W. Wang, and J. G. Powers. 2008. A Description of the Advanced Research WRF Version 3. Technical Report.

Google Scholar

[6]

Dan Stanzione, Bill Barth, Niall Gaffney, Kelly Gaither, Chris Hempel, Tommy Minyard, S. Mehringer, Eric Wernert, H. Tufo, D. Panda, and P. Teller. 2017. Stampede 2: The Evolution of an XSEDE Supercomputer. In Proceedings of the Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and Impact (PEARC17). ACM, New York, NY, USA, Article 15, 8 pages.

Digital Library

Google Scholar

[7]

W. Wang, X. Huang, H. Fu, Y. Hu, S. Xu, and G. Yang. 2013. CFIO: A Fast I/O Library for Climate Models. In 2013 12th IEEE International Conference on Trust, Security and Privacy in Computing and Communications. 911--918.

Digital Library

Google Scholar

[8]

Cong Xu, Shane Snyder, Omkar Kulkarni, Vishwanath Venkatesan, Philip Carns, Suren Byna, Robert Sisneros, and Kalyana Chadalavada. 2017. DXT: Darshan Extended Tracing. In Cray User Group Proceedings.

Google Scholar

Cited By

View all

Pang RYu FZhang YYuan Y(2023)An Asynchronous Parallel I/O Framework for Mass Conservation Ocean ModelApplied Sciences10.3390/app13241323013:24(13230)Online publication date: 13-Dec-2023
https://doi.org/10.3390/app132413230
Huang ZHou KAgrawal AChoudhary ARoss RLiao WMohror KArnold DBadia R(2023)I/O in WRF: A Case Study in Modern Parallel I/O TechniquesProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3581784.3613216(1-13)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3581784.3613216
Bez JKarimi APaul AXie BByna SCarns POral SWang FHanley JWeissman JChandra AGavrilovska ATiwari D(2022)Access Patterns and Performance Behaviors of Multi-layer Supercomputer I/O Subsystems under Production LoadProceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing10.1145/3502181.3531461(43-55)Online publication date: 27-Jun-2022
https://dl.acm.org/doi/10.1145/3502181.3531461
Show More Cited By

Index Terms

Understanding I/O Bottlenecks and Tuning for High Performance I/O on Large HPC Systems: A Case Study
1. Computing methodologies
  1. Parallel computing methodologies
    1. Parallel algorithms

Recommendations

Understanding Parallel I/O Performance Trends Under Various HPC Configurations
SNTA '19: Proceedings of the ACM Workshop on Systems and Network Telemetry and Analytics

In high-performance computing (HPC) environments, an appropriate amount of hardware resources must be used for the best parallel I/O performance. For this reason, HPC users are provided with tunable parameters to change the HPC configurations, which ...
Understanding Parallel I/O Performance and Tuning
SNTA '22: Fifth International Workshop on Systems and Network Telemetry and Analytics

Performance of parallel I/O is critical for large-scale scientific applications to store and access data from parallel file systems on high-performance computing (HPC) systems. These applications use HPC systems often to generate and analyze large ...
Dynamic Model-Driven Parallel I/O Performance Tuning
CLUSTER '15: Proceedings of the 2015 IEEE International Conference on Cluster Computing

Parallel I/O performance depends highly on the interactions among multiple layers of the parallel I/O stack. The most common layers include high-level I/O libraries, MPI-IO middleware, and parallel file system. Each of these layers offers various ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

PEARC '18: Proceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity

July 2018

652 pages

ISBN:9781450364461

DOI:10.1145/3219104

General Chair:
Sergiu Sanielevici
Pittsburgh Supercomputing Center

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 July 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

PEARC '18

PEARC '18: Practice and Experience in Advanced Research Computing

July 22 - 26, 2018

PA, Pittsburgh, USA

Acceptance Rates

PEARC '18 Paper Acceptance Rate 79 of 123 submissions, 64%;

Overall Acceptance Rate 133 of 202 submissions, 66%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
261
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)1

Reflects downloads up to 31 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Pang RYu FZhang YYuan Y(2023)An Asynchronous Parallel I/O Framework for Mass Conservation Ocean ModelApplied Sciences10.3390/app13241323013:24(13230)Online publication date: 13-Dec-2023
https://doi.org/10.3390/app132413230
Huang ZHou KAgrawal AChoudhary ARoss RLiao WMohror KArnold DBadia R(2023)I/O in WRF: A Case Study in Modern Parallel I/O TechniquesProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3581784.3613216(1-13)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3581784.3613216
Bez JKarimi APaul AXie BByna SCarns POral SWang FHanley JWeissman JChandra AGavrilovska ATiwari D(2022)Access Patterns and Performance Behaviors of Multi-layer Supercomputer I/O Subsystems under Production LoadProceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing10.1145/3502181.3531461(43-55)Online publication date: 27-Jun-2022
https://dl.acm.org/doi/10.1145/3502181.3531461
Woo MJordan TNandi TDietiker JGuenther CVan Essendelft D(2022)Development of an equation-based parallelization method for multiphase particle-in-cell simulationsEngineering with Computers10.1007/s00366-022-01768-639:5(3577-3591)Online publication date: 22-Dec-2022
https://dl.acm.org/doi/10.1007/s00366-022-01768-6
Giabbanelli PDevita JKöster TKohrt JLiu JGiabbanelli PCarothers C(2020)Optimizing Discrete Simulations of the Spread of HIV-1 to Handle Billions of Cells on a WorkstationProceedings of the 2020 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation10.1145/3384441.3395987(67-78)Online publication date: 15-Jun-2020
https://dl.acm.org/doi/10.1145/3384441.3395987

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Understanding Parallel I/O Performance Trends Under Various HPC Configurations

Understanding Parallel I/O Performance and Tuning

Dynamic Model-Driven Parallel I/O Performance Tuning