Abstract
This paper aims to create a transition path from file-based IO to streaming-based workflows for scientific applications in an HPC environment. By using the openPMP-api, traditional workflows limited by filesystem bottlenecks can be overcome and flexibly extended for in situ analysis. The openPMD-api is a library for the description of scientific data according to the Open Standard for Particle-Mesh Data (openPMD). Its approach towards recent challenges posed by hardware heterogeneity lies in the decoupling of data description in domain sciences, such as plasma physics simulations, from concrete implementations in hardware and IO. The streaming backend is provided by the ADIOS2 framework, developed at Oak Ridge National Laboratory. This paper surveys two openPMD-based loosely-coupled setups to demonstrate flexible applicability and to evaluate performance. In loose coupling, as opposed to tight coupling, two (or more) applications are executed separately, e.g. in individual MPI contexts, yet cooperate by exchanging data. This way, a streaming-based workflow allows for standalone codes instead of tightly-coupled plugins, using a unified streaming-aware API and leveraging high-speed communication infrastructure available in modern compute clusters for massive data exchange. We determine new challenges in resource allocation and in the need of strategies for a flexible data distribution, demonstrating their influence on efficiency and scaling on the Summit compute system. The presented setups show the potential for a more flexible use of compute resources brought by streaming IO as well as the ability to increase throughput by avoiding filesystem bottlenecks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
"QueueFullPolicy" = "Discard" The alternative is to block. A queue of steps can be held for some additional leeway, but it requires additional memory.
- 6.
After recent successful streaming setups at 1024 nodes, the likely cause for this were scalability issues in the metadata strategy used in the openPMD-api.
References
Abbasi, H., Wolf, M., Eisenhauer, G., Klasky, S., Schwan, K., Zheng, F.: Datastager: scalable data staging services for petascale applications. Cluster Comput. 13(3), 277–290 (2010). https://doi.org/10.1145/1551609.1551618
Ahn, D.H., Garlick, J., Grondona, M., Lipari, D., Springmeyer, B., Schulz, M.: Flux: a next-generation resource management framework for large HPC centers. In: 2014 43rd International Conference on Parallel Processing Workshops, pp. 9–17 (2014). https://doi.org/10.1109/ICPPW.2014.15
Bussmann, M., et al.: Radiative signatures of the relativistic Kelvin-Helmholtz instability. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC 2013, pp. 5:1–5:12. ACM, New York (2013). ISBN 978-1-4503-2378-9, https://doi.org/10.1145/2503210.2504564
Rocklin, M.: Dask: parallel computation with blocked algorithms and task scheduling. In: Huff, K., Bergstra, J. (eds.) Proceedings of the 14th Python in Science Conference, pp. 130–136 (2015). https://dask.org
E, J.C., Wang, L., Chen, S., Zhang, Y.Y., Luo, S.N.: GAPD: a GPU-accelerated atom-based polychromatic diffraction simulation code. J. Synchrotron Radiat. 25(2), 604–611 (2018). https://doi.org/10.1107/S1600577517016733
Fortmann-Grote, C., et al.: SIMEX: simulation of experiments at advanced light sources. In: 11th NOBUGS Conference, Copenhagen, pp. 29–34 (2016). https://doi.org/10.17199/NOBUGS2016.21
Fortmann-Grote, C., et al.: Simulations of ultrafast X–ray laser experiments. In: Proceedings of SPIE, Advances in X-ray Free-Electron Lasers Instrumentation IV, vol. 10237, p. 102370S (2017). https://doi.org/10.1117/12.2270552
Godoy, W.F., et al.: ADIOS 2: the adaptable input output system. A framework for high-performance data management. SoftwareX 12, 100561 (2020). https://doi.org/10.1016/j.softx.2020.100561
Huebl, A.: PIConGPU: predictive simulations of laser-particle accelerators with manycore hardware. Ph.D. thesis, Technische Universität Dresden (2019). https://doi.org/10.5281/zenodo.3266820
Huebl, A., et al.: On the scalability of data reduction techniques in current and upcoming HPC systems from an application perspective. In: Kunkel, J.M., Yokota, R., Taufer, M., Shalf, J. (eds.) ISC High Performance 2017. LNCS, vol. 10524, pp. 15–29. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67630-2_2
Huebl, A., et al.: openPMD: a meta data standard for particle and mesh based data (2015). https://doi.org/10.5281/zenodo.591699, https://github.com/openPMD
Johnson, D.: Near-optimal bin packing algorithms. Ph.D. thesis, Massachusetts Institute of Technology (1973)
Koller, F., Poeschel, F., Gu, J., Huebl, A.: openPMD-api: C++ & Python API for scientific I/O with openPMD (2018). https://github.com/openPMD/openPMD-api. https://doi.org/10.14278/rodare.27
Kress, J., et al.: Comparing time-to-solution for in situ visualization paradigms at scale. In: 2020 IEEE 10th Symposium on Large Data Analysis and Visualization (LDAV), pp. 22–26 (2020). https://doi.org/10.1109/LDAV51489.2020.00009
Larsen, M., et al.: The alpine in situ infrastructure: ascending from the ashes of strawman. In: Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization, pp. 42–46 (2017). https://doi.org/10.1145/3144769.3144778
Liu, Q., et al.: Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks. Concurr. Comput.: Pract. Exp. 26(7), 1453–1473 (2014). https://doi.org/10.1002/cpe.3125
Logan, J., et al.: Extending the publish/subscribe abstraction for high-performance I/O and data management at extreme scale. IEEE Data Eng. Bull. 43, 35–46 (2020)
Loring, B., et al.: Improving performance of M-to-N processing and data redistribution in In Transit analysis and visualization. In: Frey, S., Huang, J., Sadlo, F. (eds.) Eurographics Symposium on Parallel Graphics and Visualization. The Eurographics Association (2020). ISBN 978-3-03868-107-6, https://doi.org/10.2312/pgv.20201073
Mayes, C.E., et al.: Lightsource unified modeling environment (LUME) - a start-to-end simulation ecosystem. In: IPAC (2021)
Poeschel, F., et al.: Supplementary material: transitioning from file-based HPC workflows to streaming data pipelines with openPMD and ADIOS2. https://doi.org/10.5281/zenodo.4906276
Vazhkudai, S.S., et al.: The design, deployment, and evaluation of the coral pre-exascale systems. In: SC18: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 661–672 (2018). https://doi.org/10.1109/SC.2018.00055
Wan, L., et al.: Improving I/O performance for exascale applications through online data layout reorganization. Under review (2021)
Wilkinson, M.D., et al.: The FAIR Guiding Principles for scientific data management and stewardship. Sci. Data 3(1), 160018 (2016). ISSN 2052-4463, https://doi.org/10.1038/sdata.2016.18
Zaharia, M., et al.: Apache spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016). ISSN 0001-0782, https://doi.org/10.1145/2934664
Acknowledgements
This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. Supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of two U.S. Department of Energy organizations (Office of Science and the National Nuclear Security Administration). Supported by EC through Laserlab-Europe, H2020 EC-GA 871124. Supported by the Consortium for Advanced Modeling of Particles Accelerators (CAMPA), funded by the U.S. DOE Office of Science under Contract No. DE-AC02-05CH11231. This work was partially funded by the Center of Advanced Systems Understanding (CASUS), which is financed by Germany’s Federal Ministry of Education and Research (BMBF) and by the Saxon Ministry for Science, Culture and Tourism (SMWK) with tax funds on the basis of the budget approved by the Saxon State Parliament.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Poeschel, F. et al. (2022). Transitioning from File-Based HPC Workflows to Streaming Data Pipelines with openPMD and ADIOS2. In: Nichols, J., et al. Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation. SMC 2021. Communications in Computer and Information Science, vol 1512. Springer, Cham. https://doi.org/10.1007/978-3-030-96498-6_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-96498-6_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-96497-9
Online ISBN: 978-3-030-96498-6
eBook Packages: Computer ScienceComputer Science (R0)