research-article

Public Access

Developing Accurate Slurm Simulator

Authors:

William R. MathiasAuthors Info & Claims

PEARC '22: Practice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You

Article No.: 59, Pages 1 - 4

https://doi.org/10.1145/3491418.3535178

Published: 08 July 2022 Publication History

All formats PDF

Abstract

A new Slurm simulator compatible with the latest Slurm version has been produced. It was constructed by systematically transforming the Slurm code step by step to maintain the proper scheduler output realization while speeding up simulation time. To test this simulator, a container-based Virtual Cluster was generated which fully mimicked a production HPC cluster. As for all Slurm simulators, the realization is a stochastic process dependent on the computational hardware. Under favorable conditions the simulator is able to approximate the actual Slurm scheduling realization. The simulation fidelity is sufficient to use the simulator for its main function, that is, to test Slurm parameter configurations without having to experiment on full production systems.

References

[1]

Ahmed Eleliemy and Florina M Ciorba. 2021. A Resourceful Coordination Approach for Multilevel Scheduling. arXiv preprint arXiv:2103.05809(2021).

Google Scholar

[2]

Ana Jokanovic, Marco D’Amico, and Julita Corbalan. 2018. Evaluating SLURM simulator with real-machine SLURM and vice versa. In 2018 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS). IEEE, 72–82.

Google Scholar

[3]

Jana Jurečková and Jan Kalina. 2012. Nonparametric multivariate rank tests and their unbiasedness. Bernoulli 18, 1 (2012), 229–251.

Crossref

Google Scholar

[4]

Maxime Martinasso, Miguel Gila, Mauro Bianco, Sadaf R. Alam, Colin McMurtrie, and Thomas C. Schulthess. 2018. RM-Replay: A High-Fidelity Tuning, Optimization and Exploration Tool for Resource Management. In SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. 320–332. https://doi.org/10.1109/SC.2018.00028

Digital Library

Google Scholar

[5]

Nikolay A. Simakov, Robert L. DeLeon, Martins D. Innus, Matthew D. Jones, Joseph P. White, Steven M. Gallo, Abani K. Patra, and Thomas R. Furlani. 2018. Slurm Simulator: Improving Slurm Scheduler Performance on Large HPC Systems by Utilization of Multiple Controllers and Node Sharing. In Proceedings of the Practice and Experience on Advanced Research Computing (Pittsburgh, PA, USA) (PEARC ’18). Association for Computing Machinery, New York, NY, USA, Article 25, 8 pages. https://doi.org/10.1145/3219104.3219111

Digital Library

Google Scholar

[6]

Nikolay A Simakov, Martins D Innus, Matthew D Jones, Robert L DeLeon, Joseph P White, Steven M Gallo, Abani K Patra, and Thomas R Furlani. 2017. A slurm simulator: Implementation and parametric analysis. In International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems. Springer International Publishing, Cham, 197–217. https://doi.org/10.1007/978-3-319-72971-8_10

Crossref

Google Scholar

[7]

Mohammed Tanash, Daniel Andresen, and William Hsu. 2021. AMPRO-HPCC: A Machine-Learning Tool for Predicting Resources on Slurm HPC Clusters. In The Fifteenth International Conference on Advanced Engineering Computing and Applications in Sciences ADVCOMP. 20–27.

Google Scholar

[8]

Aira Villapando and Jessi Christa Rubio. 2021. Simulation vs Actual Walltime Correction in a Real Production Resource-Constrained HPC. In Practice and Experience in Advanced Research Computing. 1–7.

Google Scholar

[9]

Hao Wang, Yi-Qin Dai, Jie Yu, and Yong Dong. 2021. Predicting running time of aerodynamic jobs in HPC system by combining supervised and unsupervised learning method. Advances in Aerodynamics 3, 1 (2021), 1–18.

Crossref

Google Scholar

Cited By

View all

Lyakhovets DBaranov ATelegin P(2024)Scale Ratio Tuning of Group Based Job Scheduling in HPC SystemsLobachevskii Journal of Mathematics10.1134/S199508022311024044:11(5012-5026)Online publication date: 14-Mar-2024
https://doi.org/10.1134/S1995080223110240
Ohmura TTakahashi KEgawa RTakizawa H(2024)A QA-Assisted Job Scheduler for Minimizing the Impact of Urgent Computing on HPC System Operation2024 Twelfth International Symposium on Computing and Networking Workshops (CANDARW)10.1109/CANDARW64572.2024.00039(197-203)Online publication date: 26-Nov-2024
https://doi.org/10.1109/CANDARW64572.2024.00039
Nikitenko DVoevodin VPaokin A(2024)How Can HPC System Holder Help Users to Reduce Time to ResultParallel Computational Technologies10.1007/978-3-031-73372-7_7(92-104)Online publication date: 30-Dec-2024
https://doi.org/10.1007/978-3-031-73372-7_7

Recommendations

Slurm Simulator: Improving Slurm Scheduler Performance on Large HPC systems by Utilization of Multiple Controllers and Node Sharing
PEARC '18: Proceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity

A Slurm simulator was used to study the potential benefits of using multiple Slurm controllers and node-sharing on the TACC Stampede 2 system. Splitting a large cluster into smaller sub-clusters with separate Slurm controllers can offer better scheduling ...
Ensemble Prediction of Job Resources to Improve System Performance for Slurm-Based HPC Systems
PEARC '21: Practice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions

In this paper, we present a novel methodology for predicting job resources (memory and time) for submitted jobs on HPC systems. Our methodology based on historical jobs data (saccount data) provided from the Slurm workload manager using supervised ...
Enabling Workflow-Aware Scheduling on HPC Systems
HPDC '17: Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing

Scientific workflows are increasingly common in the workloads of current High Performance Computing (HPC) systems. However, HPC schedulers do not incorporate workflow-specific mechanisms beyond the capacity to declare dependencies between their jobs. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

PEARC '22: Practice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You

July 2022

455 pages

ISBN:9781450391610

DOI:10.1145/3491418

Editors:
Julie Wernert
Indiana University
,
Alan Chalker
Ohio Supercomputer Center
,
Shava Smallen
San Diego Supercomputer Center, University of California, San Diego
,
Tabitha Samuel
University of Tennessee, Knoxville
,
JP Navarro
Argonne National Laboratory

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

NSF (National Science Foundation)

Conference

PEARC '22

Sponsor:

PEARC '22: Practice and Experience in Advanced Research Computing

July 10 - 14, 2022

MA, Boston, USA

Acceptance Rates

Overall Acceptance Rate 133 of 202 submissions, 66%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
666
Total Downloads

Downloads (Last 12 months)389
Downloads (Last 6 weeks)40

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Lyakhovets DBaranov ATelegin P(2024)Scale Ratio Tuning of Group Based Job Scheduling in HPC SystemsLobachevskii Journal of Mathematics10.1134/S199508022311024044:11(5012-5026)Online publication date: 14-Mar-2024
https://doi.org/10.1134/S1995080223110240
Ohmura TTakahashi KEgawa RTakizawa H(2024)A QA-Assisted Job Scheduler for Minimizing the Impact of Urgent Computing on HPC System Operation2024 Twelfth International Symposium on Computing and Networking Workshops (CANDARW)10.1109/CANDARW64572.2024.00039(197-203)Online publication date: 26-Nov-2024
https://doi.org/10.1109/CANDARW64572.2024.00039
Nikitenko DVoevodin VPaokin A(2024)How Can HPC System Holder Help Users to Reduce Time to ResultParallel Computational Technologies10.1007/978-3-031-73372-7_7(92-104)Online publication date: 30-Dec-2024
https://doi.org/10.1007/978-3-031-73372-7_7

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

Slurm Simulator: Improving Slurm Scheduler Performance on Large HPC systems by Utilization of Multiple Controllers and Node Sharing

Ensemble Prediction of Job Resources to Improve System Performance for Slurm-Based HPC Systems

Enabling Workflow-Aware Scheduling on HPC Systems