More Web Proxy on the site http://driver.im/

research-article

Reproducible and Portable Workflows for Scientific Computing and HPC in the Cloud

Authors:

Peter Vaillancourt,

Bennett Wineholt,

Brandon Barker,

Plato Deliyannis,

Rich WolskiAuthors Info & Claims

PEARC '20: Practice and Experience in Advanced Research Computing 2020: Catch the Wave

Pages 311 - 320

https://doi.org/10.1145/3311790.3396659

Published: 26 July 2020 Publication History

Abstract

The increasing availability of cloud computing services for science has changed the way scientific code can be developed, deployed, and run. Many modern scientific workflows are capable of running on cloud computing resources. Consequently, there is an increasing interest in the scientific computing community in methods, tools, and implementations that enable moving an application to the cloud and simplifying the process, and decreasing the time to meaningful scientific results. In this paper, we have applied the concepts of containerization for portability and multi-cloud automated deployment with industry-standard tools to three scientific workflows. We show how our implementations provide reduced complexity to portability of both the applications themselves, and their deployment across private and public clouds. Each application has been packaged in a Docker container with its dependencies and necessary environment setup for production runs. Terraform and Ansible have been used to automate the provisioning of compute resources and the deployment of each scientific application in a Multi-VM cluster. Each application has been deployed on the AWS and Aristotle Cloud Federation platforms. Variation in data management constraints, Multi-VM MPI communication, and embarrassingly parallel instance deployments were all explored and reported on. We thus present a sample of scientific workflows that can be simplified using the tools and our proposed implementation to deploy and run in a variety of cloud environments.

Supplemental Material

MP4 File

Presentation video

Download
283.89 MB

References

[1]

Devansh Agarwal, Kshitij Aggarwal, Sarah Burke-Spolaor, Duncan R. Lorimer, and Nathaniel Garver-Daniels. 2019. Towards Deeper Neural Networks for Fast Radio Burst Detection. arxiv:1902.06343 [astro-ph.IM]

[2]

Amazon Web Services Inc.2020. Amazon Web Services (AWS). Amazon Web Services Inc. Retrieved October 23, 2019 from https://aws.amazon.com/

[3]

Amazon Web Services Inc.2020. Amazon Web Services (AWS) storage services. Amazon Web Services Inc. Retrieved October 23, 2019 from https://aws.amazon.com/products/storage/

[4]

Andrew Jameson and Ben Barsdell. 2019. HEIMDALL: Transient Detection Pipeline. SourceForge. Retrieved February 13, 2020 from https://sourceforge.net/projects/heimdall-astro/

[5]

Aristotle Cloud Federation. 2020. Aristotle Cloud Federation Science Use Cases. Aristotle Cloud Federation. Retrieved February 16, 2020 from https://federatedcloud.org/science/index.php

[6]

Berkeley SETI Research Center. 2018. Breakthrough Listen: 4-8 GHz Detections of FRB 121102. Berkeley SETI Research Center. Retrieved January 28, 2020 from https://seti.berkeley.edu/frb121102/technical.html

[7]

Berkeley SETI Research Center. 2018. Breakthrough Listen: Machine Learning Enables New Detections of FRB 121102. Berkeley SETI Research Center. Retrieved January 28, 2020 from https://seti.berkeley.edu/frb-machine/technical.html

[8]

Bruno Bzeznik, Oliver Henriot, Valentin Reis, Olivier Richard, and Laure Tavard. 2017. Nix As HPC Package Management System. In Proceedings of the Fourth International Workshop on HPC User Support Tools (Denver, CO, USA) (HUST’17). ACM, New York, NY, USA, Article 4, 6 pages. https://doi.org/10.1145/3152493.3152556

Digital Library

[9]

Canonical Ltd.2020. Linux Containers. Canonical Ltd. Retrieved February 17, 2020 from https://linuxcontainers.org

[10]

S. R. Carpenter, D. Ludwig, and W. A. Brock. 1999. Management oF Eutrophication for Lakes Subject to Potentially Irreversible Change. Ecological Applications 9, 3 (1999), 751–771. https://doi.org/10.1890/1051-0761(1999)009[0751:MOEFLS]2.0.CO;2

[11]

Ryan Chamberlain and Jennifer Schommer. 2014. Using Docker to Support Reproducible Research. org/10.6084/m9. figshare 1101910 (2014), 44.

[12]

Liam Connor and Joeri van Leeuwen. 2018. Applying Deep Learning to Fast Radio Burst Classification. The Astronomical Journal 156, 6 (Nov 2018), 256. https://doi.org/10.3847/1538-3881/aae649

[13]

James M. Cordes and Shami Chatterjee. 2019. Fast Radio Bursts: An Extragalactic Enigma. Annual Review of Astronomy and Astrophysics 57, 1 (Aug 2019), 417–465. https://doi.org/10.1146/annurev-astro-091918-104501

[14]

Docker Inc.2020. Docker. Docker Inc. Retrieved October 2019 from https://www.docker.com

[15]

Eelco Dolstra. 2006. The purely functional software deployment model. Ph.D. Dissertation. Utrecht University.

[16]

NixOS Foundation. 2020. Nix Package Manager. NixOS Foundation. Retrieved October 22, 2019 from https://nixos.org/nix/

[17]

NixOS Foundation. 2020. NixOS Manual. NixOS Foundation. Retrieved February 06, 2020 from https://nixos.org/nixos/manual/ch-containers

[18]

freedesktop.org. 2020. Systemd-nspawn. X.org Foundation. Retrieved February 06, 2020 from https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html

[19]

V. Gajjar, A. P. V. Siemion, D. C. Price, C. J. Law, D. Michilli, J. W. T. Hessels, S. Chatterjee, A. M. Archibald, G. C. Bower, C. Brinkman, S. Burke-Spolaor, J. M. Cordes, S. Croft, J. Emilio Enriquez, G. Foster, N. Gizani, G. Hellbourg, H. Isaacson, V. M. Kaspi, T. J. W. Lazio, M. Lebofsky, R. S. Lynch, D. MacMahon, M. A. McLaughlin, S. M. Ransom, P. Scholz, A. Seymour, L. G. Spitler, S. P. Tendulkar, D. Werthimer, and Y. G. Zhang. 2018. Highest Frequency Detection of FRB 121102 at 4-8 GHz Using the Breakthrough Listen Digital Backend at the Green Bank Telescope. The Astrophysical Journal 863, 1, Article 2 (Aug 2018), 13 pages. https://doi.org/10.3847/1538-4357/aad005arxiv:1804.04101 [astro-ph.HE]

[20]

HashiCorp. 2020. HashiCorp Configuration Language (HCL). HashiCorp Inc. Retrieved February 17, 2020 from https://github.com/hashicorp/hcl

[21]

HashiCorp Inc.2020. Terraform. HashiCorp, Inc. Retrieved October 21, 2019 from https://www.terraform.io/

[22]

Red Hat Inc.2020. Ansible. Red Hat, Inc. Retrieved October 21, 2019 from https://www.ansible.com/

[23]

Prassanna Jayachandran, Anjali Pawar, and Neelanarayanan Venkataraman. 2017. A review of existing cloud automation tools. Asian Journal of Pharmaceutical and Clinical Research 10 (07 2017), 471. https://doi.org/10.22159/ajpcr.2017.v10s1.20519

[24]

Julianne Quinn. 2019. Lake_Problem_DPS. Cornell University. Retrieved January 28, 2020 from https://github.com/julianneq/Lake_Problem_DPS

[25]

Julianne Quinn and Peter Vaillancourt. 2019. Lake_Problem_DPS. Aristotle Cloud Federation. Retrieved January 28, 2020 from https://github.com/federatedcloud/Lake_Problem_DPS

[26]

Joseph R. Kasprzyk, Shanthi Nataraj, Patrick M. Reed, and Robert J. Lempert. 2013. Many objective robust decision making for complex environmental systems undergoing change. Environmental Modelling & Software 42 (2013), 55 – 71. https://doi.org/10.1016/j.envsoft.2012.12.007

Digital Library

[27]

Richard Knepper, Susan Mehringer, Adam Brazier, Brandon Barker, and Resa Reynolds. 2019. Red Cloud and Aristotle: Campus Clouds and Federations. In Proceedings of the Humans in the Loop: Enabling and Facilitating Research on Cloud Computing (Chicago, IL, USA) (HARC ’19). Association for Computing Machinery, New York, NY, USA, Article 4, 6 pages. https://doi.org/10.1145/3355738.3355755

Digital Library

[28]

Gregory M. Kurtzer, Vanessa Sochat, and Michael W. Bauer. 2017. Singularity: Scientific containers for mobility of compute. PLOS ONE 12, 5 (05 2017), 1–20. https://doi.org/10.1371/journal.pone.0177459

[29]

Matthew Lebofsky, Steve Croft, Andrew P. V. Siemion, Danny C. Price, J. Emilio Enriquez, Howard Isaacson, David H. E. MacMahon, David Anderson, Bryan Brzycki, Jeff Cobb, Daniel Czech, David DeBoer, Julia DeMarines, Jamie Drew, Griffin Foster, Vishal Gajjar, Nectaria Gizani, Greg Hellbourg, Eric J. Korpela, Brian Lacki, Sofia Sheikh, Dan Werthimer, Pete Worden, Alex Yu, and Yunfan Gerry Zhang. 2019. The Breakthrough Listen Search for Intelligent Life: Public Data, Formats, Reduction, and Archiving. Publications of the Astronomical Society of the Pacific 131, 1006 (Dec 2019), 124505. https://doi.org/10.1088/1538-3873/ab3e82arxiv:1906.07391 [astro-ph.IM]

[30]

Google LLC. 2020. Google Cloud. Google LLC. Retrieved October 23, 2019 from https://cloud.google.com/

[31]

D. R. Lorimer, M. Bailes, M. A. McLaughlin, D. J. Narkevic, and F. Crawford. 2007. A Bright Millisecond Radio Burst of Extragalactic Origin. Science 318, 5851 (Nov 2007), 777–780. https://doi.org/10.1126/science.1147532

[32]

Microsoft. 2020. Microsoft Azure Cloud Computing Service. Microsoft. Retrieved May 18, 2020 from https://azure.microsoft.com/en-us/

[33]

NANOGrav. 2020. North American Nanohertz Observatory for Gravitational Waves (NANOGrav). NANOGrav. Retrieved January 28, 2020 from http://nanograv.org/

[34]

Nate Garver. 2019. nanopulsar Docker Container. NANOGrav. Retrieved January 28, 2020 from https://github.com/nanograv/nanopulsar

[35]

National Center for Atmospheric Research (NCAR). 2019. WRF_DOCKER. NCAR. Retrieved February 16, 2020 from https://github.com/NCAR/WRF_DOCKER/

[36]

Peter Vaillancourt and Adam Brazier. 2019. modulation_index Docker container. Aristotle Cloud Federation. Retrieved January 28, 2020 from https://github.com/federatedcloud/modulation_index/tree/master/docker

[37]

Peter Vaillancourt and Nate Garver. 2019. nanopulsar Docker container updated. Aristotle Cloud Federation. Retrieved January 28, 2020 from https://github.com/federatedcloud/nanopulsar

[38]

Peter Vaillancourt, Plato Deliyannis, and Akshay Suresh. 2020. FRB_Pipeline. Aristotle Cloud Federation. Retrieved January 28, 2020 from https://github.com/federatedcloud/FRB_pipeline

[39]

Poul-henning Kamp and Robert N. M. Watson. 2000. Jails: Confining the omnipotent root. The FreeBSD Project. Retrieved May 28, 2020 from http://www.sane.nl/events/sane2000/papers/kamp.pdf

[40]

Daniel Price and Andrew Tucker. 2004. Solaris Zones: Operating System Support for Consolidating Commercial Workloads. In Proceedings of the 18th USENIX Conference on System Administration (Atlanta, GA) (LISA ’04). USENIX Association, USA, 241–254.

Digital Library

[41]

Julianne D. Quinn, Patrick M. Reed, and Klaus Keller. 2017. Direct policy search for robust multi-objective management of deeply uncertain socio-ecological tipping points. Environmental Modelling & Software 92 (2017), 125 – 141. https://doi.org/10.1016/j.envsoft.2017.02.017

Digital Library

[42]

Scott Mitchell Ransom. 2001. New search techniques for binary pulsars. Ph.D. Dissertation. Harvard University.

[43]

Michael Rosenstein and Andrew Barto. 2001. Robot Weightlifting By Direct Policy Search.

[44]

L. G. Spitler, J. M. Cordes, J. W. T. Hessels, D. R. Lorimer, M. A. McLaughlin, S. Chatterjee, F. Crawford, J. S. Deneva, V. M. Kaspi, R. S. Wharton, and et al.2014. Fast Radio Burst Discovered in the Arecibo Pulsar ALFA Survey. The Astrophysical Journal 790, 2 (Jul 2014), 101. https://doi.org/10.1088/0004-637x/790/2/101

[45]

Sylabs. 2020. Running Services — Singularity Container 3.0 Documentation. https://sylabs.io/guides/3.0/user-guide/running_services.html?highlight=instance[Online; accessed 13-Jan-2020].

[46]

WRF Release Committee. 2020. WRF. National Center for Atmospheric Research (NCAR). Retrieved February 16, 2020 from https://www.mmm.ucar.edu/weather-research-and-forecasting-model

[47]

Yunfan Gerry Zhang, Vishal Gajjar, Griffin Foster, Andrew Siemion, James Cordes, Casey Law, and Yu Wang. 2018. Fast Radio Burst 121102 Pulse Detection and Periodicity: A Machine Learning Approach. The Astrophysical Journal 866, 2, Article 149 (Oct 2018), 32 pages. https://doi.org/10.3847/1538-4357/aadf31arxiv:1809.03043 [astro-ph.HE]

Cited By

Lofton MHoward DThomas RCarey C(2023)Progress and opportunities in advancing near‐term forecasting of freshwater qualityGlobal Change Biology10.1111/gcb.1659029:7(1691-1714)Online publication date: 23-Jan-2023
https://doi.org/10.1111/gcb.16590
Nicolae BIslam TRoss RVan Dam HAssogba KShpilker PTitov MTurilli MWang TKilic OJha SPouchard L(2023)Building the I (Interoperability) of FAIR for Performance Reproducibility of Large-Scale Composable Workflows in RECUP2023 IEEE 19th International Conference on e-Science (e-Science)10.1109/e-Science58273.2023.10254808(1-7)Online publication date: 9-Oct-2023
https://doi.org/10.1109/e-Science58273.2023.10254808
Lazzaro AHaus UCharousset SMujkanovic N(2023)A User-Oriented Portable, Reproducible, and Scalable Software EcosystemHigh Performance Computing. ISC High Performance 2024 International Workshops10.1007/978-3-031-73716-9_28(402-414)Online publication date: 12-May-2023
https://dl.acm.org/doi/10.1007/978-3-031-73716-9_28
Show More Cited By

Recommendations

Comparing FutureGrid, Amazon EC2, and Open Science Grid for Scientific Workflows

Scientists have many computing infrastructures available to conduct their research, including grids and public or private clouds. This article explores the use of these cyberinfrastructures to execute scientific workflows, an important class of ...
Cloud Computing

The guest editors discuss this special issue on cloud computing, exploring how cloud platforms and abstractions can be effectively used to support real-world science and engineering applications.
Virtual I/O Forwarding for Cloud-based HPC Applications
ISPA '12: Proceedings of the 2012 IEEE 10th International Symposium on Parallel and Distributed Processing with Applications

Over the past years, the cloud, especially Infrastructure as a Service (IaaS) offerings, have seen a staggering rise in popularity due to the convenience and the economic pricing models. Even cloud service providers are starting to offer custom-sized ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

PEARC '20: Practice and Experience in Advanced Research Computing 2020: Catch the Wave

July 2020

556 pages

ISBN:9781450366892

DOI:10.1145/3311790

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science Foundation

Conference

PEARC '20

Sponsor:

PEARC '20: Practice and Experience in Advanced Research Computing

July 26 - 30, 2020

OR, Portland, USA

Acceptance Rates

Overall Acceptance Rate 133 of 202 submissions, 66%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
179
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lofton MHoward DThomas RCarey C(2023)Progress and opportunities in advancing near‐term forecasting of freshwater qualityGlobal Change Biology10.1111/gcb.1659029:7(1691-1714)Online publication date: 23-Jan-2023
https://doi.org/10.1111/gcb.16590
Nicolae BIslam TRoss RVan Dam HAssogba KShpilker PTitov MTurilli MWang TKilic OJha SPouchard L(2023)Building the I (Interoperability) of FAIR for Performance Reproducibility of Large-Scale Composable Workflows in RECUP2023 IEEE 19th International Conference on e-Science (e-Science)10.1109/e-Science58273.2023.10254808(1-7)Online publication date: 9-Oct-2023
https://doi.org/10.1109/e-Science58273.2023.10254808
Lazzaro AHaus UCharousset SMujkanovic N(2023)A User-Oriented Portable, Reproducible, and Scalable Software EcosystemHigh Performance Computing. ISC High Performance 2024 International Workshops10.1007/978-3-031-73716-9_28(402-414)Online publication date: 12-May-2023
https://dl.acm.org/doi/10.1007/978-3-031-73716-9_28
Munhoz VCastro M(2023) Enabling the execution of HPC applications on public clouds with HPC@Cloud toolkit Concurrency and Computation: Practice and Experience10.1002/cpe.797636:8Online publication date: 4-Dec-2023
https://doi.org/10.1002/cpe.7976
Keller Tesser RBorin E(2022)Containers in HPC: a surveyThe Journal of Supercomputing10.1007/s11227-022-04848-y79:5(5759-5827)Online publication date: 27-Oct-2022
https://dl.acm.org/doi/10.1007/s11227-022-04848-y

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents