More Web Proxy on the site http://driver.im/

research-article

Flexible Partitioning of Scientific Workflows Using the JX Workflow Language

Authors:

Nathaniel Kremer-Herman,

Douglas ThainAuthors Info & Claims

PEARC '19: Practice and Experience in Advanced Research Computing 2019: Rise of the Machines (learning)

Article No.: 103, Pages 1 - 8

https://doi.org/10.1145/3332186.3338100

Published: 28 July 2019 Publication History

Abstract

Scientific workflows are typically expressed as a graph of logical tasks, each one representing a single program along with its input and output files. A conventional workflow manager transforms each logical task into a discrete batch job and submits it to an underlying execution system. However, converting every logical task into one batch job is not necessarily the most efficient partitioning of a workflow. By grouping multiple logical tasks into a single batch job, we may decrease data transfer, increase system utilization, and reduce the execution time of a workflow. This paper presents JX (JSON eXtended), a declarative language that can express complex workloads as an assembly of sub-graphs that can be partitioned in flexible ways. We present a case study of using JX to represent complex workflows for the Lifemapper biodiversity project. We evaluate partitioning approaches across several computing environments, including HTCondor at the University of Notre Dame, TACC Stampede2, and SDSC Comet, and show that a coarse partitioning results in faster turnaround times, reduced data transfer, and lower master utilization across all three systems.

References

[1]

Michael Albrecht, Patrick Donnelly, Peter Bui, and Douglas Thain. 2012. Make-flow: A Portable Abstraction for Data Intensive Computing on Clusters, Clouds, and Grids. In Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies (SWEET '12). ACM, New York, NY, USA, Article 1, 13 pages.

Digital Library

[2]

Peter Amstutz, Michael R. Crusoe, Nebojša Tijanić, Brad Chapman, John Chilton, Michael Heuer, Andrey Kartashov, Dan Leehr, Hervé Ménager, Maya Nedeljkovich, Matt Scales, Stian Soiland-Reyes, and Luka Stojanovic. 2016. Common Workflow Language, v1.0.

[3]

V. Chaudhary and J. K. Aggarwal. 1993. A generalized scheme for mapping parallel algorithms. IEEE Transactions on Parallel and Distributed Systems 4, 3 (Mar 1993), 328--346.

Digital Library

[4]

Mauro Enrique de Souza Muñoz, Renato De Giovanni, Marinez Ferreira de Siqueira, Tim Sutton, Peter Brewer, Ricardo Scachetti Pereira, Dora Ann Lange Canhos, and Vanderlei Perez Canhos. 2011. openModeller: a generic approach to species' potential distribution modelling. GeoInformatica 15, 1 (01 Jan 2011), 111--135.

Digital Library

[5]

Ewa Deelman, Gurmeet Singh, Mei-Hui Su, James Blythe, Yolanda Gil, Carl Kesselman, Gaurang Mehta, Karan Vahi, Bruce Berriman, John Good, Anastasia Laity, Joseph Jacob, and Daniel Katz. 2005. Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems. Scientific Programming Journal 13, 3 (2005), 219--237.

Digital Library

[6]

Stuart I. Feldman. 1979. Make --- a program for maintaining computer programs. Software: Practice and Experience 9, 4 (1979), 255--265.

[7]

Michael R. Garey and David S.Johnson. 1979. Computers and intractability. W. H. Freeman and Co., San Francisco, Calif., New York. x+338 pages. A guide to the theory of NP-completeness, A Series of Books in the Mathematical Sciences.

Digital Library

[8]

Belinda Giardine, Cathy Riemer, Ross C Hardison, Richard Burhans, Laura Elnitski, Prachi Shah, Yi Zhang, Daniel Blankenberg, Istvan Albert, James Taylor, Webb C Miller, W James Kent, and Anton Nekrutenko. 2005. Galaxy: a platform for interactive large-scale genome analysis. Genome research 15, 10 (2005), 1451--1455.

[9]

Johannes Köster and Sven Rahmann. 2012. Snakemake--a scalable bioinformatics workflow engine. Bioinformatics 28, 19 (08 2012), 2520--2522.

Digital Library

[10]

M.J. Litzkow, M.Livny, and M.W. Mutka. 1988. Condor-a hunter of idle workstations. In {1988} Proceedings. The 8th International Conference on Distributed. IEEE Press, Piscataway, NJ, USA, 104--111.

[11]

Dan Stanzione, Bill Barth, Niall Gaffney, Kelly Gaither, Chris Hempel, Tommy Minyard, S. Mehringer, Eric Wernert, H. Tufo, D. Panda, and P. Teller. 2017. Stampede 2: The Evolution of an XSEDE Super computer. In Proceedings of the Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and Impact (PEARC17). ACM, New York, NY, USA, Article 15, 8 pages.

Digital Library

[12]

J. Towns, T. Cockerill, M. Dahan, I. Foster, K. Gaither, A. Grimshaw, V. Hazlewood, S. Lathrop, D. Lifka, G. D. Peterson, R. Roskies, J. R. Scott, and N. Wilkins-Diehr. 2014. XSEDE: Accelerating Scientific Discovery. Computing in Science Engineering 16, 5 (Sept 2014), 62--74.

[13]

Kate Voss, Jeff Gentry, and Geraldine Van der Auwera. 2017. Full-stack genomics pipelining with GATK4 + WDL + Cromwell. Poster {version 1; not peer reviewed}. Presented at the 18th Annual Bioinformatics Open Source Conference (BOSC 2017).

[14]

Michael Wilde, Mihael Hategan, Justin M. Wozniak, Ben Clifford, Daniel S. Katz, and Ian Foster. 2011. Swift: A language for distributed parallel scripting. Parallel Comput. 37, 9 (2011), 633--652. Emerging Programming Paradigms for Large-Scale Scientific Computing.

Digital Library

[15]

Chen Wu, Andreas Wicenec, and Rodrigo Tobar. 2018. Partitioning SKA Dataflows for Optimal Graph Execution. In Proceedings of the 9th Workshop on Scientific Cloud Computing (ScienceCloud'18). ACM, New York, NY, USA, Article 6, 6 pages.

Digital Library

Cited By

Kremer-Herman NThain D(2020)Log Discovery for Troubleshooting Open Distributed Systems with TLQPractice and Experience in Advanced Research Computing 2020: Catch the Wave10.1145/3311790.3396633(224-231)Online publication date: 26-Jul-2020
https://dl.acm.org/doi/10.1145/3311790.3396633

Index Terms

Flexible Partitioning of Scientific Workflows Using the JX Workflow Language

Recommendations

Scientific Workflow Partitioning in Multisite Cloud
Revised Selected Papers, Part I, of the Euro-Par 2014 International Workshops on Parallel Processing - Volume 8805

Scientific workflows allow scientists to conduct experiments that manipulate data with multiple computational activities using Scientific Workflow Management Systems SWfMSs. As the scale of the data increases, SWfMSs need to support workflow execution ...
Integration of Workflow Partitioning and Resource Provisioning
CCGRID '12: Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012)

The recent increased use of workflow management systems by large scientific collaborations presents the challenge of scheduling large-scale workflows onto distributed resources. This work aims to partition large-scale scientific workflows in conjunction ...
A Case Study into Using Common Real-Time Workflow Monitoring Infrastructure for Scientific Workflows

Scientific workflow systems support various workflow representations, operational modes, and configurations. Regardless of the system used, end users have common needs: to track the status of their workflows in real time, be notified of execution ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

PEARC '19: Practice and Experience in Advanced Research Computing 2019: Rise of the Machines (learning)

July 2019

775 pages

ISBN:9781450372275

DOI:10.1145/3332186

General Chair:
Tom Furlani
Roswell Park Comprehensive Cancer Center

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 July 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

PEARC '19

PEARC '19: Practice and Experience in Advanced Research Computing

July 28 - August 1, 2019

IL, Chicago, USA

Acceptance Rates

Overall Acceptance Rate 133 of 202 submissions, 66%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
53
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kremer-Herman NThain D(2020)Log Discovery for Troubleshooting Open Distributed Systems with TLQPractice and Experience in Advanced Research Computing 2020: Catch the Wave10.1145/3311790.3396633(224-231)Online publication date: 26-Jul-2020
https://dl.acm.org/doi/10.1145/3311790.3396633

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents