poster

Towards a common model for pilot-jobs

Authors:

Shantenu JhaAuthors Info & Claims

HPDC '12: Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing

Pages 123 - 124

https://doi.org/10.1145/2287076.2287094

Published: 18 June 2012 Publication History

Get Access

Abstract

Pilot-Jobs have become one of the most successful abstractions in distributed computing. In spite of extensive uptake, there does not exist a well defined, unifying conceptual model of pilot-jobs which can be used to define, compare and contrast different implementations. This presents a barrier to extensibility and interoperability. This paper is an attempt to, (i) provide a minimal but complete model (P*) of pilot-jobs, (ii) establish the generality of the P* Model by mapping various existing and well known pilot-jobs frameworks such as Condor and DIANE to P*, (iii) demonstrate the interoperable and concurrent usage of distinct pilot-job frameworks on different production distributed cyberinfrastructures via the use of an extensible API for the P* Model (Pilot-API).

References

[1]

. Luckow, L. Lacinski, and S. Jha, "SAGA BigJob: An Extensible and Interoperable Pilot-Job Abstraction for Distributed Applications and Systems," in The 10th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, 2010, pp. 135--144. {Online}. Available: http://doi.ieeecomputersociety.org/10.1109/CCGRID.2010.91

Digital Library

Google Scholar

[2]

J. Frey, T. Tannenbaum, M. Livny, I. Foster, and S. Tuecke, "Condor-G: A Computation Management Agent for Multi-Institutional Grids," Cluster Computing, vol. 5, no. 3, pp. 237--246, July 2002.

Digital Library

Google Scholar

[3]

J. Moscicki, "Diane - distributed analysis environment for grid-enabled simulation and analysis of physics data," in phNuclear Science Symposium Conference Record, 2003 IEEE, vol. 3, 2003, pp. 1617 -- 1620.

Google Scholar

[4]

Pilot API, https://github.com/saga-project/BigJob/blob/master/pilot/api/, 2012.

Google Scholar

Cited By

View all

Groen DArabnejad HJancauskas VEdeling WJansson FRichardson RLakhlili JVeen LBosak BKopta PWright DMonnier NKarlshoefer PSuleimenova DSinclair RVassaux MNikishova ABieniek MLuk OKulczewski MRaffin ECrommelin DHoenen OCoster DPiontek TCoveney P(2021)VECMAtk: a scalable verification, validation and uncertainty quantification toolkit for scientific simulationsPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences10.1098/rsta.2020.0221379:2197Online publication date: 29-Mar-2021
https://doi.org/10.1098/rsta.2020.0221
Kamburugamuve SGovindarajan KWickramasinghe PAbeykoon VFox G(2019)Twister2: Design of a big data toolkitConcurrency and Computation: Practice and Experience10.1002/cpe.518932:3Online publication date: 6-Mar-2019
https://doi.org/10.1002/cpe.5189
Belkin MHaas RArnold GLeong HHuerta ELesny DNeubauer MSanielevici S(2018)Container solutions for HPC SystemsProceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity10.1145/3219104.3219145(1-8)Online publication date: 22-Jul-2018
https://dl.acm.org/doi/10.1145/3219104.3219145
Show More Cited By

Index Terms

Towards a common model for pilot-jobs
1. Software and its engineering
  1. Software creation and management
    1. Designing software

Recommendations

Towards a Powerful European DCI Based on Desktop Grids

Service Grids like the EGEE Grid can not provide the required number of resources for many VOs. Therefore extending the capacity of these VOs with volunteer or institutional desktop Grids would significantly increase the number of accessible computing ...
Interoperability of BOINC and EGEE

Today basically two types of grid systems are in use: service grids and desktop grids. Service grids offer an infrastructure for grid users, thus require notable management to keep the service running. On the other hand, desktop grids aim to utilize ...
Towards workflow ecosystems through semantic and standard representations
WORKS '14: Proceedings of the 9th Workshop on Workflows in Support of Large-Scale Science

Workflows are increasingly used to manage and share scientific computations and methods. Workflow tools can be used to design, validate, execute and visualize scientific workflows and their execution results. Other tools manage workflow libraries or ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

HPDC '12: Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing

June 2012

308 pages

ISBN:9781450308052

DOI:10.1145/2287076

General Chair:
Dick Epema
Delft University of Technology and Eindhoven University of Technology, The Netherlands
,
Program Chairs:
Thilo Kielmann
Vrije Universiteit, The Netherlands
,
Matei Ripeanu
The University of British Columbia, Canada

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster

Conference

HPDC'12

Sponsor:

University of Arizona
SIGARCH

HPDC'12: The 21st International Symposium on High-Performance Parallel and Distributed Computing

June 18 - 22, 2012

Delft, The Netherlands

Acceptance Rates

HPDC '12 Paper Acceptance Rate 23 of 143 submissions, 16%;

Overall Acceptance Rate 166 of 966 submissions, 17%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
171
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Groen DArabnejad HJancauskas VEdeling WJansson FRichardson RLakhlili JVeen LBosak BKopta PWright DMonnier NKarlshoefer PSuleimenova DSinclair RVassaux MNikishova ABieniek MLuk OKulczewski MRaffin ECrommelin DHoenen OCoster DPiontek TCoveney P(2021)VECMAtk: a scalable verification, validation and uncertainty quantification toolkit for scientific simulationsPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences10.1098/rsta.2020.0221379:2197Online publication date: 29-Mar-2021
https://doi.org/10.1098/rsta.2020.0221
Kamburugamuve SGovindarajan KWickramasinghe PAbeykoon VFox G(2019)Twister2: Design of a big data toolkitConcurrency and Computation: Practice and Experience10.1002/cpe.518932:3Online publication date: 6-Mar-2019
https://doi.org/10.1002/cpe.5189
Belkin MHaas RArnold GLeong HHuerta ELesny DNeubauer MSanielevici S(2018)Container solutions for HPC SystemsProceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity10.1145/3219104.3219145(1-8)Online publication date: 22-Jul-2018
https://dl.acm.org/doi/10.1145/3219104.3219145
Huerta EHaas RFajardo EKatz DAnderson SCouvares PWillis JBouvet TEnos JKramer WLeong HWheeler D(2017)BOSS-LDG: A Novel Computational Framework That Brings Together Blue Waters, Open Science Grid, Shifter and the LIGO Data Grid to Accelerate Gravitational Wave Discovery2017 IEEE 13th International Conference on e-Science (e-Science)10.1109/eScience.2017.47(335-344)Online publication date: Oct-2017
https://doi.org/10.1109/eScience.2017.47
Jakimovski BIlijoski BVelinov GSahpaski D(2014)Framework for Genetic Algorithms Using Pilot Jobs in Adaptive Grid WorkflowsLarge-Scale Scientific Computing10.1007/978-3-662-43880-0_59(515-522)Online publication date: 26-Jun-2014
https://doi.org/10.1007/978-3-662-43880-0_59
Santcroos Mvan Schaik BShahand SOlabarriaga SLuckow AJha SEpema D(2013)Exploring dynamic enactment of scientific workflows using pilot-abstractionsProceedings of the 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2013.17(311-318)Online publication date: 13-May-2013
https://dl.acm.org/doi/10.1109/CCGrid.2013.17
Mantha PLuckow AJha SFedak GFox GDelamare S(2012)Pilot-MapReduceProceedings of third international workshop on MapReduce and its Applications Date10.1145/2287016.2287020(17-24)Online publication date: 18-Jun-2012
https://dl.acm.org/doi/10.1145/2287016.2287020
Katz DJha SSantcroos MDelgado Olabarriaga S(2012)Pilot abstractions for compute, data, and networkProceedings of the 2012 IEEE 8th International Conference on E-Science (e-Science)10.1109/eScience.2012.6404459(1-2)Online publication date: 8-Oct-2012
https://dl.acm.org/doi/10.1109/eScience.2012.6404459
Mantha PSantcroos MLuckow AJha SWeidner OMerzky A(2012)P∗Proceedings of the 2012 IEEE 8th International Conference on E-Science (e-Science)10.1109/eScience.2012.6404423(1-10)Online publication date: 8-Oct-2012
https://dl.acm.org/doi/10.1109/eScience.2012.6404423
da Silva RGlatard T(2012)A science-gateway workload archive to study pilot jobs, user activity, bag of tasks, task sub-steps, and workflow executionsProceedings of the 18th international conference on Parallel processing workshops10.1007/978-3-642-36949-0_10(79-88)Online publication date: 27-Aug-2012
https://dl.acm.org/doi/10.1007/978-3-642-36949-0_10

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Towards a Powerful European DCI Based on Desktop Grids

Interoperability of BOINC and EGEE

Towards workflow ecosystems through semantic and standard representations