More Web Proxy on the site http://driver.im/

research-article

Supporting dynamic parameter sweep in adaptive and user-steered workflow

Authors:

Eduardo Ogasawara,

Daniel de Oliveira,

Alvaro L.G.A. Coutinho,

Marta MattosoAuthors Info & Claims

WORKS '11: Proceedings of the 6th workshop on Workflows in support of large-scale science

Pages 31 - 36

https://doi.org/10.1145/2110497.2110502

Published: 14 November 2011 Publication History

Abstract

Large-scale experiments in computational science are complex to manage. Due to its exploratory nature, several iterations evaluate a large space of parameter combinations. Scientists analyze partial results and dynamically interfere on the next steps of the simulation. Scientific workflow management systems can execute those experiments by providing process management, distributed execution and provenance data. However, supporting scientists in complex exploratory processes involving dynamic workflows is still a challenge. Features, such as user steering on workflows to track, evaluate and adapt the execution need to be designed to support iterative methods. We provide an approach to support dynamic parameter sweep, in which scientists can use the results obtained in a slice of the parameter space to improve the remainder of the execution. We propose new control structures to enable adaptive and user-steered workflows supporting iterative methods using dynamic mechanisms. We evaluate our approach using a proof of concept (Lanczos algorithm) workflow and the results show up to 78% of execution time saved.

References

[1]

Y.L. Simmhan, B. Plale, e D. Gannon, 2005, A survey of data provenance in e-science, ACM SIGMOD Record, v. 34, n. 3, p. 31--36.

Digital Library

[2]

I. Altintas, C. Berkley, E. Jaeger, M. Jones, B. Ludascher, e S. Mock, 2004, Kepler: an extensible system for design and execution of scientific workflows, In: Scientific and Statistical Database Management, p. 423--424, Greece.

Digital Library

[3]

S.P. Callahan, J. Freire, E. Santos, C.E. Scheidegger, C.T. Silva, e H.T. Vo, 2006, VisTrails: visualization meets data management, In: SIGMOD, p. 745--747, Chicago, Illinois, USA.

Digital Library

[4]

A.M.R. Dávila, P.N. Mendes, G. Wagner, D.A. Tschoeke, R.R.C. Cuadrat, F. Liberman, L. Matos, T. Satake, K.A.C.S. Ocaña, et al., 2008, ProtozoaDB: dynamic visualization and exploration of protozoan genomes, Nucleic Acids Research, v. 36, n. Database issue, p. D547-D552.

[5]

E. Ogasawara, D. Oliveira, F. Chirigati, C.E. Barbosa, R. Elias, V. Braganholo, A. Coutinho, e M. Mattoso, 2009, Exploring many task computing in scientific workflows, In: MTAGS, p. 1--10, Portland, Oregon.

Digital Library

[6]

Y. Gil, E. Deelman, M. Ellisman, T. Fahringer, G. Fox, D. Gannon, C. Goble, M. Livny, L. Moreau, et al., 2007, Examining the Challenges of Scientific Workflows, Computer, v. 40, n. 12, p. 24--32.

Digital Library

[7]

E. Ogasawara, J. Dias, D. Oliveira, F. Porto, P. Valduriez, e M. Mattoso, 2011, An Algebraic Approach for Data-Centric Scientific Workflows, Proceedings of the VLDB Endowment, v. 4, n. 12, p. 1328--1339.

[8]

D. Oliveira, K. Ocana, E. Ogasawara, J. Dias, F. Baião, e M. Mattoso, 2011, A Performance Evaluation of X-ray Crystallography Scientific Workflow using SciCumulus, In: International Conference on Cloud ComputingInternational Conference on Cloud Computing, Washington D.C.

Digital Library

[9]

G. Guerra, F. Rochinha, R. Elias, D. Oliveira, E. Ogasawara, J. Dias, M. Mattoso, e A.L.G.A. Coutinho, 2011, Uncertainty Quantification in Computational Predictive Models for Fluid Dynamics Using Workflow Management Engine, International Journal for Uncertainty Quantification, accepted

[10]

M. Heath, 2002, Scientific computing: an introductory survey. 2 ed. Boston, McGraw-Hill.

Digital Library

[11]

Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. von Laszewski, V. Nefedova, I. Raicu, T. Stef-Praun, e M. Wilde, 2007, Swift: Fast, Reliable, Loosely Coupled Parallel Computation, In: Proc. of the 3rd IEEE World Congress on Services, p. 206, 199, Salt Lake City, USA.

[12]

B. de S. Leite Pires de Lima, B. Pinheiro Jacob, e N. Francisco Favilla Ebecken, 2005, A hybrid fuzzy/genetic algorithm for the design of offshore oil production risers, International Journal for Numerical Methods in Engineering, v. 64, n. 11, p. 1459--1482.

[13]

X. Ma e N. Zabaras, 2009, An adaptive hierarchical sparse grid collocation algorithm for the solution of stochastic differential equations, J. Comput. Phys., v. 228, p. 3084--3113.

Digital Library

[14]

A.L.G.A. Coutinho, L. Landau, L.C. Wrobel, e N.F.F. Ebecken, 1989, Modal solution of transient heat conduction utilizing Lanczos algorithm, International Journal for Numerical Methods in Engineering, v. 28, n. 1, p. 13--25.

[15]

S. Wang, E. de Sturler, e G.H. Paulino, 2007, Large-scale topology optimization using preconditioned Krylov subspace methods with recycling, International Journal for Numerical Methods in Engineering, v. 69, n. 12, p. 2441--2468.

[16]

D. Amsallem e C. Farhat, 2008, Interpolation Method for Adapting Reduced-Order Models and Application to Aeroelasticity, AIAA Journal, v. 46, p. 1803--1813.

[17]

M.L. Parks, E. de Sturler, G. Mackey, D.D. Johnson, e S. Maiti, 2006, Recycling Krylov Subspaces for Sequences of Linear Systems, SIAM Journal on Scientific Computing, v. 28, n. 5, p. 1651.

Digital Library

[18]

M.E. Kilmer e E. de Sturler, 2006, Recycling Subspace Information for Diffuse Optical Tomography, SIAM Journal on Scientific Computing, v. 27, n. 6, p. 2140.

Digital Library

[19]

S.W. Sorde, S.K. Aggarwal, J. Song, M. Koh, e S. See, 2007, Modeling and Verifying Non-DAG Workflows for Computational Grids, In: Services, IEEE Congress on, p. 237--243, Los Alamitos, CA, USA.

[20]

M. Mattoso, A. Coutinho, R. Elias, D. Oliveira, e E. Ogasawara, 2010, Exploring Parallel Parameter Sweep in Scientific Workflows, In: WCCM -- World Congress on Computational Mechanics, Australia.

[21]

J. Dean e S. Ghemawat, 2008, MapReduce: simplified data processing on large clusters, Commun. ACM, v. 51, n. 1, p. 107--113.

Digital Library

[22]

D. Abramson, B. Bethwaite, C. Enticott, S. Garic, e T. Peachey, 2011, Parameter Exploration in Science and Engineering Using Many-Task Computing, IEEE Trans. Parallel Distrib. Syst., v. 22, n. 6 (jun.), p. 960--973.

Digital Library

[23]

V.S. Kumar, P. Sadayappan, G. Mehta, K. Vahi, E. Deelman, V. Ratnakar, J. Kim, Y. Gil, M. Hall, et al., 2009, An integrated framework for performance-based optimization of scientific workflows, In: Proceedings of the 18th ACM international symposium on High performance distributed computing, p. 177--186, New York, NY, USA.

Digital Library

Cited By

Krämer MWürz HAltenhofen C(2021)Executing cyclic scientific workflows in the cloudJournal of Cloud Computing: Advances, Systems and Applications10.1186/s13677-021-00229-710:1Online publication date: 6-Apr-2021
https://dl.acm.org/doi/10.1186/s13677-021-00229-7
Souza RSilva VCoutinho AValduriez PMattoso M(2020)Data reduction in scientific workflows using provenance monitoring and user steeringFuture Generation Computer Systems10.1016/j.future.2017.11.028110(481-501)Online publication date: Sep-2020
https://doi.org/10.1016/j.future.2017.11.028
Kail EKacsuk PKozlovszky M(2016)Specification of user and provenance-based adaptive control points at workflow composition level2016 IEEE 14th International Symposium on Intelligent Systems and Informatics (SISY)10.1109/SISY.2016.7601477(93-98)Online publication date: Aug-2016
https://doi.org/10.1109/SISY.2016.7601477
Show More Cited By

Index Terms

Supporting dynamic parameter sweep in adaptive and user-steered workflow

Recommendations

Evaluating parameter sweep workflows in high performance computing
SWEET '12: Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies

Scientific experiments based on computer simulations can be defined, executed and monitored using Scientific Workflow Management Systems (SWfMS). Several SWfMS are available, each with a different goal and a different engine. Due to the exploratory ...
Scheduling parameter sweep workflow in the Grid based on resource competition

Workflow technology has been adopted in scientific domains to orchestrate and automate scientific processes in order to facilitate experimentation. Such scientific workflows often involve large data sets and intensive computation that necessitate the ...
Scheduling Multiple Parameter Sweep Workflow Instances on the Grid
E-SCIENCE '09: Proceedings of the 2009 Fifth IEEE International Conference on e-Science

Due to its ability to provide high-performance computing environment, the grid has become an important infrastructure to support eScience. To utilise the grid for parameter sweep experiments, workflow technology combined with tools such as Nimrod/K are ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WORKS '11: Proceedings of the 6th workshop on Workflows in support of large-scale science

November 2011

154 pages

ISBN:9781450311007

DOI:10.1145/2110497

General Chairs:
Ian Taylor
Cardiff University, UK
,
Johan Montagnat
CNRS, France

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 November 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SC '11

Sponsor:

SIGARCH

SC '11: International Conference for High Performance Computing, Networking, Storage and Analysis

November 14, 2011

Washington, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 30 of 54 submissions, 56%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
287
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Krämer MWürz HAltenhofen C(2021)Executing cyclic scientific workflows in the cloudJournal of Cloud Computing: Advances, Systems and Applications10.1186/s13677-021-00229-710:1Online publication date: 6-Apr-2021
https://dl.acm.org/doi/10.1186/s13677-021-00229-7
Souza RSilva VCoutinho AValduriez PMattoso M(2020)Data reduction in scientific workflows using provenance monitoring and user steeringFuture Generation Computer Systems10.1016/j.future.2017.11.028110(481-501)Online publication date: Sep-2020
https://doi.org/10.1016/j.future.2017.11.028
Kail EKacsuk PKozlovszky M(2016)Specification of user and provenance-based adaptive control points at workflow composition level2016 IEEE 14th International Symposium on Intelligent Systems and Informatics (SISY)10.1109/SISY.2016.7601477(93-98)Online publication date: Aug-2016
https://doi.org/10.1109/SISY.2016.7601477
Kail EKacsuk PKozlovszky M(2015)A novel approach to user-steering in scientific workflows2015 IEEE 10th Jubilee International Symposium on Applied Computational Intelligence and Informatics10.1109/SACI.2015.7208205(233-236)Online publication date: May-2015
https://doi.org/10.1109/SACI.2015.7208205
Mattoso MDias JOcaña KOgasawara ECosta FHorta FSilva Vde Oliveira D(2015)Dynamic steering of HPC scientific workflowsFuture Generation Computer Systems10.1016/j.future.2014.11.01746:C(100-113)Online publication date: 1-May-2015
https://dl.acm.org/doi/10.1016/j.future.2014.11.017
Dias JGuerra GRochinha FCoutinho AValduriez PMattoso M(2015)Data-centric iteration in dynamic workflowsFuture Generation Computer Systems10.1016/j.future.2014.10.02146:C(114-126)Online publication date: 1-May-2015
https://dl.acm.org/doi/10.1016/j.future.2014.10.021
Nguyen HAbramson DKipouros TJanke AGalloway G(2015) WorkWays : interacting with scientific workflows Concurrency and Computation: Practice and Experience10.1002/cpe.352527:16(4377-4397)Online publication date: 21-May-2015
https://doi.org/10.1002/cpe.3525
Pintas Jde Oliveira DOcaña KOgasawara EMattoso M(2014)SciLightning: A Cloud Provenance-Based Event Notification for Parallel WorkflowsService-Oriented Computing – ICSOC 2013 Workshops10.1007/978-3-319-06859-6_31(352-365)Online publication date: 2014
https://doi.org/10.1007/978-3-319-06859-6_31
Mattoso MOcaña KHorta FDias JOgasawara ESilva Vde Oliveira DCosta FAraújo IHidders JMissier PSroka J(2013)User-steering of HPC workflowsProceedings of the 2nd ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies10.1145/2499896.2499900(1-6)Online publication date: 23-Jun-2013
https://dl.acm.org/doi/10.1145/2499896.2499900
Horta FSilva VCosta Fde Oliveira DOcaña KOgasawara EDias JMattoso MGuerrini G(2013)Provenance traces from Chiron parallel workflow engineProceedings of the Joint EDBT/ICDT 2013 Workshops10.1145/2457317.2457379(337-338)Online publication date: 18-Mar-2013
https://dl.acm.org/doi/10.1145/2457317.2457379
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents