More Web Proxy on the site http://driver.im/

research-article

A Declarative Optimization Engine for Resource Provisioning of Scientific Workflows in IaaS Clouds

Authors:

Amelie Chi Zhou,

Chiew Tong LauAuthors Info & Claims

HPDC '15: Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing

Pages 223 - 234

https://doi.org/10.1145/2749246.2749251

Published: 15 June 2015 Publication History

Abstract

Resource provisioning for scientific workflows in Infrastructure-as-a-service (IaaS) clouds is an important and complicated problem for budget and performance optimizations of workflows. Scientists are facing the complexities resulting from severe cloud performance dynamics and various user requirements on performance and cost. To address those complexity issues, we propose a declarative optimization engine named Deco for resource provisioning of scientific workflows in IaaS clouds. Deco allows users to specify their workflow optimization goals and constraints of specific problems with an extended declarative language. We propose a novel probabilistic optimization approach for evaluating the declarative optimization goals and constraints in dynamic clouds. To accelerate the solution finding, Deco leverages the available power of GPUs to find the solution in a fast and timely manner. We evaluate Deco with several common provisioning problems. We integrate Deco into a popular workflow management system (Pegasus) and show that Deco can achieve more effective performance/cost optimizations than the state-of-the-art approaches.

References

[1]

L. Abeni and G. Buttazzo. Qos guarantee using probabilistic deadlines. In ECRTS 1999.

[2]

S. Abrishami, M. Naghibzadeh, and D. H. J. Epema. Deadline-constrained workflow scheduling algorithms for infrastructure as a service clouds. FGCS, 2013.

Digital Library

[3]

P. Alvaro, T. Condie, N. Conway, K. Elmeleegy, J. M. Hellerstein, and R. Sears. Boom Analytics: Exploring Data-centric, Declarative Programming for the Cloud. In EuroSys '10.

Digital Library

[4]

Amazon Case Studies. http://aws.amazon.com/solutions/case-studies/. accessed on July 2014.

[5]

A. Arbelaez and P. Codognet. A GPU Implementation of Parallel Constraint-based Local Search. In PDP '14.

Digital Library

[6]

E.-K. Byun, Y.-S. Kee, J.-S. Kim, and S. Maeng. Cost Optimized Provisioning of Elastic Resources for Application Workflows. FGCS, 2011.

Digital Library

[7]

R. N. Calheiros and R. Buyya. Meeting Deadlines of Scientific Workflows in Public Clouds with Tasks Replication. IEEE TPDS, 2013.

Digital Library

[8]

R. N. Calheiros, R. Ranjan, A. Beloglazov, C. A. De Rose, and R. Buyya. CloudSim: A Toolkit for Modeling and Simulation of Cloud Computing Environments and Evaluation of Resource Provisioning Algorithms. Softw. Pract. Exper., 2011.

Digital Library

[9]

J. Cao, S. Jarvis, S. Saini, and G. R. Nudd. GridFlow: Workflow Management for Grid Computing. In CCGrid '03.

Digital Library

[10]

Z. Chen, S. W. Son, W. Hendrix, A. Agrawal, W. keng Liao, and A. Choudhary. Numarck: Machine learning algorithm for resiliency and checkpointing. In SC '14.

Digital Library

[11]

DAGMan. http://research.cs.wisc.edu/htcondor/dagman/dagman.html. accessed on May 2014.

[12]

L. De Raedt, A. Kimmig, and H. Toivonen. ProbLog: a Probabilistic Prolog and its Application in Link Discovery. In IJCAI '07.

Digital Library

[13]

E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G. B. Berriman, J. Good, et al. Pegasus: A Framework for Mapping Complex Scientific Workflows Onto Distributed Systems. Sci. Program., 2005.

Digital Library

[14]

K. Deng, J. Song, K. Ren, and A. Iosup. Exploring Portfolio Scheduling for Long-term Execution of Scientific Workloads in IaaS Clouds. In SC '13.

Digital Library

[15]

I. Foster, Y. Zhao, I. Raicu, and S. Lu. Cloud Computing and Grid Computing 360-Degree Compared. In GCE '08.

[16]

A. Iosup, S. Ostermann, M. N. Yigitbasi, R. Prodan, T. Fahringer, and D. H. Epema. Performance Analysis of Cloud Computing Services for Many-Tasks Scientific Computing. IEEE TPDS, 2011.

Digital Library

[17]

Iperf. http://iperf.sourceforge.net. accessed on July 2014.

[18]

G. Juve, A. Chervenak, E. Deelman, S. Bharathi, G. Mehta, and K. Vahi. Characterizing and Profiling Scientific Workflows. FGCS, 2013.

Digital Library

[19]

K. Keahey, R. Figueiredo, J. Fortes, T. Freeman, and M. Tsugawa. Science Clouds: Early Experiences in Cloud Computing for Scientific Applications. In CCA '08.

[20]

A. Kimmig, B. Demoen, L. De Raedt, V. S. Costa, and R. Rocha. On the Implementation of the Probabilistic Logic Programming Language Problog. Theory Pract. Log. Program., 2011.

Digital Library

[21]

C. Liu, B. T. Loo, and Y. Mao. Declarative Automated Cloud Resource Orchestration. In SoCC '11.

Digital Library

[22]

J. W. Lloyd. Foundations of Logic Programming. 1984.

Digital Library

[23]

B. Ludäscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, E. A. Lee, J. Tao, and Y. Zhao. Scientific Workflow Management and the Kepler System: Research Articles. Concurr. Comput.: Pract. Exper., 2006.

Digital Library

[24]

M. Malawski, G. Juve, E. Deelman, and J. Nabrzyski. Cost- and Deadline-constrained Provisioning for Scientific Workflow Ensembles in IaaS Clouds. In SC '12.

Digital Library

[25]

M. Mao and M. Humphrey. Auto-scaling to Minimize Cost and Meet Application Deadlines in Cloud Workflows. In SC '11.

Digital Library

[26]

M. Mao and M. Humphrey. Scaling and Scheduling to Maximize Application Performance within Budget Constraints in Cloud Workflows. In IPDPS '13.

Digital Library

[27]

Montage Archive. http://hachi.ipac.caltech.edu:8080/montage/. accessed on July 2014.

[28]

Montage Workflow. http://montage.ipac.caltech.edu/docs/download2.html. accessed on July 2014.

[29]

D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, and D. Zagorodnov. The Eucalyptus Open-Source Cloud-Computing System. In CCGRID '09.

Digital Library

[30]

R. A. O'Keefe. The Craft of Prolog. 1990.

Digital Library

[31]

I. Pietri, G. Juve, E. Deelman, and R. Sakellariou. A Performance Model to Estimate Execution Time of Scientific Workflows on the Cloud. WORKS '14.

Digital Library

[32]

A. Rai, R. Bhagwan, and S. Guha. Generalized Resource Allocation for the Cloud. In SoCC '12.

Digital Library

[33]

J. Schad, J. Dittrich, and J.-A. Quiané-Ruiz. Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance. Proc. VLDB Endow., 2010.

Digital Library

[34]

P. Sevon, L. Eronen, P. Hintsanen, K. Kulovesi, and H. Toivonen. Link Discovery in Graphs Derived from Biological Databases. In DILS'06.

Digital Library

[35]

W. Tang, J. Wilkening, N. Desai, W. Gerlach, A. Wilke, and F. Meyer. A Scalable Data Analysis Platform for Metagenomics. In BigData '13.

[36]

J. Van der Merwe, K. Ramakrishnan, M. Fairchild, A. Flavel, J. Houle, H. A. Lagar-Cavilla, and J. Mulligan. Towards a Ubiquitous Cloud Computing Infrastructure. In LANMAN '10.

[37]

T. Van Luong, N. Melab, and E.-G. Talbi. GPU Computing for Parallel Local Search Metaheuristic Algorithms. IEEE TC, 2013.

Digital Library

[38]

J. Wang and I. Altintas. Early Cloud Experiences with the Kepler Scientific Workflow System. Procedia Computer Science, 2012.

[39]

L. Wang, J. Tao, M. Kunze, D. Rattu, and A. C. Castellanos. The Cumulus Project: Build a Scientific Cloud for a Data Center. In CCA '08.

[40]

Windows Azure Case Studies. http://azure.microsoft.com/en-us/case-studies/. acessed on July 2014.

[41]

Workflow Generator. https://confluence.pegasus.isi.edu/display/pegasus/WorkflowGenerator. accessed on July 2014.

[42]

J. Yu and R. Buyya. A Taxonomy of Scientific Workflow Systems for Grid Computing. SIGMOD Rec., 2005.

Digital Library

[43]

J. Yu, R. Buyya, and C. K. Tham. Cost-based Scheduling of Scientific Workflow Applications on Utility Grids. In e-Science '05.

Digital Library

[44]

M. Zhang, R. Ranjan, S. Nepal, M. Menzel, and A. Haller. A Declarative Recommender System for Cloud Infrastructure Services Selection. In GECON'12.

Digital Library

[45]

A. Zhou and B. He. Simplified resource provisioning for workflows in iaas clouds. In CloudCom'14, pages 650--655, 2014.

Digital Library

[46]

A. C. Zhou and B. He. Transformation-based Monetary Cost Optimizations for Workflows in the Cloud. IEEE TCC, 2013.

[47]

A. C. Zhou and B. He. A Declarative Optimization Engine for Resource Provisioning of Scientific Workflows in IaaS Clouds. Technical Report 2015-TR-Deco, http://pdcc.ntu.edu.sg/xtra/tr/2015-TR-Deco.pdf, 2015.

[48]

A. C. Zhou, B. He, and C. Liu. Monetary Cost Optimizations for Hosting Workflow-as-a-Service in IaaS Clouds. IEEE TCC, 2015.

[49]

T. Zou, R. Le Bras, M. V. Salles, A. Demers, and J. Gehrke. ClouDiA: a Deployment Advisor for Public Clouds. In PVLDB'13, pages 121--132.

Digital Library

Cited By

Zhou AXue WXiao YHe BIbrahim SCheng R(2022)Taming System Dynamics on Resource Optimization for Data Processing Workflows: A Probabilistic ApproachIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2021.309140033:1(231-248)Online publication date: 1-Jan-2022
https://doi.org/10.1109/TPDS.2021.3091400
Hu JLi KLiu CLi K(2021)A Game-Based Price Bidding Algorithm for Multi-Attribute Cloud Resource ProvisionIEEE Transactions on Services Computing10.1109/TSC.2018.286002214:4(1111-1122)Online publication date: 1-Jul-2021
https://doi.org/10.1109/TSC.2018.2860022
Versluis LMatha RTalluri SHegeman TProdan RDeelman EIosup A(2020)The Workflow Trace Archive: Open-Access Data From Public and Private Computing InfrastructuresIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2020.298482131:9(2170-2184)Online publication date: 1-Sep-2020
https://doi.org/10.1109/TPDS.2020.2984821
Show More Cited By

Index Terms

A Declarative Optimization Engine for Resource Provisioning of Scientific Workflows in IaaS Clouds

Recommendations

Simplified Resource Provisioning for Workflows in IaaS Clouds
CLOUDCOM '14: Proceedings of the 2014 IEEE 6th International Conference on Cloud Computing Technology and Science

Resource provisioning is an important and complicated problem for scientific workflows in Infrastructure-as-a-service (IaaS) clouds. Scientists are facing the complexities resulting from the diverse cloud offerings, complex workflow structures and ...
A Declarative Optimization Engine for Resource Provisioning of Scientific Workflows in Geo-Distributed Clouds

Geo-distributed clouds are becoming increasingly popular for cloud providers, and data centers with different regions often offer different prices, even for the same type of virtual machines. Resource provisioning in geo-distributed clouds is an ...
Elastic resource provisioning for scientific workflow scheduling in cloud under budget and deadline constraints

With the popularization and development of cloud computing, lots of scientific computing applications are conducted in cloud environments. However, current application scenario of scientific computing is also becoming increasingly dynamic and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

HPDC '15: Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing

June 2015

296 pages

ISBN:9781450335508

DOI:10.1145/2749246

General Chair:
Thilo Kielmann
VU University Amsterdam, The Netherlands
,
Program Chairs:
Dean Hildebrand
IBM Research Almaden
,
Michela Taufer
University of Delaware

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

University of Arizona: University of Arizona
SIGARCH: ACM Special Interest Group on Computer Architecture

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 June 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

MOE Academic Research Funding
Singapore National Research Foundation

Conference

HPDC'15

Sponsor:

University of Arizona
SIGARCH

HPDC'15: The 24th International Symposium on High-Performance Parallel and Distributed Computing

June 15 - 19, 2015

Oregon, Portland, USA

Acceptance Rates

HPDC '15 Paper Acceptance Rate 19 of 116 submissions, 16%;

Overall Acceptance Rate 166 of 966 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
349
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhou AXue WXiao YHe BIbrahim SCheng R(2022)Taming System Dynamics on Resource Optimization for Data Processing Workflows: A Probabilistic ApproachIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2021.309140033:1(231-248)Online publication date: 1-Jan-2022
https://doi.org/10.1109/TPDS.2021.3091400
Hu JLi KLiu CLi K(2021)A Game-Based Price Bidding Algorithm for Multi-Attribute Cloud Resource ProvisionIEEE Transactions on Services Computing10.1109/TSC.2018.286002214:4(1111-1122)Online publication date: 1-Jul-2021
https://doi.org/10.1109/TSC.2018.2860022
Versluis LMatha RTalluri SHegeman TProdan RDeelman EIosup A(2020)The Workflow Trace Archive: Open-Access Data From Public and Private Computing InfrastructuresIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2020.298482131:9(2170-2184)Online publication date: 1-Sep-2020
https://doi.org/10.1109/TPDS.2020.2984821
Ahmad ZNazir BUmer A(2020)A fault‐tolerant workflow management system with Quality‐of‐Service‐aware scheduling for scientific workflows in cloud computingInternational Journal of Communication Systems10.1002/dac.464934:1Online publication date: 8-Nov-2020
https://doi.org/10.1002/dac.4649
Deelman EMandal AJiang MSakellariou R(2019)The role of machine learning in scientific workflowsThe International Journal of High Performance Computing Applications10.1177/1094342019852127(109434201985212)Online publication date: 30-May-2019
https://doi.org/10.1177/1094342019852127
Zhou AXiao YHe BIbrahim SCheng R(2019)Incorporating Probabilistic Optimizations for Resource Provisioning of Data Processing WorkflowsProceedings of the 48th International Conference on Parallel Processing10.1145/3337821.3337847(1-10)Online publication date: 5-Aug-2019
https://dl.acm.org/doi/10.1145/3337821.3337847
Zhao YCalheiros RVasilakos ABailey JSinnott R(2019)Profit Maximization and Time Minimization Admission Control and Resource Scheduling for Cloud-Based Big Data Analytics-as-a-Service PlatformsWeb Services – ICWS 201910.1007/978-3-030-23499-7_3(26-47)Online publication date: 14-Jun-2019
https://doi.org/10.1007/978-3-030-23499-7_3
Zhou APhan TIbrahim SHe B(2018)Energy-Efficient Speculative Execution using Advanced Reservation for Heterogeneous ClustersProceedings of the 47th International Conference on Parallel Processing10.1145/3225058.3225084(1-10)Online publication date: 13-Aug-2018
https://dl.acm.org/doi/10.1145/3225058.3225084
Zhou AHe BCheng XLau C(2017)A Declarative Optimization Engine for Resource Provisioning of Scientific Workflows in Geo-Distributed CloudsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2016.259952928:3(647-661)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1109/TPDS.2016.2599529
Phan TIbrahim SZhou AAupy GAntoniu G(2017)Energy-Driven Straggler Mitigation in MapReduceEuro-Par 2017: Parallel Processing10.1007/978-3-319-64203-1_28(385-398)Online publication date: 1-Aug-2017
https://doi.org/10.1007/978-3-319-64203-1_28
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents