More Web Proxy on the site http://driver.im/

research-article

Exploring application and infrastructure adaptation on hybrid grid-cloud infrastructure

Authors:

Yaakoub el-Khamra,

Manish ParasharAuthors Info & Claims

HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Pages 402 - 412

https://doi.org/10.1145/1851476.1851536

Published: 21 June 2010 Publication History

Abstract

Clouds are emerging as an important class of distributed computational resources and are quickly becoming an integral part of production computational infrastructures. An important but oft-neglected question is, what new applications and application capabilities can be supported by clouds as part of a hybrid computational platform? In this paper we use the ensemble Kalman-filter based dynamic application workflow and investigate how clouds can be effectively used as an accelerator to address changing computational requirements as well as changing Quality of Service constraints (e.g., deadlines). Furthermore, we explore how application and system-level adaptivity can be used to improve application performance and achieve a more effective utilization of the hybrid platform. Specifically, we adapt the ensemble Kalman-filter based application formulation (serial versus parallel, different solvers etc.) so as to execute efficiently on a range of different infrastructure (from High Performance Computing grids to clouds that support single core and many-core virtual machines). Our results show that there are performance advantages to be had by supporting application and infrastructure level adaptivity. In general, we find that grid-cloud infrastructure can support novel usage modes, such as deadline-driven scheduling, for applications with tunable characteristics that can adapt to varying resource types.

References

[1]

}}C. Vecchiola, S. Pandey, and R. Buyya, "High-performance cloud computing: A view of scientific applications," in Pervasive Systems, Algorithms, and Networks (ISPAN), 2009 10th International Symposium on, dec. 2009, pp. 4--16.

Digital Library

[2]

}}S. Ostermann, R. Prodan, and T. Fahringer, "Extending grids with cloud resource management for scientific computing," in Grid Computing, 2009 10th IEEE/ACM International Conference on, oct. 2009, pp. 42--49.

[3]

}}K. Chine, "Scientific computing environments in the age of virtualization toward a universal platform for the cloud," in Open-source Software for Scientific Computation (OSSC), 2009 IEEE International Workshop on, sept. 2009, pp. 44--48.

[4]

}}Cyclone. http://www.sgi.com/products/hpc_cloud/cyclone/.

[5]

}}H. Kim, Y. el Khamra, S. Jha, and M. Parashar, "An autonomic approach to integrated hpc grid and cloud usage," in e-Science, 2009. e-Science '09. Fifth IEEE International Conference on, Dec. 2009, pp. 366--373.

Digital Library

[6]

}}Y. E. Khamra and S. Jha, "Title: Developing Autonomic Distributed Scientific Applications: A Case Study From History Matching Using Ensemble Kalman-Filters," in Sixth International Conference on Autonomic Computing, 2009. ICAC '09 (Barcelona). IEEE, 2009.

Digital Library

[7]

}}Y. Y. El-Khamra, "Real-time reservoir characterization and beyond: Cyberinfrastructure tools and technologies," Master's thesis, Louisiana State University, Baton Rouge, Louisiana, 2009.

[8]

}}M. D. de Assuncao, A. di Costanzo, and R. Buyya, "Evaluating the cost-benefit of using cloud computing to extend the capacity of clusters," in HPDC '09: Proceedings of the 18th ACM international symposium on High performance distributed computing. New York, NY, USA: ACM, 2009, pp. 141--150.

Digital Library

[9]

}}C. Vazquez, E. Huedo, R. Montero, and I. Llorente, "Dynamic provision of computing resources from grid infrastructures and cloud providers," in Grid and Pervasive Computing Conference, 2009. GPC '09. Workshops at the, may 2009, pp. 113--120.

Digital Library

[10]

}}T. Freeman and K. Keahey, "Flying low: Simple leases with workspace pilot," in Euro-Par, 2008, pp. 499--509.

Digital Library

[11]

}}H. Kim, S. Chaudhari, M. Parashar, and C. Marty, "Online risk analytics on the cloud," in Cluster Computing and the Grid, 2009. CCGRID '09. 9th IEEE/ACM International Symposium on, May 2009, pp. 484--489.

Digital Library

[12]

}}A. Ozer and C. Ozturan, "An auction based mathematical model and heuristics for resource co-allocation problem in grids and clouds," in Soft Computing, Computing with Words and Perceptions in System Analysis, Decision and Control, 2009. ICSCCW 2009. Fifth International Conference on, sept. 2009, pp. 1--4.

[13]

}}P. Martinaitis, C. Patten, and A. Wendelborn, "Remote interaction and scheduling aspects of cloud based streams," in E-Science Workshops, 2009 5th IEEE International Conference on, dec. 2009, pp. 39--47.

[14]

}}L. Nie and Z. Xu, "An adaptive scheduling mechanism for elastic grid computing," Semantics, Knowledge and Grid, International Conference on, vol. 0, pp. 184--191, 2009.

Digital Library

[15]

}}K. Czajkowski, S. Fitzgerald, I. Foster, and C. Kesselman, "Grid information services for distributed resource sharing," in High Performance Distributed Computing, 2001. Proceedings. 10th IEEE International Symposium on, 2001, pp. 181--194.

Digital Library

[16]

}}M. Siddiqui, A. Villazon, J. Hofer, and T. Fahringer, "Glare: A grid activity registration, deployment and provisioning framework," SC Conference, vol. 0, p. 52, 2005.

Digital Library

[17]

}}T. Dornemann, E. Juhnke, and B. Freisleben, "On-demand resource provisioning for bpel workflows using amazon's elastic compute cloud," in CCGRID '09: Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid. Washington, DC, USA: IEEE Computer Society, 2009, pp. 140--147.

Digital Library

[18]

}}S Jha, D S Katz, A Luckow, A Merzky and K Stamou, Understanding Scientific Applications for Cloud Environments, submitted to book on Cloud Computing, Edited by Raj Kumar Buyya, to be published by Wiley draft available at: http://cct.lsu.edu/~sjha/select\_publications/cloud_book_chapter.pdf.

[19]

}}S. Jha, M. Parashar, and O. Rana, "Investigating autonomic behaviours in grid-based computational science applications," in GMAC '09: Proceedings of the 6th international conference industry session on Grids meets autonomic computing. New York, NY, USA: ACM, 2009, pp. 29--38.

Digital Library

[20]

}}R. J. Al-Ali, K. Amin, G. von Laszewski, O. F. Rana, D. W. Walker, M. Hategan, and N. J. Zaluzec, "Analysis and Provision of QoS for Distributed Grid Applications," Journal of Grid Computing (Springer), vol. 2, no. 2, pp. 163--182, 2004.

[21]

}}R. E. Kalman, "A new approach to linear filtering and prediction problems." {Online}. Available: http://www.cs.unc.edu/~welch/kalman/media/pdf/Kalman1960.pdf

[22]

}}Y. Gu and D. S. Oliver, "An iterative ensemble kalman filter for multiphase fluid flow data assimilation," SPE Journal, vol. 12, no. 4, pp. 438--446, 2007.

[23]

}}X. Li, C. White, Z. Lei, and G. Allen, "Reservoir model updating by ensemble kalman filter-practical approaches using grid computing technology," in Petroleum Geostatistics 2007, Cascais, Portugal, August 2007.

[24]

}}Y. Gu and D. S. Oliver, "The ensemble kalman filter for continuous updating of reservoir simulation models," Journal of Engineering Resources Technology, vol. 128, no. 1, pp. 79--87, 2006.

[25]

}}H. Kim, M. Parashar, L. Yang, and D. Foran, "Investigating the use of cloudbursts for high throughput medical image registration," in Proceedings of the 10th IEEE/ACM International Conference on Grid Computing (Grid 2009), 2009.

[26]

}}I. Stoica, R. Morris, D. Liben-Nowell, D. R. Karger, M. F. Kaashoek, F. Dabek, and H. Balakrishnan, "Chord: A scalable peer-to-peer lookup protocol for internet applications," in ACM SIGCOMM, 2001, pp. 149--160.

Digital Library

[27]

}}C. Schmidt and M. Parashar, "Squid: Enabling search in dht-based systems," J. Parallel Distrib. Comput., vol. 68, no. 7, pp. 962--975, 2008.

Digital Library

[28]

}}N. Carriero and D. Gelernter, "Linda in context," Commun. ACM, vol. 32, no. 4, pp. 444--458, 1989.

Digital Library

[29]

}}J. Dean and S. Ghemawat, "Mapreduce: simplified data processing on large clusters," Commun. ACM, vol. 51, no. 1, pp. 107--113, 2008.

Digital Library

[30]

}}Hadoop. http://hadoop.apache.org/core/.

[31]

}}Thain D, Tannenbaum T and Livny M 2005 Distributed Computing in Practice: The Condor Experience Concurrency - Practice and Experience 17 2--4 323--56.

Digital Library

[32]

}}S. Balay, K. Buschelman, W. D. Gropp, D. Kaushik, M. G. Knepley, L. C. McInnes, B. F. Smith, and H. Zhang, "PETSc Web page," 2001, http://www.mcs.anl.gov/petsc.

Cited By

Hoppe DSandoval YSulistio AMalawski MBalis BPawlik MFigiela KKrol DOrzechowski MKitowski JBubak M(2018)Bridging the Gap Between HPC and Cloud Using HyperFlow and PaaSageParallel Processing and Applied Mathematics10.1007/978-3-319-78024-5_38(432-442)Online publication date: 23-Mar-2018
https://doi.org/10.1007/978-3-319-78024-5_38
Feoktistov AGorsky SSidorov IKostromin REdelev AMassel L(2018)Orlando Tools: Energy Research Application Development Through Convergence of Grid and Cloud ComputingSupercomputing10.1007/978-3-030-05807-4_25(289-300)Online publication date: 31-Dec-2018
https://doi.org/10.1007/978-3-030-05807-4_25
Fakhfakh FKacem HKacem A(2017)CloudSim4DWf: A CloudSim-extension for simulating dynamic workflows in a cloud environment2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA)10.1109/SERA.2017.7965728(195-202)Online publication date: Jun-2017
https://doi.org/10.1109/SERA.2017.7965728
Show More Cited By

Index Terms

Exploring application and infrastructure adaptation on hybrid grid-cloud infrastructure
1. Computing methodologies
  1. Concurrent computing methodologies
    1. Concurrent programming languages
2. Software and its engineering
  1. Software notations and tools
    1. General programming languages
      1. Language types
        Concurrent programming languages

Recommendations

Autonomic management of application workflows on hybrid computing infrastructure
Science-Driven Cloud Computing

In this paper, we present a programming and runtime framework that enables the autonomic management of complex application workflows on hybrid computing infrastructures. The framework is designed to address system and application heterogeneity and ...
Cloud Storage as the Infrastructure of Cloud Computing
ICICCI '10: Proceedings of the 2010 International Conference on Intelligent Computing and Cognitive Informatics

As an emerging technology and business paradigm, Cloud Computing has taken commercial computing by storm. Cloud computing platforms provide easy access to a company’s high-performance computing and storage infrastructure through web services. With cloud ...
Harnessing Cloud Technologies for a Virtualized Distributed Computing Infrastructure

The InterGrid system aims to provide an execution environment for running applications on top of interconnected infrastructures. The system uses virtual machines as building blocks to construct execution environments that span multiple computing sites. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

June 2010

911 pages

ISBN:9781605589428

DOI:10.1145/1851476

General Chairs:
Salim Hariri
University of Arizona
,
Kate Keahey
University of Chicago

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

University of Arizona: University of Arizona
SIGARCH: ACM Special Interest Group on Computer Architecture

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 June 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

HPDC '10

Sponsor:

University of Arizona
SIGARCH

HPDC '10: The 19th International Symposium on High Performance Distributed Computing

June 21 - 25, 2010

Illinois, Chicago

Acceptance Rates

Overall Acceptance Rate 166 of 966 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
669
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hoppe DSandoval YSulistio AMalawski MBalis BPawlik MFigiela KKrol DOrzechowski MKitowski JBubak M(2018)Bridging the Gap Between HPC and Cloud Using HyperFlow and PaaSageParallel Processing and Applied Mathematics10.1007/978-3-319-78024-5_38(432-442)Online publication date: 23-Mar-2018
https://doi.org/10.1007/978-3-319-78024-5_38
Feoktistov AGorsky SSidorov IKostromin REdelev AMassel L(2018)Orlando Tools: Energy Research Application Development Through Convergence of Grid and Cloud ComputingSupercomputing10.1007/978-3-030-05807-4_25(289-300)Online publication date: 31-Dec-2018
https://doi.org/10.1007/978-3-030-05807-4_25
Fakhfakh FKacem HKacem A(2017)CloudSim4DWf: A CloudSim-extension for simulating dynamic workflows in a cloud environment2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA)10.1109/SERA.2017.7965728(195-202)Online publication date: Jun-2017
https://doi.org/10.1109/SERA.2017.7965728
Fakhfakh FKacem HKacem A(2017)Dealing with structural changes on provisioning resources for deadline-constrained workflowThe Journal of Supercomputing10.1007/s11227-016-1823-773:7(2896-2918)Online publication date: 1-Jul-2017
https://dl.acm.org/doi/10.1007/s11227-016-1823-7
Fakhfakh FKacem HKacem A(2015)Towards a Provisioning Algorithm for Dynamic Workflows in the CloudProceedings of the 2015 IEEE 24th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises10.1109/WETICE.2015.16(35-40)Online publication date: 15-Jun-2015
https://dl.acm.org/doi/10.1109/WETICE.2015.16
Fakhfakh FKacem HKacem A(2015)A Provisioning Approach of Cloud Resources for Dynamic WorkflowsProceedings of the 2015 IEEE 8th International Conference on Cloud Computing10.1109/CLOUD.2015.69(469-476)Online publication date: 27-Jun-2015
https://dl.acm.org/doi/10.1109/CLOUD.2015.69
Clematis AD'Agostino DGalizia AQuarati A(2014)Profiling e-Science infrastructures with kernel and application benchmarksInternational Journal of Grid and Utility Computing10.1504/IJGUC.2014.0602045:2(123-134)Online publication date: 1-Mar-2014
https://dl.acm.org/doi/10.1504/IJGUC.2014.060204
Das AAdhikary TRazzaque MCho EHong CKim DKim SLee SHanzo KIsmail R(2014)A QoS and profit aware cloud confederation model for IaaS service providersProceedings of the 8th International Conference on Ubiquitous Information Management and Communication10.1145/2557977.2558064(1-7)Online publication date: 9-Jan-2014
https://dl.acm.org/doi/10.1145/2557977.2558064
Król DOrzechowski MKitowski JNiethammer CSulisto AWafai A(2014)A Cloud-Based Data Farming Platform for Molecular Dynamics SimulationsProceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing10.1109/UCC.2014.89(579-584)Online publication date: 8-Dec-2014
https://dl.acm.org/doi/10.1109/UCC.2014.89
Zhang JDong FShen DLuo J(2014)Game theory based dynamic resource allocation for hybrid environment with cloud and big data application2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC.2014.6974065(1128-1133)Online publication date: Oct-2014
https://doi.org/10.1109/SMC.2014.6974065
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents