More Web Proxy on the site http://driver.im/

research-article

Umbrella: A Portable Environment Creator for Reproducible Computing on Clusters, Clouds, and Grids

Authors:

Douglas ThainAuthors Info & Claims

VTDC '15: Proceedings of the 8th International Workshop on Virtualization Technologies in Distributed Computing

Pages 23 - 30

https://doi.org/10.1145/2755979.2755982

Published: 15 June 2015 Publication History

Abstract

Environment configuration is a significant challenge in large scale computing. An application that runs correctly on one carefully-prepared machine may fail completely on another machine, creating wasted effort and serious concerns about long-term reproducibility. Virtual machines and system containers provide a partial solution to this problem, in that they allow for the accurate reconstruction of an entire computing environment. However, when used directly, they have the dual problems of significant overhead and a lack of portability. To avoid this problem, we present Umbrella, a tool for specifying and materializing comprehensive execution environments from the hardware all the way up to software and data. A user simply invokes Umbrella with the desired task, and Umbrella determines the minimum mechanism necessary to run the task - direct execution, a system container, a local virtual machine, or submission to a cloud or grid environment. We present the overall design of Umbrella and demonstrate its use to precisely execute a high energy physics application across many platforms using combinations of chroot, Docker, Parrot, Condor, and Amazon EC2.

References

[1]

D. P. Anderson. BOINC: A System for Public-Resource Computing and Storage. In Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing, GRID '04, pages 4--10, Washington, DC, USA, 2004. IEEE Computer Society.

Digital Library

[2]

J. Blomer, P. Buncic, and T. Fuhrmann. CernVM-FS: delivering scientific software to globally distributed computing resources. In Proceedings of the first international workshop on Network-aware data management, pages 49--56. ACM, 2011.

Digital Library

[3]

R. Bolze, F. Cappello, E. Caron, M. Daydé, F. Desprez, E. Jeannot, Y. Jégou, S. Lanteri, J. Leduc, N. Melab, et al. Grid'5000: a large scale and highly reconfigurable experimental grid testbed. International Journal of High Performance Computing Applications, 20(4):481--494, 2006.

Digital Library

[4]

J. Estublier. Software configuration management: a roadmap. In Proceedings of the Conference on the Future of Software Engineering, pages 279--289. ACM, 2000.

Digital Library

[5]

S. Friedl. Go directly to jail: Secure untrusted applications with chroot. Linux Magazine, pages 2002--12, 2002.

[6]

R. P. Goldberg. Survey of virtual machine research. Computer, 7(6):34--45, 1974.

Digital Library

[7]

P. J. Guo and D. R. Engler. CDE: Using System Call Interposition to Automatically Create Portable Software Packages. In USENIX Annual Technical Conference, 2011.

Digital Library

[8]

L. Kanies. Puppet: Next-generation configuration management. The USENIX Magazine, 31(1):19--25, 2006.

[9]

S. Khatua and N. Mukherjee. A Novel Checkpointing Scheme for Amazon EC2 Spot Instances. 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, 0:180--181, 2013.

[10]

A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori. kvm: the Linux virtual machine monitor. In Proceedings of the Linux Symposium, volume 1, pages 225--230, 2007.

[11]

I. Krsul, A. Ganguly, J. Zhang, J. A. Fortes, and R. J. Figueiredo. Vmplants: Providing and managing virtual machine execution environments for grid computing. In Supercomputing, 2004. Proceedings of the ACM/IEEE SC2004 Conference, pages 7--7. IEEE, 2004.

Digital Library

[12]

A. Lenk, M. Klems, J. Nimis, S. Tai, and T. Sandholm. What's inside the Cloud? An architectural map of the Cloud landscape. In Proceedings of the 2009 ICSE Workshop on Software Engineering Challenges of Cloud Computing, pages 23--31. IEEE Computer Society, 2009.

Digital Library

[13]

R. McClatchey. The CMS experiment at the CERN LHC. The Journal of Instrumentation, 3(S08004), 2008.

[14]

D. Merkel. Docker: Lightweight Linux Containers for Consistent Development and Deployment. Linux J., 2014(239), Mar. 2014.

Digital Library

[15]

R. Pordes, D. Petravick, B. Kramer, D. Olson, M. Livny, A. Roy, P. Avery, K. Blackburn, T. Wenaus, F. Würthwein, et al. The open science grid. In Journal of Physics: Conference Series, volume 78, page 012057. IOP Publishing, 2007.

[16]

X.-H. Sun, C. Du, H. Zou, Y. Chen, and P. Shukla. V-mcs: A configuration system for virtual machines. In Cluster Computing and Workshops, 2009. CLUSTER'09. IEEE International Conference on, pages 1--7. IEEE, 2009.

[17]

D. Thain and M. Livny. Parrot: An application environment for data-intensive computing. Scalable Computing: Practice and Experience, 6(3):9--18, 2005.

[18]

D. Thain, T. Tannenbaum, and M. Livny. Condor and the Grid. Grid computing: Making the global infrastructure a reality, pages 299--335, 2003.

[19]

G. Von Laszewski, G. C. Fox, F. Wang, A. J. Younge, A. Kulshrestha, G. G. Pike, W. Smith, J. Voeckler, R. J. Figueiredo, J. Fortes, et al. Design of the futuregrid experiment management framework. In Gateway computing environments workshop (GCE), pages 1--10, 2010.

[20]

C. A. Waldspurger. Memory Resource Management in VMware ESX Server. SIGOPS Oper. Syst. Rev., 36(SI):181--194, Dec. 2002.

Digital Library

[21]

M. G. Xavier, M. V. Neves, F. D. Rossi, T. C. Ferreto, T. Lange, and C. A. De Rose. Performance evaluation of container-based virtualization for high performance computing environments. In Parallel, Distributed and Network-Based Processing (PDP), 2013 21st Euromicro International Conference on, pages 233--240. IEEE, 2013.

Digital Library

Cited By

Costa LBarbosa SCunha J(2024)Evaluating Tools for Enhancing Reproducibility in Computational Scientific ExperimentsProceedings of the 2nd ACM Conference on Reproducibility and Replicability10.1145/3641525.3663623(46-51)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3641525.3663623
Hamzaoui Duthil BCourboulay VMedromi H(2024)A Topical Review on Container-Based Cloud Revolution: Multi-Directional Challenges, and Future TrendsSN Computer Science10.1007/s42979-024-02763-y5:4Online publication date: 9-Apr-2024
https://dl.acm.org/doi/10.1007/s42979-024-02763-y
Wilkinson DOliveira LMosse DChilders B(2019)Long-term Preservation of Repeatable Builds in Occam2019 IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)10.1109/CANOPIE-HPC49598.2019.00008(21-30)Online publication date: Nov-2019
https://doi.org/10.1109/CANOPIE-HPC49598.2019.00008
Show More Cited By

Index Terms

Umbrella: A Portable Environment Creator for Reproducible Computing on Clusters, Clouds, and Grids
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
  2. Software notations and tools
    1. System description languages
      1. Specification languages

Recommendations

Integrating Containers into Workflows: A Case Study Using Makeflow, Work Queue, and Docker
VTDC '15: Proceedings of the 8th International Workshop on Virtualization Technologies in Distributed Computing

Workflows are a widely used abstraction for representing large scientific applications and executing them on distributed systems such as clusters, clouds, and grids. However, workflow systems have been largely silent on the question of precisely what ...
My VM is Lighter (and Safer) than your Container
SOSP '17: Proceedings of the 26th Symposium on Operating Systems Principles

Containers are in great demand because they are lightweight when compared to virtual machines. On the downside, containers offer weaker isolation than VMs, to the point where people run containers in virtual machines to achieve proper isolation. In this ...
Co-residency Attacks on Containers are Real
CCSW'20: Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop

Public clouds are inherently multi-tenant: applications deployed by different parties (including malicious ones) may reside on the same physical machines and share various hardware resources. With the introduction of newer hypervisors, containerization ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

VTDC '15: Proceedings of the 8th International Workshop on Virtualization Technologies in Distributed Computing

June 2015

48 pages

ISBN:9781450335737

DOI:10.1145/2755979

General Chairs:
Frédéric Desprez
Inria, France
,
Adrien Lebre
Inria, France
,
Program Chair:
Erik Elmroth
Umeå University, Sweden

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

University of Arizona: University of Arizona
SIGARCH: ACM Special Interest Group on Computer Architecture

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 June 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF

Conference

HPDC'15

Sponsor:

University of Arizona
SIGARCH

HPDC'15: The 24th International Symposium on High-Performance Parallel and Distributed Computing

June 15, 2015

Oregon, Portland, USA

Acceptance Rates

Overall Acceptance Rate 5 of 10 submissions, 50%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
208
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)1

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Costa LBarbosa SCunha J(2024)Evaluating Tools for Enhancing Reproducibility in Computational Scientific ExperimentsProceedings of the 2nd ACM Conference on Reproducibility and Replicability10.1145/3641525.3663623(46-51)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3641525.3663623
Hamzaoui Duthil BCourboulay VMedromi H(2024)A Topical Review on Container-Based Cloud Revolution: Multi-Directional Challenges, and Future TrendsSN Computer Science10.1007/s42979-024-02763-y5:4Online publication date: 9-Apr-2024
https://dl.acm.org/doi/10.1007/s42979-024-02763-y
Wilkinson DOliveira LMosse DChilders B(2019)Long-term Preservation of Repeatable Builds in Occam2019 IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)10.1109/CANOPIE-HPC49598.2019.00008(21-30)Online publication date: Nov-2019
https://doi.org/10.1109/CANOPIE-HPC49598.2019.00008
Ramon-Cortes CServen AEjarque JLezzi DBadia R(2018)Transparent Orchestration of Task-based Parallel Applications in Containers PlatformsJournal of Grid Computing10.5555/3190792.319080516:1(137-160)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.5555/3190792.3190805
Wilkinson DOliveira LMossé DChilders B(2018)Software ProvenanceProceedings of the First International Workshop on Practical Reproducible Evaluation of Computer Systems10.1145/3214239.3214244(1-6)Online publication date: 11-Jun-2018
https://dl.acm.org/doi/10.1145/3214239.3214244
Ivie PThain D(2018)Reproducibility in Scientific ComputingACM Computing Surveys10.1145/318626651:3(1-36)Online publication date: 16-Jul-2018
https://dl.acm.org/doi/10.1145/3186266
Ramon-Cortes CServen AEjarque JLezzi DBadia R(2018)Transparent Orchestration of Task-based Parallel Applications in Containers PlatformsJournal of Grid Computing10.1007/s10723-017-9425-z16:1(137-160)Online publication date: 1-Feb-2018
https://doi.org/10.1007/s10723-017-9425-z
Meng HThain D(2017)Facilitating the Reproducibility of Scientific Workflows with Execution Environment SpecificationsProcedia Computer Science10.1016/j.procs.2017.05.116108(705-714)Online publication date: 2017
https://doi.org/10.1016/j.procs.2017.05.116
Meng HThain DVyushkov AWolf MWoodard A(2016)Conducting reproducible research with Umbrella: Tracking, creating, and preserving execution environments2016 IEEE 12th International Conference on e-Science (e-Science)10.1109/eScience.2016.7870889(91-100)Online publication date: Oct-2016
https://doi.org/10.1109/eScience.2016.7870889
Ivie PThain D(2016)PRUNE: A preserving run environment for reproducible scientific computing2016 IEEE 12th International Conference on e-Science (e-Science)10.1109/eScience.2016.7870886(61-70)Online publication date: Oct-2016
https://doi.org/10.1109/eScience.2016.7870886

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents