[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

Orchestrating Complex Application Architectures in Heterogeneous Clouds

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

Private cloud infrastructures are now widely deployed and adopted across technology industries and research institutions. Although cloud computing has emerged as a reality, it is now known that a single cloud provider cannot fully satisfy complex user requirements. This has resulted in a growing interest in developing hybrid cloud solutions that bind together distinct and heterogeneous cloud infrastructures. In this paper we describe the orchestration approach for heterogeneous clouds that has been implemented and used within the INDIGO-DataCloud project. This orchestration model uses existing open-source software like OpenStack and leverages the OASIS Topology and Specification for Cloud Applications (TOSCA) open standard as the modeling language. Our approach uses virtual machines and Docker containers in an homogeneous and transparent way providing consistent application deployment for the users. This approach is illustrated by means of two different use cases in different scientific communities, implemented using the INDIGO-DataCloud solutions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aguilar Gómez, F., de Lucas, J.M., García, D., Monteoliva, A.: Hydrodynamics and water quality forecasting over a cloud computing environment: indigo-datacloud. In: EGU General Assembly Conference Abstracts, vol. 19, p 9684 (2017)

  2. de Alfonso, C., Caballer, M., Alvarruiz, F., Hernández, V.: An energy management system for cluster infrastructures. Comput. Electr. Eng. 39(8), 2579–2590 (2013). http://www.sciencedirect.com/science/article/pii/S0045790613001365

    Article  Google Scholar 

  3. Amazon Web Services (AWS): Amazon Web Services (AWS). https://aws.amazon.com/ (2017)

  4. Amazon Web Services (AWS): CloudFormation. https://aws.amazon.com/cloudformation/ (2017)

  5. Apache Software Foundation: Apache Mesos. http://mesos.apache.org/ (2017)

  6. ARIA, ARIA. http://ariatosca.incubator.apache.org/ (2017)

  7. Bumpus, W.: NIST Cloud Computing Standards Roadmap. Tech. rep., National Institute of Standards and Technology (NIST). https://doi.org/10.6028/NIST.SP.500-291r2 (2013)

  8. Caballer, M., Blanquer, I., Moltó, G., de Alfonso, C.: Dynamic management of virtual infrastructures. J Grid Comput. 13(1), 53–70 (2015). https://doi.org/10.1007/s10723-014-9296-5

    Article  Google Scholar 

  9. Campos Plasencia, I., Fernández-del Castillo, E., Heinemeyer, S., López García, Á., Pahlen, F., Borges, G.: Phenomenology tools on cloud infrastructures using OpenStack. Eur. Phys. J. C 73(4), 2375 (2013). https://doi.org/10.1140/epjc/s10052-013-2375-0

    Article  Google Scholar 

  10. Celar: Celar. http://www.cloudwatchhub.eu/celar (2017)

  11. Chen, Y., de Lucas, J.M., Aguilar, F., Fiore, S., Rossi, M., Ferrari, T.: Indigo: building a datacloud framework to support open science. In: EGU General Assembly Conference Abstracts, vol. 18, p 16610 (2016)

  12. Chronos: Chronos. https://mesos.github.io/chronos/ (2017)

  13. Cloudify: Cloudify. http://getcloudify.org (2017)

  14. Davidović, D., Cetinić, E., Skala, K.: European research area and digital humanities

  15. Distefano, S., Serazzi, G.: Performance driven WS orchestration and deployment in service oriented infrastructure. J Grid Comput. 12(2), 347–369 (2014). https://doi.org/10.1007/s10723-014-9293-8

    Article  Google Scholar 

  16. EGI FedCloud: EGI FedCloud. https://www.egi.eu/federation/egi-federated-cloud/ (2017)

  17. Eucalyptus: Eucalyptus. https://www.eucalyptus.com/ (2017)

  18. Fiore, S., D’Anca, A., Palazzo, C., Foster, I., Williams, D.N., Aloisio, G.: Ophidia: toward big data analytics for eScience. Procedia Comput. Sci. 18, 2376–2385 (2013). https://doi.org/10.1016/j.procs.2013.05.409

    Article  Google Scholar 

  19. Fiore, S., Palazzo, C., D’Anca, A., Elia, D., Londero, E., Knapic, C., Monna, S., Marcucci, N.M., Aguilar, F., Płóciennik, M., et al.: Big data analytics on large-scale scientific datasets in the indigo-datacloud project. In: Proceedings of the Computing Frontiers Conference, pp 343–348. ACM (2017)

  20. Fiore, S., Płóciennik, M., Doutriaux, C., Palazzo, C., Boutte, J., żok, T., Elia, D., Owsiak, M., D’Anca, A., Shaheen, Z., et al.: Distributed and cloud-based multi-model analytics experiments on large volumes of climate change data in the earth system grid federation eco-system. In: 2016 IEEE International Conference on Big Data (Big Data), pp 2911–2918. IEEE (2016)

  21. Galante, G., Erpen de Bona, L.C., Mury, A.R., Schulze, B., da Rosa Righi, R.: An analysis of public clouds elasticity in the execution of scientific applications: a survey. J Grid Comput.,1–24. https://doi.org/10.1007/s10723-016-9361-3 (2016)

  22. Google Cloud Platform (GCP): Google Cloud Platform (GCP). https://cloud.google.com/ (2017)

  23. Hochstein, L. (ed.): Ansible: Up and Running, Automating Configuration Management and Deployment the Easy Way. O’Reilly Media (2014)

  24. Idabc: European Interoperability Framework for pan-European eGovernment Services. European Commission version 1, 1–25. https://doi.org/10.1109/HICSS.2007.68 (2004)

  25. IM: IM. http://www.grycap.upv.es/im (2017)

  26. INDIGO-DataCloud: D1.8 - General Architecture. Tech. rep., INDIGO-DataCloud Consortium (2015)

  27. INDIGO-DataCloud: Ansible Galaxy repository for INDIGO-DataCloud. https://galaxy.ansible.com/indigo-dc/ (2017)

  28. INDIGO-DataCloud: Disvis/Powerfit Ansible Role in Ansible Galaxy. https://galaxy.ansible.com/indigo-dc/disvis-powerfit/ (2017)

  29. INDIGO-DataCloud: INDIGO-DataCloud. https://www.indigo-datacloud.eu/ (2017)

  30. INDIGO-DataCloud: INDIGO-DataCloud DockerHub application repository. https://hub.docker.com/u/indigodatacloudapps/ (2017)

  31. INDIGO-DataCloud: INDIGO-DataCloud PaaS Orchestrator. https://github.com/indigo-dc/orchestrator (2017)

  32. INDIGO-DataCloud: INDIGO-DataCloud RepoSync. https://github.com/indigo-dc/java-reposync(2017)

  33. INDIGO-DataCloud: INDIGO-DataCloud TOSCA templates. https://github.com/indigo-dc/tosca-templates (2017)

  34. INDIGO-DataCloud: TOSCA Across Clouds. https://github.com/indigo-dc/tosca-types/blob/master/examples/web_mysql_tosca_across_clouds.yaml (2017)

  35. INDIGO-DataCloud: TOSCA template for deploying an Elastic Mesos Cluster. http://github.com/indigo-dc/tosca-types/blob/master/examples/mesos_elastic_cluster.yaml (2017)

  36. INDIGO-DataCloud: TOSCA template for Powerfit application. https://github.com/indigo-dc/tosca-types/blob/master/examples/powerfit.yaml (2017)

  37. Kacsuk, P., Kecskemeti, G., Kertesz, A., Nemeth, Z., Kovȧcs, J., Farkas, Z.: Infrastructure aware scientific workflows and infrastructure aware workflow managers in science gateways. J Grid Comput., 641–654. https://doi.org/10.1007/s10723-016-9380-0 (2016)

  38. Korambath, P., Wang, J., Kumar, A., Hochstein, L., Schott, B., Graybill, R., Baldea, M., Davis, J.: Deploying kepler workflows as services on a cloud infrastructure for smart manufacturing. Procedia Comput. Sci. 29, 2254–2259 (2014)

    Article  Google Scholar 

  39. Koski, K., Hormia-Poutanen, K., Chatzopoulos, M., Legrė, Y., Day, B.: Position Paper: European Open Science Cloud for Research. Tech. Rep. october, EUDAT, LIBER, OpenAIRE, EGI, GĖANT Bari (2015)

  40. Krieger, M.T., Torreno, O., Trelles, O., Kranzlmüller, D.: Building an open source cloud environment with auto-scaling resources for executing bioinformatics and biomedical workflows. Futur. Gener. Comput. Syst. 67, 329–340 (2017). https://doi.org/10.1016/j.future.2016.02.008

    Article  Google Scholar 

  41. Kurkcuoglu Soner, Z., Bonvin, A.: Science in the clouds: virtualizing haddock powerfit and disvis using indigo-datacloud solutions (2016)

  42. Lipton, P.C.T., Moser, S.I., Palma, D.V., Spatzier, T.I.: Topology and Orchestration Specification for Cloud Applications. Tech. rep., OASIS Standard (2013)

  43. Liu, C., Mao, Y., Van der Merwe, J., Fernandez, M.: Cloud Resource Orchestration: a Data-Centric Approach. In: Proceedings of the Biennial Conference on Innovative Data Systems Research (CIDR), pp 1–8. Citeseer (2011)

  44. López García, Á., Fernández-del Castillo, E.: Analysis of scientific cloud computing requirements. In: Proceedings of the IBERGRID 2013 Conference, p 147 158 (2013)

  45. López García, Á., Fernández-del Castillo, E., Orviz Fernández, P.: Standards for enabling heterogeneous IaaS cloud federations. Comput. Standard Inter. 47, 19–23 (2016). https://doi.org/10.1016/j.csi.2016.02.002

    Article  Google Scholar 

  46. López García, Á., Zangrando, L., Sgaravatto, M., Llorens, V., Vallero, S., Zaccolo, V., Bagnasco, S., Taneja, S., Pra, S.D., Salomoni, D., Donvito G.: Improved cloud resource allocation: how INDIGO-datacloud is overcoming the current limitations in cloud schedulers. arXiv:1707.06403 (2017)

  47. Lorido-Botran, T., Miguel-Alonso, J., Lozano, J.A.: A review of auto-scaling techniques for elastic applications in cloud environments. J Grid Comput. 12(4), 559–592 (2014). https://doi.org/10.1007/s10723-014-9314-7

    Article  Google Scholar 

  48. Marathon: Marathon. https://mesosphere.github.io/marathon/ (2017)

  49. Metsch, T., Edmonds, A.: Open Cloud Computing Interface-Infrastructure. Tech. rep., Open Grid Forum (2010)

  50. Metsch, T., Edmonds, A.: Open Cloud Computing Interface-RESTful HTTP Rendering. Tech. rep., Open Grid Forum (2011)

  51. Microsoft Azure: Microsoft Azure. https://azure.microsoft.com/ (2017)

  52. Moltó, G., Caballer, M., Pérez, A., Alfonso, D.C., Blanquer, I.: Coherent application delivery on hybrid distributed computing infrastructures of virtual machines and docker containers. In: 2017 25Th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP). https://doi.org/10.1109/PDP.2017.29, pp 486–490 (2017)

  53. Monna, S., Marcucci, N.M., Marinaro, G., Fiore, S., D’Anca, A., Antonacci, M., Beranzoli, L., Favali, P.: An Emso data case study within the indigo-Dc project. In: EGU General Assembly Conference Abstracts, vol. 19, p 12493 (2017)

  54. Nyrén, R., Metsch, T., Edmonds, A., Papaspyrou, A.: Open Cloud Computing Interface–Core. Tech. rep., Open Grid Forum (2010)

  55. OASIS: Organization for the Advancement of Structured Information Standards (OASIS). https://www.oasis-open.org (2015)

  56. Open Telekom Cloud (OTC): Open Telekom Cloud (OTC). https://cloud.telekom.de/en/ (2017)

  57. OpenNebula: OneFlow. http://docs.opennebula.org/5.2/advanced_components/application_flow_and_auto-scaling/index.html (2017)

  58. OpenNebula Project: OpenNebula. https://www.opennebula.org (2017)

  59. OpenStack Foundation: Heat Orchestration Template (HOT) Guide. https://docs.openstack.org/heat/latest/template_guide/hot_guide.html (2017)

  60. OpenStack Foundation: OpenStack. https://www.openstack.org (2017)

  61. OpenStack Foundation: Openstack Heat. http://wiki.openstack.org/wiki/Heat (2017)

  62. OpenStack Foundation: OpenStack Heat Translator. https://github.com/openstack/heat-translator(2017)

  63. OpenStack Foundation: OpenStack heat-translator project contribution statistics. http://stackalytics.com/?release=all&metric=commits&module=heat-translator (2017)

  64. OpenStack Foundation: OpenStack Tacker. https://wiki.openstack.org/wiki/Tacker (2017)

  65. OpenStack Foundation: OpenStack tosca-parser project contribution statistics. http://stackalytics.com/?release=all&metric=commits&module=tosca-parser (2017)

  66. OpenStack Foundation: TOSCA Parser. https://github.com/openstack/tosca-parser (2017)

  67. OpenTOSCA: OpenTOSCA. http://www.opentosca.org/ (2017)

  68. Owsiak, M., Plociennik, M., Palak, B., Zok, T., Reux, C., Di Gallo, L., Kalupin, D., Johnson, T., Schneider, M.: Running simultaneous kepler sessions for the parallelization of parametric scans and optimization studies applied to complex workflows. J Comput. Sci. 20, 103–111 (2017)

    Google Scholar 

  69. Palma, D., Rutkowski, M., Spatzier T.: TOSCA Simple Profile in YAML Version 1.1. Tech. rep., OASIS Standard. http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.1/TOSCA-Simple-Profile-YAML-v1.1.html (2016)

  70. Petcu, D.: Consuming resources and services from multiple clouds: from terminology to cloudware support. J Grid Comput. 12(2), 321–345 (2014). https://doi.org/10.1007/s10723-013-9290-3

    Article  Google Scholar 

  71. Plóciennik, M., Fiore, S., Donvito, G., Owsiak, M., Fargetta, M., Barbera, R., Bruno, R., Giorgio, E., Williams, D.N., Aloisio, G.: Two-level dynamic workflow orchestration in the INDIGO DataCloud for large-scale, climate change data analytics experiments. Procedia Comput. Sci. 80, 722–733 (2016). https://doi.org/10.1016/j.procs.2016.05.359

    Article  Google Scholar 

  72. Płóciennik, M., Fiore, S., Donvito, G., Owsiak, M., Fargetta, M., Barbera, R., Bruno, R., Giorgio, E., Williams, D.N., Aloisio, G.: Two-level dynamic workflow orchestration in the indigo datacloud for large-scale, climate change data analytics experiments. Procedia Comput. Sci. 80, 722–733 (2016)

    Article  Google Scholar 

  73. Python: Python Package Index (PyPI). https://pypi.python.org/pypi (2017)

  74. Ramakrishnan, L., Jackson, K.R., Canon, S., Cholia, S., Shalf, J.: Defining future platform requirements for e-Science clouds. In: Proceedings of the 1st ACM Symposium on Cloud Computing - SoCC ’10. https://doi.org/10.1145/1807128.1807145, p 101 (2010)

  75. Ramakrishnan, L., Zbiegel, P.T.T.T.: Magellan: experiences from a science cloud. In: Proceedings of the 2Nd International Workshop on Scientific Cloud Computing. http://dl.acm.org/citation.cfm?id=1996119, pp 49–58 (2011)

  76. Salomoni, D., Campos, I., Gaido, L., Donvito, G., Antonacci, M., Fuhrman, P., Marco, J., Lopez-Garcia, A., Orviz, P., Blanquer, I., et al.: Indigo-datacloud: foundations and architectural description of a platform as a service oriented to scientific computing. arXiv:http://arXiv.org/abs/1603.09536 (2016)

  77. Sánchez-Expósito, S., Martín, P., Ruiz, J.E., Verdes-Montenegro, L., Garrido, J., Sirvent, R., Falcó, A.R., Badia, R.M., Lezzi, D.: Web services as building blocks for science gateways in astrophysics. J Grid Comput. 14(4), 673–685 (2016). https://doi.org/10.1007/s10723-016-9382-y

    Article  Google Scholar 

  78. SlipStream: SlipStream. http://sixsq.com/products/slipstream/ (2017)

  79. Stockton, D.B., Santamaria, F.: Automating NEURON simulation deployment in cloud resources. Neuroinformatics 15(1), 51–70 (2017). https://doi.org/10.1007/s12021-016-9315-8

    Article  Google Scholar 

  80. Teckelmann, R., Reich, C., Sulistio, A.: Mapping of Cloud Standards to the Taxonomy of Interoperability in Iaas. In: 2011 IEEE Third International Conference on Cloud Computing Technology and Science (Cloudcom), pp 522–526. IEEE (2011)

  81. Toor, S., Osmani, L., Eerola, P., Kraemer, O., Lindén, T., Tarkoma, S., White, J.: A scalable infrastructure for CMS data analysis based on OpenStack Cloud and Gluster file system. J Phys.: Conf. Ser. 513(6), 062,047 (2014). https://doi.org/10.1088/1742-6596/513/6/062047. http://stacks.iop.org/1742-6596/513/i=6/a=062047?key=crossref.84033a04265ce343371c7f38064e7143

    Google Scholar 

  82. UK Government Cabinet Office: Open Standards Principles. https://www.gov.uk/government/publications/open-standards-principles/open-standards-principles (2015)

  83. Yangui, S., Marshall, I.J., Laisne, J.P., Tata, S.: Compatibleone: the open source cloud broker. J Grid Comput. 12(1), 93–109 (2014)

    Article  Google Scholar 

  84. Zhao, Y., Li, Y., Raicu, I., Lu, S., Tian, W., Liu, H.: Enabling scalable scientific workflow management in the cloud. Futur. Gener. Comput. Syst. 46, 3–16 (2015). https://doi.org/10.1016/j.future.2014.10.023

    Article  Google Scholar 

  85. van Zundert, G., Trellet, M., Schaarschmidt, J., Kurkcuoglu, Z., David, M., Verlato, M., Rosato, A., Bonvin, A.: The DisVis and PowerFit web servers: explorative and integrative modeling of biomolecular complexes. J. Mol. Biol. 429(3), 399–407 (2013). http://www.sciencedirect.com/science/article/pii/S0022283616305277

    Article  Google Scholar 

Download references

Acknowledgements

The authors want to acknowledge the support of the INDIGO-Datacloud (grant number 653549) project, funded by the European Commission’s Horizon 2020 Framework Program.

Author information

Authors and Affiliations

  1. Instituto de Física de Cantabria, Centro Mixto CSIC - UC, Santander, Spain

    Álvaro López García & Pablo Orviz Fernández

  2. Instituto de Instrumentación para Imagen Molecular (I3M), Centro Mixto CSIC, Universitat Politècnica de València, Valencia, Spain

    Miguel Caballer & Germán Moltó

  3. IBM Raleigh, Raleigh, NC, 27609, USA

    Sahdev Zala

  4. European Organization for Nuclear Research (CERN), Geneva, Switzerland

    Mathieu Velten

Authors
  1. Miguel Caballer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sahdev Zala
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Álvaro López García
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Germán Moltó
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pablo Orviz Fernández
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mathieu Velten
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Álvaro López García.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Caballer, M., Zala, S., García, Á. et al. Orchestrating Complex Application Architectures in Heterogeneous Clouds. J Grid Computing 16, 3–18 (2018). https://doi.org/10.1007/s10723-017-9418-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-017-9418-y

Keywords

Search

Navigation