poster

Stocator: a high performance object store connector for spark

Authors:

Francesco PaceAuthors Info & Claims

SYSTOR '17: Proceedings of the 10th ACM International Systems and Storage Conference

Article No.: 27, Page 1

https://doi.org/10.1145/3078468.3078496

Published: 22 May 2017 Publication History

Get Access

Abstract

Data is the natural resource of the 21st century. It is being produced at dizzying rates, e.g., for genomics by sequencers, for Media and Entertainment with very high resolution formats, and for Internet of Things (IoT) by multitudes of sensors. Object Stores such as AWS S3, Azure Blob storage, and IBM Cloud Object Storage, are highly scalable distributed storage systems that offer high capacity, cost effective storage for this data. But it is not enough just to store data; we also need to derive value from it. Apache Spark is the leading big data analytics processing engine. It runs up to one hundred times faster than Hadoop MapReduce and combines SQL, streaming and complex analytics. In this poster we present Stocator, a high performance storage connector, that enables Spark to work directly on data stored in object storage systems.

References

[1]

Hadoop-AWS Module: Integration with Amazon Web Services. https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html.

Google Scholar

[2]

Hadoop OpenStack Support: Swift Object Store. http://hadoop.apache.org/docs/current//hadoop-openstack/index.html.

Google Scholar

[3]

IBM Stocator Source Code. https://github.com/SparkTC/stocator.

Google Scholar

[4]

[SPARK-10063][SQL] Remove DirectParquetOutputCommitter. https://github.com/apache/spark/pull/12229.

Google Scholar

[5]

Using Apache Spark with Amazon S3. https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.11.0/bk_hdcloud-aws/content/s3-spark/index.html.

Google Scholar

Cited By

View all

Padulano VTejedor Saavedra EAlonso-Jordá PLópez Gómez JBlomer J(2022)A caching mechanism to exploit object store speed in High Energy Physics analysisCluster Computing10.1007/s10586-022-03757-226:5(2757-2772)Online publication date: 14-Oct-2022
https://doi.org/10.1007/s10586-022-03757-2
Gracia-Tinedo RSanchez-Artigas MGarcia-Lopez PMoatti YGluszak F(2019)Lamda-Flow: Automatic Pushdown of Dataflow Operators Close to the Data2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)10.1109/CCGRID.2019.00022(112-121)Online publication date: May-2019
https://doi.org/10.1109/CCGRID.2019.00022

Index Terms

Stocator: a high performance object store connector for spark
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Parallel and distributed DBMSs
        MapReduce-based systems
  2. Information storage systems
    1. Storage architectures
      1. Cloud based storage

Recommendations

Stocator: an object store aware connector for apache spark
SoCC '17: Proceedings of the 2017 Symposium on Cloud Computing

Data is the natural resource of the 21st century. It is being produced at dizzying rates, e.g., for genomics, for media and entertainment, and for Internet of Things. Object storage systems such as Amazon S3, Azure Blob storage, and IBM Cloud Object ...
Stocator: providing high performance and fault tolerance for apache spark over object storage
CCGrid '18: Proceedings of the 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

Until now object storage has not been a first-class citizen of the Apache Hadoop ecosystem including Apache Spark. Hadoop connectors to object storage have been based on file semantics, an impedance mismatch, which leads to low performance and the need ...
Using the Gfarm File System as a POSIX Compatible Storage Platform for Hadoop MapReduce Applications
GRID '11: Proceedings of the 2011 IEEE/ACM 12th International Conference on Grid Computing

MapReduce is a promising parallel programming model for processing large data sets. Hadoop is an up-and-coming open-source implementation of MapReduce. It uses the Hadoop Distributed File System (HDFS) to store input and output data. Due to a lack of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SYSTOR '17: Proceedings of the 10th ACM International Systems and Storage Conference

May 2017

195 pages

ISBN:9781450350358

DOI:10.1145/3078468

General Chair:
Doron Chen
IBM Research
,
Program Chairs:
Peter Desnoyers
Northeastern University
,
Eyal de Lara
University of Toronto

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

In-Cooperation

TCE: Technion Computer Engineering Center
USENIX Assoc: USENIX Assoc

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 May 2017

Check for updates

Qualifiers

Poster

Conference

SYSTOR'17

Sponsor:

SIGOPS

SYSTOR'17: International Systems and Storage Conference

May 22 - 24, 2017

Haifa, Israel

Acceptance Rates

Overall Acceptance Rate 108 of 323 submissions, 33%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
128
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Padulano VTejedor Saavedra EAlonso-Jordá PLópez Gómez JBlomer J(2022)A caching mechanism to exploit object store speed in High Energy Physics analysisCluster Computing10.1007/s10586-022-03757-226:5(2757-2772)Online publication date: 14-Oct-2022
https://doi.org/10.1007/s10586-022-03757-2
Gracia-Tinedo RSanchez-Artigas MGarcia-Lopez PMoatti YGluszak F(2019)Lamda-Flow: Automatic Pushdown of Dataflow Operators Close to the Data2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)10.1109/CCGRID.2019.00022(112-121)Online publication date: May-2019
https://doi.org/10.1109/CCGRID.2019.00022

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Stocator: an object store aware connector for apache spark

Stocator: providing high performance and fault tolerance for apache spark over object storage

Using the Gfarm File System as a POSIX Compatible Storage Platform for Hadoop MapReduce Applications