demonstration

ReStore: reusing results of MapReduce jobs in pig

Authors:

Iman Elghandour,

Ashraf AboulnagaAuthors Info & Claims

SIGMOD '12: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data

Pages 701 - 704

https://doi.org/10.1145/2213836.2213937

Published: 20 May 2012 Publication History

Get Access

Abstract

Analyzing large scale data has become an important activity for many organizations, and is now facilitated by the MapReduce programming and execution model and its implementations, most notably Hadoop. Query languages such as Pig Latin, Hive, and Jaql make it simpler for users to express complex analysis tasks, and the compilers of these languages translate these complex tasks into workflows of MapReduce jobs. Each job in these workflows reads its input from the distributed file system used by the MapReduce system (e.g., HDFS in the case of Hadoop) and produces output that is stored in this distributed file system. This output is then read as input by the next job in the workflow. The current practice is to delete these intermediate results from the distributed file system at the end of executing the workflow. It would be more useful if these intermediate results can be stored and reused in future workflows. We demonstrate ReStore, an extension to Pig that enables it to manage storage and reuse of intermediate results of the MapReduce workflows executed in the Pig data analysis system. ReStore matches input workflows of MapReduce jobs with previously executed jobs and rewrites these workflows to reuse the stored results of the matched jobs. ReStore also creates additional reuse opportunities by materializing and reserving the output of query execution operators that are executed within a MapReduce job. In this demonstration we showcase the MapReduce jobs and sub-jobs recommended by ReStore for a given Pig query, the rewriting of input queries to reuse stored intermediate results, and a what-if analysis of the effectiveness of reusing stored outputs of previously executed jobs.

References

[1]

Apache Hadoop. http://hadoop.apache.org/.

Google Scholar

[2]

PigMix. http://wiki.apache.org/pig/PigMix.

Google Scholar

[3]

J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In Proc. OSDI, pages 137--150, 2004.

Digital Library

Google Scholar

[4]

I. Elghandour and A. Aboulnaga. Restore: Reusing results of mapreduce jobs. Proc. VLDB Endow. (PVLDB), 5(6):586--597, 2012.

Digital Library

Google Scholar

[5]

C. Olston et al. Pig Latin: a not-so-foreign language for data processing. In Proc. SIGMOD, pages 1099--1110, 2008.

Digital Library

Google Scholar

Cited By

View all

Alsudais SKumar ALi C(2023)RavenProceedings of the Workshop on Human-In-the-Loop Data Analytics10.1145/3597465.3605219(1-7)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3597465.3605219
Kaoudi ZManolescu IZampetakis S(2020)Cloud-Based RDF Data ManagementSynthesis Lectures on Data Management10.2200/S00986ED1V01Y202001DTM06215:1(1-103)Online publication date: 24-Feb-2020
https://doi.org/10.2200/S00986ED1V01Y202001DTM062
Sakr SSakr S(2020)Conclusions and OutlookBig Data 2.0 Processing Systems10.1007/978-3-030-44187-6_7(127-133)Online publication date: 10-Jul-2020
https://doi.org/10.1007/978-3-030-44187-6_7
Show More Cited By

Index Terms

ReStore: reusing results of MapReduce jobs in pig
1. Information systems
  1. Data management systems

Recommendations

Piglet: Interactive and Platform Transparent Analytics for RDF & Dynamic Data
WWW '16 Companion: Proceedings of the 25th International Conference Companion on World Wide Web

Data analytics has gained more and more focus during recent years and many data processing platforms have been developed. They all provide a powerful but often complex API that users have to learn. Furthermore, results can only be stored or printed, ...
The optimization for recurring queries in big data analysis system with MapReduce
Abstract
As data-intensive cluster computing systems like MapReduce grow in popularity, there is a strong need to promote the efficiency. Recurring queries, repeatedly being executed for long periods of time on rapidly evolving data-intensive ...
Highlights
- A new recurring queries model for MapReduce was proposed.
- Data scheduling in ...
ReStore: reusing results of MapReduce jobs

Analyzing large scale data has emerged as an important activity for many organizations in the past few years. This large scale data analysis is facilitated by the MapReduce programming and execution model and its implementations, most notably Hadoop. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SIGMOD '12: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data

May 2012

886 pages

ISBN:9781450312479

DOI:10.1145/2213836

General Chairs:
K. Selçuk Candan
Arizona State University
,
Yi Chen
Arizona State University
,
Richard Snodgrass
University of Arizona
,
Program Chair:
Luis Gravano
Columbia University
,
Publications Chair:
Ariel Fuxman
Microsoft Research

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 May 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Demonstration

Conference

SIGMOD/PODS '12

Sponsor:

SIGMOD

SIGMOD/PODS '12: International Conference on Management of Data

May 20 - 24, 2012

Arizona, Scottsdale, USA

Acceptance Rates

SIGMOD '12 Paper Acceptance Rate 48 of 289 submissions, 17%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
417
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Alsudais SKumar ALi C(2023)RavenProceedings of the Workshop on Human-In-the-Loop Data Analytics10.1145/3597465.3605219(1-7)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3597465.3605219
Kaoudi ZManolescu IZampetakis S(2020)Cloud-Based RDF Data ManagementSynthesis Lectures on Data Management10.2200/S00986ED1V01Y202001DTM06215:1(1-103)Online publication date: 24-Feb-2020
https://doi.org/10.2200/S00986ED1V01Y202001DTM062
Sakr SSakr S(2020)Conclusions and OutlookBig Data 2.0 Processing Systems10.1007/978-3-030-44187-6_7(127-133)Online publication date: 10-Jul-2020
https://doi.org/10.1007/978-3-030-44187-6_7
Sakr SSakr S(2020)Large-Scale Machine/Deep Learning FrameworksBig Data 2.0 Processing Systems10.1007/978-3-030-44187-6_6(117-126)Online publication date: 10-Jul-2020
https://doi.org/10.1007/978-3-030-44187-6_6
Sakr SSakr S(2020)Large-Scale Stream Processing SystemsBig Data 2.0 Processing Systems10.1007/978-3-030-44187-6_5(95-115)Online publication date: 10-Jul-2020
https://doi.org/10.1007/978-3-030-44187-6_5
Sakr SSakr S(2020)Large-Scale Graph Processing SystemsBig Data 2.0 Processing Systems10.1007/978-3-030-44187-6_4(59-93)Online publication date: 10-Jul-2020
https://doi.org/10.1007/978-3-030-44187-6_4
Sakr SSakr S(2020)Large-Scale Processing Systems of Structured DataBig Data 2.0 Processing Systems10.1007/978-3-030-44187-6_3(45-58)Online publication date: 10-Jul-2020
https://doi.org/10.1007/978-3-030-44187-6_3
Sakr SSakr S(2020)General-Purpose Big Data Processing SystemsBig Data 2.0 Processing Systems10.1007/978-3-030-44187-6_2(17-43)Online publication date: 10-Jul-2020
https://doi.org/10.1007/978-3-030-44187-6_2
Sakr SSakr S(2020)IntroductionBig Data 2.0 Processing Systems10.1007/978-3-030-44187-6_1(1-16)Online publication date: 10-Jul-2020
https://doi.org/10.1007/978-3-030-44187-6_1
Hwang EKim HNam BChoi Y(2018)CAVA: Exploring Memory Locality for Big Data Analytics in Virtualized Clusters2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)10.1109/CCGRID.2018.00017(21-30)Online publication date: May-2018
https://doi.org/10.1109/CCGRID.2018.00017
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Piglet: Interactive and Platform Transparent Analytics for RDF & Dynamic Data

The optimization for recurring queries in big data analysis system with MapReduce

ReStore: reusing results of MapReduce jobs