More Web Proxy on the site http://driver.im/

research-article

Reuse-based Optimization for Pig Latin

Authors:

Jesús Camacho-Rodríguez,

Melanie Herschel,

Ioana Manolescu,

Soudip Roy ChowdhuryAuthors Info & Claims

CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management

Pages 2215 - 2220

https://doi.org/10.1145/2983323.2983669

Published: 24 October 2016 Publication History

Abstract

Pig Latin is a popular language which is widely used for parallel processing of massive data sets. Currently, subexpressions occurring repeatedly in Pig Latin scripts are executed as many times as they appear, and the current Pig Latin optimizer does not identify reuse opportunities. We present a novel optimization approach aiming at identifying and reusing repeated subexpressions in Pig Latin scripts. Our optimization algorithm, named PigReuse, identifies subexpression merging opportunities, selects the best ones to execute based on a cost function, and reuses their results as needed in order to compute exactly the same output as the original scripts. Our experiments demonstrate the effectiveness of our approach.

References

[1]

P. Agrawal, D. Kifer, and C. Olston. Scheduling shared scans of large data files. PVLDB, 2008.

Digital Library

[2]

P. Alvaro, N. Conway, and A. Krioukov. Multi-query optimization for parallel dataflow systems, 2009.

[3]

J. Camacho-Rodríguez, D. Colazzo, M. Herschel, I. Manolescu, and S. R. Chowdhury. PigReuse: A Reuse-based Optimizer for Pig Latin. Technical report, 2016. https://hal.inria.fr/hal-01353891.

[4]

CDH. http://www.cloudera.com/.

[5]

I. Elghandour and A. Aboulnaga. ReStore: reusing results of MapReduce jobs. PVLDB, 2012.

Digital Library

[6]

G. Giannikis, D. Makreshanski, G. Alonso, and D. Kossmann. Shared workload optimization. PVLDB, 2014.

Digital Library

[7]

G. Graefe. The Volcano optimizer generator: Extensibility and efficient search. In ICDE, 1993.

Digital Library

[8]

S. Grumbach and T. Milo. Towards Tractable Algebras for Bags. In PODS, 1993.

Digital Library

[9]

Apache Hadoop. http://hadoop.apache.org/.

[10]

HDP. http://www.hortonworks.com/.

[11]

M. Jarke. Common subexpression isolation in multiple query optimization. In Query Processing in Database Systems. Springer, 1985.

[12]

K. Karanasos, A. Katsifodimos, and I. Manolescu. Delta: Scalable data dissemination under capacity constraints. PVLDB, 2013.

Digital Library

[13]

J. LeFevre, J. Sankaranarayanan, H. Hacigumus, J. Tatemura, N. Polyzotis, and M. J. Carey. Opportunistic Physical Design for Big Data Analytics. In SIGMOD, 2014.

Digital Library

[14]

F. Nagel, P. Boncz, and S. D. Viglas. Recycling in pipelined query evaluation. In ICDE, 2013.

Digital Library

[15]

T. Neumann and G. Moerkotte. Generating optimal dag-structured query evaluation plans. Computer Science-Research and Development, 24(3):103--117, 2009.

[16]

T. Nykiel, M. Potamias, C. Mishra, G. Kollios, and N. Koudas. MRShare: sharing across multiple queries in MapReduce. PVLDB, 2010.

Digital Library

[17]

C. Olston, B. Reed, A. Silberstein, and U. Srivastava. Automatic optimization of parallel dataflow programs. In USENIX, 2008.

Digital Library

[18]

C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: a not-so-foreign language for data processing. In SIGMOD, 2008.

Digital Library

[19]

Apache Pig. http://pig.apache.org/.

[20]

http://cwiki.apache.org/confluence/display/PIG/PigMix.

[21]

P. Roy, S. Seshadri, S. Sudarshan, and S. Bhobe. Efficient and Extensible Algorithms for Multi Query Optimization. In SIGMOD, 2000.

Digital Library

[22]

T. K. Sellis. Multiple-query optimization. ACM Trans. Database Syst., 13(1):23--52, Mar. 1988.

Digital Library

[23]

Y. N. Silva, P.-A. Larson, and J. Zhou. Exploiting Common Subexpressions for Cloud Query Processing. In ICDE, 2012.

Digital Library

[24]

G. Wang and C.-Y. Chan. Multi-Query Optimization in MapReduce Framework. PVLDB, 2013.

Digital Library

[25]

J. Yang. Algorithms for materialized view design in data warehousing environment. PVLDB, 1997.

Digital Library

[26]

J. Zhou, P.-A. Larson, J.-C. Freytag, and W. Lehner. Efficient exploitation of similar subexpressions for query processing. In SIGMOD, 2007.

Digital Library

Cited By

Liu RPark KPsallidas FZhu XMo JSen RInterlandi MKaranasos KTian YCamacho-Rodríguez J(2023)Optimizing Data Pipelines for Machine Learning in Feature StoresProceedings of the VLDB Endowment10.14778/3625054.362506016:13(4230-4239)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.14778/3625054.3625060
Shin AJeong JKim DJung SChun B(2022)HippoProceedings of the VLDB Endowment10.14778/3510397.351040215:5(1038-1052)Online publication date: 18-May-2022
https://dl.acm.org/doi/10.14778/3510397.3510402
Han YLi GYuan HSun J(2022)AutoView: An Autonomous Materialized View Management System with Encoder-ReducerIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3163195(1-1)Online publication date: 2022
https://doi.org/10.1109/TKDE.2022.3163195
Show More Cited By

Index Terms

Reuse-based Optimization for Pig Latin
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Database query processing
        Query optimization
      2. Parallel and distributed DBMSs
        MapReduce-based systems
    2. Query languages
      1. Query languages for non-relational engines
        MapReduce languages

Recommendations

Pig latin: a not-so-foreign language for data processing
SIGMOD '08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data

There is a growing need for ad-hoc analysis of extremely large data sets, especially at internet companies where innovation critically depends on being able to analyze terabytes of data collected every day. Parallel database products, e.g., Teradata, ...
Supporting data aspects in pig latin
AOSD '13: Proceedings of the 12th annual international conference on Aspect-oriented software development

In this paper we apply the aspect-oriented programming (AOP) paradigm to Pig Latin, a dataflow language for cloud computing, used primarily for the analysis of massive data sets. Missing from Pig Latin is support for cross-cutting data concerns. Data, ...
Programming Pig: Dataflow Scripting with Hadoop

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management

October 2016

2566 pages

ISBN:9781450340731

DOI:10.1145/2983323

General Chairs:
Snehasis Mukhopadhyay
Indiana University Purdue University Indianapolis, USA
,
ChengXiang Zhai
University of Illinois at Urbana-Champaign, USA
,
Program Chairs:
Elisa Bertino
Purdue University
,
Fabio Crestani
University of Lugano
,
Javed Mostafa
University of North Carolina
,
Jie Tang
Tsinghua University
,
Luo Si
Alibaba Group Inc & Purdue University
,
Xiaofang Zhou
University of Queensland
,
Yi Chang
Yahoo Research
,
Yunyao Li
IBM Research - Almaden
,
Parikshit Sondhi
WalmartLabs

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM'16

Sponsor:

CIKM'16: ACM Conference on Information and Knowledge Management

October 24 - 28, 2016

Indiana, Indianapolis, USA

Acceptance Rates

CIKM '16 Paper Acceptance Rate 160 of 701 submissions, 23%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
170
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu RPark KPsallidas FZhu XMo JSen RInterlandi MKaranasos KTian YCamacho-Rodríguez J(2023)Optimizing Data Pipelines for Machine Learning in Feature StoresProceedings of the VLDB Endowment10.14778/3625054.362506016:13(4230-4239)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.14778/3625054.3625060
Shin AJeong JKim DJung SChun B(2022)HippoProceedings of the VLDB Endowment10.14778/3510397.351040215:5(1038-1052)Online publication date: 18-May-2022
https://dl.acm.org/doi/10.14778/3510397.3510402
Han YLi GYuan HSun J(2022)AutoView: An Autonomous Materialized View Management System with Encoder-ReducerIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3163195(1-1)Online publication date: 2022
https://doi.org/10.1109/TKDE.2022.3163195
Sahal RKhafagy MOmara F(2021)Exploiting Reused-Based Sharing Work Opportunities in Big Data Multiquery Optimization with FlinkBig Data10.1089/big.2020.01419:6(454-479)Online publication date: 1-Dec-2021
https://doi.org/10.1089/big.2020.0141
Yuan HLi GFeng LSun JHan Y(2020)Automatic View Generation with Deep Learning and Reinforcement Learning2020 IEEE 36th International Conference on Data Engineering (ICDE)10.1109/ICDE48307.2020.00133(1501-1512)Online publication date: Apr-2020
https://doi.org/10.1109/ICDE48307.2020.00133
Al-Sayeh HHagedorn SSattler K(2020)A gray-box modeling methodology for runtime prediction of Apache Spark jobsDistributed and Parallel Databases10.1007/s10619-020-07286-yOnline publication date: 10-Mar-2020
https://doi.org/10.1007/s10619-020-07286-y
Jindal APatel HRoy AQiao SYin ZSen RKrishnan S(2019)PeregrineProceedings of the ACM Symposium on Cloud Computing10.1145/3357223.3362726(416-427)Online publication date: 20-Nov-2019
https://dl.acm.org/doi/10.1145/3357223.3362726
Ramjit LInterlandi MWu ENetravali R(2019)AcornProceedings of the ACM Symposium on Cloud Computing10.1145/3357223.3362702(206-219)Online publication date: 20-Nov-2019
https://dl.acm.org/doi/10.1145/3357223.3362702
Camacho-Rodríguez JChauhan AGates AKoifman EO'Malley OGarg VHaindrich ZShelukhin SJayachandran PSeth SJaiswal DBouguerra SBangarwa NHariappan SAgarwal ADere JDai DNair TDembla NVijayaraghavan GHagleitner GBoncz PManegold SAilamaki ADeshpande AKraska T(2019)Apache HiveProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3314045(1773-1786)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3314045
Jindal AKaranasos KRao SPatel H(2018)Selecting subexpressions to materialize at datacenter scaleProceedings of the VLDB Endowment10.14778/3192965.319297111:7(800-812)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.14778/3192965.3192971
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten