research-article

Split query processing in polybase

Authors:

Josep Aguilar-Saborit,

Artin Avanes,

Miro Flasza,

Jim GramlingAuthors Info & Claims

SIGMOD '13: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data

Pages 1255 - 1266

https://doi.org/10.1145/2463676.2463709

Published: 22 June 2013 Publication History

Get Access

Abstract

This paper presents Polybase, a feature of SQL Server PDW V2 that allows users to manage and query data stored in a Hadoop cluster using the standard SQL query language. Unlike other database systems that provide only a relational view over HDFS-resident data through the use of an external table mechanism, Polybase employs a split query processing paradigm in which SQL operators on HDFS-resident data are translated into MapReduce jobs by the PDW query optimizer and then executed on the Hadoop cluster. The paper describes the design and implementation of Polybase along with a thorough performance evaluation that explores the benefits of employing a split query processing paradigm for executing queries that involve both structured data in a relational DBMS and unstructured data in Hadoop. Our results demonstrate that while the use of a split-based query execution paradigm can improve the performance of some queries by as much as 10X, one must employ a cost-based query optimizer that considers a broad set of factors when deciding whether or not it is advantageous to push a SQL operator to Hadoop. These factors include the selectivity factor of the predicate, the relative sizes of the two clusters, and whether or not their nodes are co-located. In addition, differences in the semantics of the Java and SQL languages must be carefully considered in order to avoid altering the expected results of a query.

References

[1]

http://sqoop.apache.org

Google Scholar

[2]

Bajda-Pawlikowski, et. al., "Efficient Processing of Data Warehousing Queries in a Split Execution Environment," Proceedings of the 2011 SIGMOD Conference, June 2011.

Digital Library

Google Scholar

[3]

http://www.oracle.com/technetwork/bdc/hadoop-loader/connectors-hdfs-wp-1674035.pdf

Google Scholar

[4]

http://www.greenplum.com/sites/default/files/EMC_Greenplum_Hadoop_DB_TB_0.pdf

Google Scholar

[5]

http://www.asterdata.com/sqlh/

Google Scholar

[6]

Yu Xu, et. al., Integrating Hadoop and Parallel DBMS, Proceedings of the 2010 SIGMOD Conference, June 2010.

Digital Library

Google Scholar

[7]

Yu Xu, et. al., A Hadoop Based Distributed Loading Approach to Parallel Data Warehouses", Proceedings of the 2011 SIGMOD Conference, June, 2011.

Digital Library

Google Scholar

[8]

http://developer.teradata.com/extensibility/articles/hadoop-dfs-to-teradata

Google Scholar

[9]

IBM InfoSphere BigInsights Information Center, pic.dhe.ibm.com/infocenter/bigins/v1r4/index.jsp

Google Scholar

[10]

http://www.vertica.com/2012/07/05/teaching-the-elephant-new-tricks/,

Google Scholar

[11]

Shankar, S., et. al., Query Optimization in Microsoft SQL Server PDW, Proceedings of the 2012 SIGMOD Conference, May 2012.

Digital Library

Google Scholar

[12]

Herodotos Herodotou, Shivnath Babu: Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs. PVLDB 4(11):1111--1122 (2011).

Google Scholar

[13]

Sai Wu, et. al., Query Optimization for Massively Parallel Data Processing, Proceedings of the 2011 SoCC Conference, Oct. 2011.

Digital Library

Google Scholar

Cited By

View all

Sun JMa HZhang LLiu HShi HLuo SWu KBruhwiler KZhu CNie YChen JZhang LLiang Y(2023)Accelerating Cloud-Native Databases with Distributed PMem Stores2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00233(3043-3057)Online publication date: Apr-2023
https://doi.org/10.1109/ICDE55515.2023.00233
Guo QZhang CZhang SLu J(2023)Multi-model query languages: taming the variety of big dataDistributed and Parallel Databases10.1007/s10619-023-07433-142:1(31-71)Online publication date: 31-May-2023
https://doi.org/10.1007/s10619-023-07433-1
Saravanan VHussain FKshirasagar N(2022)Role of Big Data in Internet of Things NetworksResearch Anthology on Big Data Analytics, Architectures, and Applications10.4018/978-1-6684-3662-2.ch016(336-363)Online publication date: 2022
https://doi.org/10.4018/978-1-6684-3662-2.ch016
Show More Cited By

Index Terms

Split query processing in polybase
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Database query processing
      2. Parallel and distributed DBMSs
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Database theory
      1. Database query processing and optimization (theory)

Recommendations

Materialized view selection using evolutionary algorithm for speeding up big data query processing

For speeding up query processing on Big Data, frequent sub-queries or views may be materialized such that the query processing cost is minimized with optimum cost of maintaining the materialized views and/or queries. Materializing frequent sub-queries ...
Efficient processing of data warehousing queries in a split execution environment
SIGMOD '11: Proceedings of the 2011 ACM SIGMOD International Conference on Management of data

Hadapt is a start-up company currently commercializing the Yale University research project called HadoopDB. The company focuses on building a platform for Big Data analytics in the cloud by introducing a storage layer optimized for structured data and ...
Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet
BDCAT '17: Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies

Big Data has been seen as a remedy for the efficient management of the ever-increasing genomic data. In this paper, we investigate the use of Apache Spark to store and process Variant Calling Files (VCF) on a Hadoop cluster. We demonstrate Tomatula, a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SIGMOD '13: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data

June 2013

1322 pages

ISBN:9781450320375

DOI:10.1145/2463676

General Chairs:
Kenneth Ross
Columbia University
,
Divesh Srivastava
AT&T Research
,
Program Chair:
Dimitris Papadias
HKUST

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 June 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS'13

Sponsor:

SIGMOD

SIGMOD/PODS'13: International Conference on Management of Data

June 22 - 27, 2013

New York, New York, USA

Acceptance Rates

SIGMOD '13 Paper Acceptance Rate 76 of 372 submissions, 20%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

150
Total Citations
View Citations
875
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)3

Reflects downloads up to 14 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Sun JMa HZhang LLiu HShi HLuo SWu KBruhwiler KZhu CNie YChen JZhang LLiang Y(2023)Accelerating Cloud-Native Databases with Distributed PMem Stores2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00233(3043-3057)Online publication date: Apr-2023
https://doi.org/10.1109/ICDE55515.2023.00233
Guo QZhang CZhang SLu J(2023)Multi-model query languages: taming the variety of big dataDistributed and Parallel Databases10.1007/s10619-023-07433-142:1(31-71)Online publication date: 31-May-2023
https://doi.org/10.1007/s10619-023-07433-1
Saravanan VHussain FKshirasagar N(2022)Role of Big Data in Internet of Things NetworksResearch Anthology on Big Data Analytics, Architectures, and Applications10.4018/978-1-6684-3662-2.ch016(336-363)Online publication date: 2022
https://doi.org/10.4018/978-1-6684-3662-2.ch016
Kiehn FSchmidt MGlake DPanse FWingerath WWollmer BPoppinga MRitter N(2022)Polyglot data managementProceedings of the VLDB Endowment10.14778/3554821.355489115:12(3750-3753)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.14778/3554821.3554891
Kranas PKolev BLevchenko OPacitti EValduriez PJiménez-Peris RPatiño-Martinez M(2021)Parallel query processing in a polystoreDistributed and Parallel Databases10.1007/s10619-021-07322-5Online publication date: 3-Feb-2021
https://doi.org/10.1007/s10619-021-07322-5
Fang ZChen LGao YPan LJensen C(2021)Dragoon: a hybrid and efficient big trajectory management system for offline and online analyticsThe VLDB Journal10.1007/s00778-021-00652-xOnline publication date: 3-Feb-2021
https://doi.org/10.1007/s00778-021-00652-x
Glake DKiehn FSchmidt MRitter N(2021)Towards Taming the Adaptivity ProblemService-Oriented Computing10.1007/978-3-030-87568-8_5(83-99)Online publication date: 26-Sep-2021
https://doi.org/10.1007/978-3-030-87568-8_5
Potharaju RKim TWu WAcharya VSuh SFogarty ADave ARamanujam STalius TNovik LRamakrishnan R(2020)HeliosProceedings of the VLDB Endowment10.14778/3415478.341554713:12(3231-3244)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.14778/3415478.3415547
Aissa MSfaxi LRobbana R(2020)Decisional architectures from business intelligence to big dataProceedings of the 2nd International Conference on Digital Tools & Uses Congress10.1145/3423603.3424049(1-9)Online publication date: 15-Oct-2020
https://dl.acm.org/doi/10.1145/3423603.3424049
Liu ZWu HBai TWang YXu C(2020)Towards Elastic Data Warehousing by Decoupling Data Management and ComputationProceedings of the 2020 4th International Conference on Cloud and Big Data Computing10.1145/3416921.3416935(52-57)Online publication date: 26-Aug-2020
https://dl.acm.org/doi/10.1145/3416921.3416935
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Materialized view selection using evolutionary algorithm for speeding up big data query processing

Efficient processing of data warehousing queries in a split execution environment

Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet