Presto+MySQLで分散SQL
•
23 likes•6,250 views
This document discusses Presto, an open source distributed SQL query engine for interactive analysis of large datasets. It describes Presto's architecture including its coordinator, connectors, workers and storage plugins. Presto allows querying of multiple data sources simultaneously through its connector plugins for systems like Hive, Cassandra, PostgreSQL and others. Queries are executed in a pipelined fashion without disk I/O or waiting between stages for improved performance.
Presto+MySQLで分散SQL
- 1. Sadayuki Furuhashi Founder & Software Architect Treasure Data, inc. Presto + MySQL 道玄坂LT祭り で分散SQL
- 2. A little about me... > Sadayuki Furuhashi > github/twitter: @frsyuki > Treasure Data, Inc. > Founder & Software Architect > Open-source hacker > MessagePack - Efficient object serializer > Fluentd - An unified data collection tool > ServerEngine - A Ruby framework to build multiprocess servers > Prestogres - PostgreSQL protocol gateway for Presto > LS4 - A distributed object storage with cross-region replication > kumofs - A distributed strong-consistent key-value data store
- 3. Check: www.treasuredata.com Cloud service for the entire data pipeline, including Presto. We’re hiring!
- 6. What’s Presto? A distributed SQL query engine for interactive data analisys against GBs to PBs of data.
- 7. Client Coordinator Connector Plugin Worker Worker Worker Storage / Metadata Discovery Service
- 8. Client Coordinator Connector Plugin Worker Worker Worker Storage / Metadata Discovery Service 1. find servers in a cluster
- 9. Client Coordinator Connector Plugin Worker Worker Worker Storage / Metadata Discovery Service 2. Client sends a query using HTTP
- 10. Client Coordinator Connector Plugin Worker Worker Worker Storage / Metadata Discovery Service 3. Coordinator builds a query plan Connector plugin provides metadata (table schema, etc.)
- 11. Client Coordinator Connector Plugin Worker Worker Worker Storage / Metadata Discovery Service 4. Coordinator sends tasks to workers
- 12. Client Coordinator Connector Plugin Worker Worker Worker Storage / Metadata Discovery Service 5. Workers read data through connector plugin
- 13. Client Coordinator Connector Plugin Worker Worker Worker Storage / Metadata Discovery Service 6. Workers run tasks in memory
- 14. Coordinator Connector Plugin Worker Worker Worker Storage / Metadata Discovery Service 7. Client gets the result from a worker Client
- 15. Client Coordinator Connector Plugin Worker Worker Worker Storage / Metadata Discovery Service
- 16. Client Coordinator Hive Connector Worker Worker Worker HDFS, Hive Metastore Discovery Service find servers in a cluster Hive connector
- 17. Client Coordinator JDBC Connector Worker Worker Worker Cassandra Discovery Service find servers in a cluster Cassandra connector
- 18. Client Coordinator other connectors ... Worker Worker Worker PostgreSQL Discovery Service find servers in a cluster Hive Connector HDFS / Metastore Multiple connectors in a query JDBC Connector Other data sources...
- 19. All stages are pipe-lined ✓ No wait time ✓ No fault-tolerance MapReduce vs. Presto MapReduce Presto map map reduce reduce task task task task task task memory-to-memory data transfer ✓ No disk IO ✓ Data chunk must fit in memory task disk map map reduce reduce disk disk Write data to disk Wait between stages
- 20. Presto meetup!
- 21. Presto JOIN Hive MySQL client select orderkey, orderdate, custkey, email from orders join mysql.presto_test.users on orders.custkey = users.id order by custkey, orderdate;
- 22. Presto JOIN Hive MySQLINSERT INTO client create table mysql.presto_test.recent_user_info as select users.id, users.email, count(1) as count from orders join mysql.presto_test.users on orders.custkey = users.id group by 1, 2;
- 24. Presto JOIN Hive MySQL $ psql Prestogres PostgreSQL protocol gateway for Presto