ストリーム処理を支えるキューイングシステムの選び方
•
116 likes•40,932 views
This document discusses messaging queues and platforms. It begins with an introduction to messaging queues and their core components. It then provides a table comparing 8 popular open source messaging platforms: Apache Kafka, ActiveMQ, RabbitMQ, NATS, NSQ, Redis, ZeroMQ, and Nanomsg. The document discusses using Apache Kafka for streaming and integration with Google Pub/Sub, Dataflow, and BigQuery. It also covers benchmark testing of these platforms, comparing throughput and latency. Finally, it emphasizes that messaging queues can help applications by allowing producers and consumers to communicate asynchronously.
![1000
• [ 1] → [ 1 → 1]
• AWS: EC2 /t2.medium x 2 ( VPC)
• github:laclefyoshi/queues-benchmark
• Docker
• Python
36](https://image.slidesharecdn.com/20161025realtimequeue-161026055655/75/-36-2048.jpg)
ストリーム処理を支えるキューイングシステムの選び方
- 1. / @laclefyoshi / ysaeki@r.recruit.co.jp
- 3. • 2011/04 • 2015/09 • • Druid (KDP, 2015) • RDB NoSQL ( , 2016; : HBase ) • ESP8266 Wi-Fi IoT (KDP, 2016) • • (DEIM 2014) • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • Kinesis Family (R-JAWS Bigdata; 2016) 3
- 5. 5
- 7. • Queue Queueing system • • Message • 1 • Broker • • Producer • • Consumer • • Protocol • 7
- 8. • • • 8
- 10. Apache Kafka A distributed streaming platform 10
- 11. Apache ActiveMQ The most popular and powerful open source messaging and Integration Patterns server 11
- 12. RabbitMQ Robust messaging for applications Easy to use 12
- 13. NATS For cloud native applications, IoT messaging, and microservices architectures 13
- 14. NSQ A realtime distributed messaging platform 14
- 15. Redis In-memory data structure store, used as database, cache and message broker 15
- 16. ZeroMQ An embeddable networking library but acts like a concurrency framework 16
- 17. Nanomsg A socket library that provides several common communication patterns 17
- 19. 19
- 20. 20
- 21. : • • • Lambda Architecture : • • 21
- 22. 22
- 23. : • • : • 23
- 24. 24
- 25. : • • : • 25
- 27. 27
- 29. 29
- 30. 30 : • : •
- 31. 31
- 32. 32 : • : •
- 33. 2016/10/25 official / 3rd party 33
- 34. • JIRA Github • Apache Flink Apache Spark Apache Bahir (3rd party) • Apache Kafka 34
- 36. 1000 • [ 1] → [ 1 → 1] • AWS: EC2 /t2.medium x 2 ( VPC) • github:laclefyoshi/queues-benchmark • Docker • Python 36
- 37. 1000 37 • • • • 1000 Byte 100000 Byte 1000 Byte • 1000
- 38. 38
- 40. 40
- 41. 41
- 42. Queue makes everyone happy!!