kafka-projects

Multi-module project of kafka consumers and producers using Spring boot

Instructions

Start a Kafka instance including one Zookeeper and one broker

head to the Kafka bin directory and execute the following commands:

./kafka-zookeeper-server-start.sh ../config/zookeeper.properties

./kafka-server-start.sh ../config/server.properties

Producing and consuming simple text data : kafka-consumer and kafka-producer

Create a topic named first_topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic first_topic --partitions 1 --replication-factor 1

List all topics

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --list

Describe the topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --topic first_topic --describe

Create a topic named second_topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic second_topic --partitions 1 --replication-factor 1

Producing and consuming simple text data using a key: kafka-key-consumer and kafka-key-producer

Create a topic named third_topic having 3 partitions

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic third_topic --partitions 3 --replication-factor 1

Describe the topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --topic third_topic --describe

Producing and consuming json data : kafka-json-consumer and kafka-json-producer

Create a topic named user_topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic user_topic --partitions 1 --replication-factor 1

Producing and consuming json data with two consumer groups : kafka-product-consumer and kafka-product-producer

Create a topic named product_topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic product_topic --partitions 1 --replication-factor 1

There are two consumer groups for this topic: p-analytics-cg and p-dashboard-cg

Describe the consumer groups using the commands below:

./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group p-analytics-cg --describe ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group p-dashboard-cg --describe

Re-balancing example : kafka-rebalanced-consumer and kafka-rebalanced-producer

Create a topic named rebalanced_topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic rebalanced_topic --partitions 1 --replication-factor 1

Add partition to the topic rebalanced_topic while consumer and producer are running

./kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic rebalanced_topic --partitions 2

Filtering example : kafka-person-consumer and kafka-person-producer

Create a topic named person_topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic person_topic --partitions 1 --replication-factor 1

Error Handler on KafkaListener : kafka-item-order-consumer and kafka-item-order-producer

Create a topic named item_order_topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic item_order_topic --partitions 1 --replication-factor 1

Global error handler: kafka-publication-consumer and kafka-publication-producer

Create a topic named article_topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic article_topic --partitions 1 --replication-factor 1

Create a topic named book_topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic book_topic --partitions 1 --replication-factor 1

Retry Mechanism: kafka-flight-request-consumer and kafka-flight-request-producer

Create a topic named flight_request_topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic flight_request_topic --partitions 1 --replication-factor 1

Dead Letter Topic: kafka-booking-consumer and kafka-booking-producer

Create a topic named booking_topic

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic booking_topic --partitions 1 --replication-factor 1

Create a topic named booking_topic_dlt

head to the Kafka bin directory and execute the following command:

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic booking_topic_dlt --partitions 1 --replication-factor 1

Other useful scripts

Create a topic named my_topic with one single partition

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my_topic --partitions 1 --replication-factor 1

Create a console consumer for a topic named my_topic

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my_topic --offset earliest --partition 0

Reset the offset of consumer group named my_consumer_group for the topic partition 0

./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my_consumer_group --execute --reset-offsets --to-offset 10 --topic my_topic:0

Reset the offset of consumer group named my_consumer_group for all the topic partitions

./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my_consumer_group --execute --reset-offsets --to-offset 10 --topic my_topic

Example application : kafka-order-producer, kafka-reward-consumer, kafka-storage-consumer, kafka-pattern-consumer

kafka-order-producer exposes a Rest API. Swagger is available for the API at:

http://localhost:9001/kafka-orders/swagger-ui/

kafka-order-producer saves order data to h2 database. H2 console can be accessed at:

http://localhost:9001/kafka-orders/h2

Username and password are george, orwell

Streams

Feedback Flash sale Inventory Order Payment Premium Subscription Votes

Name		Name	Last commit message	Last commit date
Latest commit History 77 Commits
kafka-booking-consumer		kafka-booking-consumer
kafka-booking-producer		kafka-booking-producer
kafka-consumer		kafka-consumer
kafka-feedback-detailed-rating-stream		kafka-feedback-detailed-rating-stream
kafka-feedback-one-streams		kafka-feedback-one-streams
kafka-feedback-rating-stream		kafka-feedback-rating-stream
kafka-feedback-streams		kafka-feedback-streams
kafka-feedback-three-streams		kafka-feedback-three-streams
kafka-feedback-two-streams		kafka-feedback-two-streams
kafka-flash-sale-vote-stream		kafka-flash-sale-vote-stream
kafka-flash-sale-vote-window-stream		kafka-flash-sale-vote-window-stream
kafka-flight-request-consumer		kafka-flight-request-consumer
kafka-flight-request-producer		kafka-flight-request-producer
kafka-hopping-window-inventory-stream		kafka-hopping-window-inventory-stream
kafka-inventory-reduce-stream		kafka-inventory-reduce-stream
kafka-inventory-stream		kafka-inventory-stream
kafka-item-order-consumer		kafka-item-order-consumer
kafka-item-order-producer		kafka-item-order-producer
kafka-json-consumer		kafka-json-consumer
kafka-json-producer		kafka-json-producer
kafka-key-consumer		kafka-key-consumer
kafka-key-producer		kafka-key-producer
kafka-order-payment-inner-join-stream		kafka-order-payment-inner-join-stream
kafka-order-payment-left-join-stream		kafka-order-payment-left-join-stream
kafka-order-producer		kafka-order-producer
kafka-orders-model		kafka-orders-model
kafka-pattern-consumer		kafka-pattern-consumer
kafka-person-consumer		kafka-person-consumer
kafka-person-producer		kafka-person-producer
kafka-premium-offer-stream		kafka-premium-offer-stream
kafka-producer		kafka-producer
kafka-product-consumer		kafka-product-consumer
kafka-product-producer		kafka-product-producer
kafka-publication-consumer		kafka-publication-consumer
kafka-publication-producer		kafka-publication-producer
kafka-rebalanced-consumer		kafka-rebalanced-consumer
kafka-rebalanced-producer		kafka-rebalanced-producer
kafka-reward-consumer		kafka-reward-consumer
kafka-storage-consumer		kafka-storage-consumer
kafka-streams-five		kafka-streams-five
kafka-streams-four		kafka-streams-four
kafka-streams-one		kafka-streams-one
kafka-streams-six		kafka-streams-six
kafka-streams-three		kafka-streams-three
kafka-streams-two		kafka-streams-two
kafka-streams		kafka-streams
kafka-subscription-offer-stream		kafka-subscription-offer-stream
kafka-timestamp-extractor-inventory-stream		kafka-timestamp-extractor-inventory-stream
kafka-web-votes-stream		kafka-web-votes-stream
.gitignore		.gitignore
README.md		README.md
Spring Boot Kafka.pdf		Spring Boot Kafka.pdf
pom.xml		pom.xml

MbqIIB/kafka-projects

Folders and files

Latest commit

History

Repository files navigation

kafka-projects

Instructions

Start a Kafka instance including one Zookeeper and one broker

head to the Kafka bin directory and execute the following commands:

Producing and consuming simple text data : kafka-consumer and kafka-producer

Create a topic named first_topic

head to the Kafka bin directory and execute the following command:

List all topics

head to the Kafka bin directory and execute the following command:

Describe the topic

head to the Kafka bin directory and execute the following command:

Create a topic named second_topic

head to the Kafka bin directory and execute the following command:

Producing and consuming simple text data using a key: kafka-key-consumer and kafka-key-producer

Create a topic named third_topic having 3 partitions

head to the Kafka bin directory and execute the following command:

Describe the topic

head to the Kafka bin directory and execute the following command:

Producing and consuming json data : kafka-json-consumer and kafka-json-producer

Create a topic named user_topic

head to the Kafka bin directory and execute the following command:

Producing and consuming json data with two consumer groups : kafka-product-consumer and kafka-product-producer

Create a topic named product_topic

head to the Kafka bin directory and execute the following command:

There are two consumer groups for this topic: p-analytics-cg and p-dashboard-cg

Describe the consumer groups using the commands below:

Re-balancing example : kafka-rebalanced-consumer and kafka-rebalanced-producer

Create a topic named rebalanced_topic

head to the Kafka bin directory and execute the following command:

Add partition to the topic rebalanced_topic while consumer and producer are running

Filtering example : kafka-person-consumer and kafka-person-producer

Create a topic named person_topic

head to the Kafka bin directory and execute the following command:

Error Handler on KafkaListener : kafka-item-order-consumer and kafka-item-order-producer

Create a topic named item_order_topic

head to the Kafka bin directory and execute the following command:

Global error handler: kafka-publication-consumer and kafka-publication-producer

Create a topic named article_topic

head to the Kafka bin directory and execute the following command:

Create a topic named book_topic

head to the Kafka bin directory and execute the following command:

Retry Mechanism: kafka-flight-request-consumer and kafka-flight-request-producer

Create a topic named flight_request_topic

head to the Kafka bin directory and execute the following command:

Dead Letter Topic: kafka-booking-consumer and kafka-booking-producer

Create a topic named booking_topic

head to the Kafka bin directory and execute the following command:

Create a topic named booking_topic_dlt

head to the Kafka bin directory and execute the following command:

Other useful scripts

Create a topic named my_topic with one single partition

Create a console consumer for a topic named my_topic

Reset the offset of consumer group named my_consumer_group for the topic partition 0

Reset the offset of consumer group named my_consumer_group for all the topic partitions

Example application : kafka-order-producer, kafka-reward-consumer, kafka-storage-consumer, kafka-pattern-consumer

kafka-order-producer exposes a Rest API. Swagger is available for the API at:

kafka-order-producer saves order data to h2 database. H2 console can be accessed at:

Username and password are george, orwell

Streams

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages