LocalStack docker setup with Scala 3 and Cats Effect to test full-fledged kinesis stream aggregation
- Local Testing: Scripts for running with LocalStack.
- Error Handling: Graceful error logging, retries on disconnects
- Concurrency: Asynchronous processing
- Scala: 3.x
- SBT: 1.9.x
- Docker: For LocalStack
- AWS CLI: For LocalStack interaction
- Java: 21 (Adoptium recommended)
-
Clone the Repository:
git clone https://github.com/aditya-K93/localstack.git cd localstack -
Install Dependencies:
- Ensure
sbtis installed. - Run
sbt compileto fetch dependencies (listed inbuild.sbt).
- Ensure
-
Set Up LocalStack:
- Ensure Docker is running.
- Use the provided scripts to start LocalStack and create a Kinesis stream:
chmod +x *.sh ./start-localstack.sh
-
Run the Application:
- Execute the app:
sbt "runMain com.github.adityak93.kinesis.Main"
- Execute the app:
-
Populate the Stream:
- Add test data to the Kinesis stream:
./populate-stream.sh
- Add test data to the Kinesis stream:
-
Verify logs:
- Check
output.txtfor aggregated results (e.g., running sums:0, 1, 3, 6, ...).
- Check
-
Full Test:
- Run the end-to-end test script:
./test-local.sh
- Stops LocalStack after execution.
- Run the end-to-end test script:
- Stream Details: Edit
Main.scalato changestreamName,shardId, oroutputFileinKinesisApp.Config. - Logging: Customize
src/main/resources/logback.xmlfor log levels or output patterns.
localstack/
├── src/
│ ├── main/
│ │ ├── scala/
│ │ │ └── com/github/adityak93/kinesis/
│ │ │ └── Main.scala
│ │ └── resources/
│ │ └── logback.xml
│ └── test/
├── build.sbt
├── start-localstack.sh
├── populate-stream.sh
├── run-app.sh
├── test-local.sh
└── README.md
2025-02-20 05:10:12.345 ERROR [main] com.github.adityak93.kinesis.Main - Unable to load region...
- Configured in
logback.xmlwith timestamp, level, thread, logger name, and message.
- Errors (e.g., missing region) are logged with no stack trace.
- Exits with code 1 on failure, configurable in
Main.run.
This project is licensed under the MIT License - see the LICENSE file for details.