bEaMR-Serverless is a project focused on deploying and managing Apache Beam applications on AWS EMR Serverless infrastructure. It includes a Terraform management script (tf) and a Python CLI script for job submission to EMR Serverless. The tf script streamlines Terraform operations across multiple environments, while the Python CLI script facilitates the submission of Apache Beam jobs to EMR Serverless.
- Terraform Script (
tf): Manages infrastructure for Apache Beam applications on EMR Serverless using Terraform. - Workspace Management: Utilizes Terraform workspaces for segregating environments like
dev,prod, etc. - Docker Integration: Handles building and pushing Docker images required for the serverless application.
- EMR Serverless Job Submission Script: A Python CLI tool for submitting Apache Beam jobs to AWS EMR Serverless.
- Terraform and hashicorp aws provider >=4.62
- Docker (for image building and pushing)
- Amazon EMR Serverless Image CLI (for validation of images)
- AWS CLI (configured for access to AWS services)
- Python with
boto3(for the EMR Serverless job submission script) - Knowledge of Apache Beam, AWS EMR, and serverless concepts
Execute Terraform commands within the project's infrastructure context:
./tf [terraform_command] [options]Examples:
./tf plan
./tf apply
./tf destroyIf you run into permissions issues, don't forget to supply credentials. There are a lot of different mechanisms available here, so review the docs as needed. Here's an example of using a pre-configured profile named "your-aws-profile":
AWS_PROFILE=your-aws-profile ./tf applyManage different deployment environments using workspaces. Automatically selects the appropriate variable file for the active workspace:
./tf workspace new [workspace_name]
./tf workspace select [workspace_name]Build and push Docker images as part of the infrastructure setup:
./tf update_imageSubmit Apache Beam jobs to EMR Serverless using the Python CLI script:
python emr_job_cli.py \
--application-id "app-id" \
--execution-role-arn "arn:aws:iam::123456789012:role/MyRole" \
--entry-point "s3://path/to/assembly.jar" \
--entry-point-arguments "arg1 arg2 arg3" \
--spark-submit-parameters "--executor-memory 1G --total-executor-cores 2" \
--name "MyBeamJob"Replace the placeholders with actual job details. entryPointArguments should be a space-separated list of arguments.
Terraform workspaces are extensively used to manage and isolate configurations for different environments in bEaMR-Serverless.
- Avoid Default Workspace: The project contains custom logic to prevent the use of Terraform's default workspace.
- Workspace-Specific Configuration: Each workspace requires a
terraform.[workspace].tfvarsfile for environment-specific configurations. - Credentials and Secrets: Handle AWS credentials and sensitive data securely, especially when using the job submission script.