Open
Description
Deploy the Airflow workflow from #16 on the cluster infrastructure (#1 and #50) to prove that an example pipeline can run on a deployment outside of developer sandboxes.
Definition of Done
- An interactive deployment of Airflow is accessible by workflow developers with services running on pods of the radps-k3s cluster and used to run example workflows
Artifacts
- Deployment configuration code (e.g., airflow.cfg in source control)
- Logs and outputs from remote workflow execution
- Notes on the design and implementation
- readme
- wiki
Success Criteria
- An example pipeline decomposition (RADPS/airflow_workflow/example_workflow.py) is successfully executed on shared infrastructure
- The Airflow server has the capability of concurrently running workflows submitted from multiple different requests
- The Airflow dashboard and database services are resilient to lifecycle changes in the underlying pods
Potential Challenges
Documentation, choosing a deployment method, creating containers, resource definitions, network rules, ...