Update the search team's DAGs to use miniforge instead of condaforge
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	BTullis
	Nov 11 2024, 11:23 AM

Description

As per the parent ticket, there are two DAGs running on the search team's airflow instance which use WMF Data Workflow Utils to build a runtime conda environment, based on miniconda.

Due to upcoming licence changes in the Anaconda project, we wish to ensure that all of these environments are using the latest version of the workflow utils conda pipeline, in which we switch from miniconda to miniforge.

Please would you take steps to upgrade these DAGs and deploy them, when convenient?

It should just be a case of ensuring the gitlab-ci.yml file references v0.19.0 of repos/data-engineering/workflow_utils as stated here:
https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide/Python_Job_Repos#GitLab_CI_setup

There is a reference GitLab MR here, which you may find useful:
https://gitlab.wikimedia.org/repos/data-engineering/example-job-project/-/merge_requests/36/

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T372417 Switch from miniconda to miniforge
		Resolved		brouberol	T379544 Update the search team's DAGs to use miniforge instead of condaforge

Event Timeline

BTullis created this task.Nov 11 2024, 11:23 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 11 2024, 11:23 AM

BTullis added a parent task: T372417: Switch from miniconda to miniforge.Nov 11 2024, 11:23 AM

bking added a project: Data-Platform-SRE (2024.11.09 - 2024.11.29).Mon, Nov 18, 4:14 PM

EBernhardson moved this task from needs triage to Current work on the Discovery-Search board.Mon, Nov 18, 4:14 PM

EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search.

Adding to Data Platform SRE sprint per today's Search Platform triage meeting, with the expectation that SREs will do this work and CC Search Platform SWEs on any change requests.

EBernhardson set the point value for this task to 2.Mon, Nov 18, 4:40 PM

EBernhardson moved this task from Incoming to DPE-SRE on the Discovery-Search (Current work) board.

brouberol opened https://gitlab.wikimedia.org/repos/search-platform/mjolnir/-/merge_requests/10

Update workflow-utils reference to use miniforge instead of condaforge

brouberol claimed this task.Tue, Nov 19, 7:36 AM

brouberol triaged this task as Medium priority.

brouberol moved this task from DPE-SRE to Needs review on the Discovery-Search (Current work) board.

brouberol moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.

brouberol moved this task from Needs Review to To Be Deployed on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Tue, Nov 19, 11:43 AM

brouberol moved this task from To Be Deployed to Needs Review on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.

ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/mjolnir/-/merge_requests/10

Update workflow-utils reference to use miniforge instead of condaforge