Move from OSTree to OCI for updates of new systems #1823

jlebon · 2024-10-30T17:44:11Z

Currently, FCOS pushes updates via an OSTree repo. To better align with the bootable containers initiative, let's move to updating FCOS via a container image. Container images are already being published in https://quay.io/repository/fedora/fedora-coreos.

Note this ticket is separate from #1263, which covers also fleshing out the story of layering. In this ticket, we're strictly scoping the effort to changing the transport used.

Take the digest pullspec for the base OS bootable container and put it in the new `oci-image` field in the release metadata. Part of coreos/fedora-coreos-tracker#1823.

When updating the release index, gather the OCI pullspecs across all arches for a given release into a single list and inject it into the new `oci-images` key, the same way we do for OSTree commits and `commits`. Part of coreos/fedora-coreos-tracker#1823.

Currently, we're starting a total of N x M scrapers, where N is the number of streams, and M the number of arches. So right now, this would be 3 x 4 = 12 scrapers. This is very wasteful because each scraper individually downloads the release index and update metadata every 30 seconds, even though that metadata is not different per architecture. I think the reason it was set up this way is in case we wanted to host separate e.g. release index or update files _per_ architecture in S3 instead of all togother. This can be seen by the fact the code supports templating those URLs with `basearch`. However, it's unlikely we'll be changing that design decision, so let's just do the saner thing and rework the scraping to be stream-based. This is done by changing the scraper to host not one single `Graph` object, but instead a `HashMap<String, Graph>` which maps architectures to graphs. Then, when a request for a graph comes in, we lookup in our cache keying off of the requested architecture. This is prep for adding another dimension to the matrix, which is whether the OCI version of the graph was reported. If we didn't do this cleanup first, it would have blown up the number of scrapers to 24. Part of coreos/fedora-coreos-tracker#1823.

When parsing the release index, check for the new `oci-images` key. If present, also build up a separate graph with only nodes containing OCI information. In that case, the node payload is the pullspec and the scheme declared in the node metadata is `oci`. When a client requests a graph, check if the `oci=` URL parameter was set. If so, return back the OCI graph instead of the OSTree one. Part of coreos/fedora-coreos-tracker#1823.

Add support in both release metadata and the release index for specifying OCI image pullspecs for a given release. In both cases, the new fields are located at the same level as the existing `commits` key holding OSTree checksums. The key is called `oci-image` in release metadata and `oci-images` in the release index. Part of coreos/fedora-coreos-tracker#1823.

jlebon · 2024-11-01T20:04:13Z

OK, some progress on this:

release: support OCI image pullspecs stream-metadata-go#75 adds support for a new oci-image key in the release metadata and a new oci-images key in the release index
Inject oci-image in release metadata and oci-images in release index coreos-assembler#3919 populates those newly supported keys from the base-oscontainer key in the build metadata
Add support for querying OCI graph fedora-coreos-cincinnati#99 teaches Cincinnati to read those keys and serve an OCI-based graph when the request includes oci=true
daemon/transaction-types: Support custom origins for digested pullspecs rpm-ostree#5120 teaches rpm-ostree to allow custom origin information when passing a digest pullspec to rpm-ostree rebase

So the next step on this is support in Zincati for a config knob that tells it to request the OCI version of the graph and deploy using rpm-ostree rebase with appropriate custom origin info.

Add support in both release metadata and the release index for specifying OCI image pullspecs for a given release. In both cases, the new fields are located at the same level as the existing `commits` key holding OSTree checksums. The key is called `oci-image` in release metadata and `oci-images` in the release index. Part of coreos/fedora-coreos-tracker#1823.

cgwalters · 2024-11-01T20:40:38Z

Awesome!

Currently, we're starting a total of N x M scrapers, where N is the number of streams, and M the number of arches. So right now, this would be 3 x 4 = 12 scrapers. This is very wasteful because each scraper individually downloads the release index and update metadata every 30 seconds, even though that metadata is not different per architecture. I think the reason it was set up this way is in case we wanted to host separate e.g. release index or update files _per_ architecture in S3 instead of all together. This can be seen by the fact the code supports templating those URLs with `basearch`. However, it's unlikely we'll be changing that design decision, so let's just do the saner thing and rework the scraping to be stream-based. This is done by changing the scraper to host not one single `Graph` object, but instead a `HashMap<String, Graph>` which maps architectures to graphs. Then, when a request for a graph comes in, we lookup in our cache keying off of the requested architecture. This is prep for adding another dimension to the matrix, which is whether the OCI version of the graph was requested. If we didn't do this cleanup first, it would have blown up the number of scrapers to 24. Part of coreos/fedora-coreos-tracker#1823.

When parsing the release index, check for the new `oci-images` key. If present, also build up a separate graph with only nodes containing OCI information. In that case, the node payload is the pullspec and the scheme declared in the node metadata is `oci`. When a client requests a graph, check if the `oci=` URL parameter was set. If so, return back the OCI graph instead of the OSTree one. Part of coreos/fedora-coreos-tracker#1823.

Add support in both release metadata and the release index for specifying OCI image pullspecs for a given release. In both cases, the new fields are located at the same level as the existing `commits` key holding OSTree checksums. The key is called `oci-image` in release metadata and `oci-images` in the release index. Part of coreos/fedora-coreos-tracker#1823.

Take the digest pullspec for the base OS bootable container and put it in the new `oci-image` field in the release metadata. Part of coreos/fedora-coreos-tracker#1823.

When updating the release index, gather the OCI pullspecs across all arches for a given release into a single list and inject it into the new `oci-images` key, the same way we do for OSTree commits and `commits`. Part of coreos/fedora-coreos-tracker#1823.

When parsing the release index, check for the new `oci-images` key. If present, also build up a separate graph with only nodes containing OCI information. In that case, the node payload is the pullspec and the scheme declared in the node metadata is `oci`. When a client requests a graph, check if the `oci=` URL parameter was set. If so, return back the OCI graph instead of the OSTree one. Part of coreos/fedora-coreos-tracker#1823.

jlebon · 2024-11-08T17:47:54Z

So the next step on this is support in Zincati for a config knob that tells it to request the OCI version of the graph

Instead of a config knob, Zincati can just cue off of the origin. If the OS is on OSTree, query the OSTree graph. If it's on OCI, query the OCI graph. So then the barrier release is just about switching the origin over from OSTree to OCI.

Discussed the rollout plan for this with @travier and @dustymabe:

Switch next bootimages to deploy-via-container whenever all the code needed has landed in FCOS.
- This means that only new nodes will be updating via containers to start, giving us a way to bake it before switching updating nodes.
After X next releases (e.g. 2 or 3), do barrier release to switch origin on existing nodes.
Switch the testing bootimages to deploy-via-container. If we're still before Beta freeze, just do it. If we're after Beta freeze, do it at GA.
- Stable follows 2 weeks after.
After X testing releases, do barrier release to switch origin on existing nodes.
- Stable follows 2 weeks after.

Take the digest pullspec for the base OS bootable container and put it in the new `oci-image` field in the release metadata. Part of coreos/fedora-coreos-tracker#1823.

When updating the release index, gather the OCI pullspecs across all arches for a given release into a single list and inject it into the new `oci-images` key, the same way we do for OSTree commits and `commits`. Part of coreos/fedora-coreos-tracker#1823.

This is part of the overall move to OCI based updates that is coming in the Fedora 42 cycle for FCOS. See: coreos/fedora-coreos-tracker#1823 Closes: #3423

This is part of the overall move to OCI based updates that is coming in the Fedora 42 cycle for FCOS. See: coreos/fedora-coreos-tracker#1823 Closes: coreos#3423 (cherry picked from commit 22d6552)

This is part of the overall move to OCI based updates that is coming in the Fedora 42 cycle for FCOS. See: coreos/fedora-coreos-tracker#1823 Closes: #3423 (cherry picked from commit 22d6552)

With the f42 rebase we will switch disk images to use OCI for updates. While this is transparent for most users, it's still a big technical change so it's worth mentionning it, at least so users using can look into it. See https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates coreos/fedora-coreos-tracker#1823

To simplify testing for coreos/fedora-coreos-tracker#1823 This script write a Zincati status override containing a fake rpm-ostree status output to appear like it's on an OCI deployement. Zincati will look under the well-known path of /run/zincati/booted-status-override.json. The content will trigger the OCI code path in Zincati. This will later run as ExecStartPre in the zincati.service environment Just ship the migration script for now, without the zincati service changes, to allow testing. See coreos/fedora-coreos-tracker#1890 See coreos/fedora-coreos-tracker#1823 (comment) Requires zincati 0.0.30 Ref coreos/zincati#1273 Ref coreos/zincati#1241

These changes are ones that we are adopting for F42+ and since F42 GA is a GO for Tuesday 04/15 we'll go ahead and make the change for `testing-devel` now. - coreos/fedora-coreos-tracker#1823 - coreos/fedora-coreos-tracker#1852

These changes are ones that we are adopting for F42+ and since F42 GA is a GO for Tuesday 04/15 we'll go ahead and make the change for `testing-devel` now. We can update image-base.yaml because now the change can apply everywhere. - coreos/fedora-coreos-tracker#1823 - coreos/fedora-coreos-tracker#1852

In [1] we moved newly provisioned systems to be deployed via container and thus retrieving updates via zincati from the OCI registry (quay.io/fedora/fedora-coreos). This starts the migration script shipped in [2] before Zincati to set up the migration at the next update. A way to override this is to add another drop-in with `ExecStartPre=` with a higher ranking name under the systemd drop in path, e.g. `/etc/systemd/system/zincati.service.d/90-no-oci.conf`. [1] coreos/fedora-coreos-tracker#1823 [2] coreos#3355 Closes coreos/fedora-coreos-tracker#1890

These changes are ones that we are adopting for F42+ and since F42 GA is a GO for Tuesday 04/15 we'll go ahead and make the change for `testing-devel` now. We can update image-base.yaml because now the change can apply everywhere. - coreos/fedora-coreos-tracker#1823 - coreos/fedora-coreos-tracker#1852

These changes are ones that we are adopting for F42+ and since F42 GA is a GO for Tuesday 04/15 we'll go ahead and make the change for `testing-devel` now. We can update image-base.yaml because now the change can apply everywhere. - coreos/fedo 8000 ra-coreos-tracker#1823 - coreos/fedora-coreos-tracker#1852

In [1] we moved newly provisioned systems to be deployed via container and thus retrieving updates via zincati from the OCI registry (quay.io/fedora/fedora-coreos). This starts the migration script shipped in [2] before Zincati to set up the migration at the next update. Slightly tweak the migration script to look for a stamp to opt-out of the migration and create one to signal failure. Also add a MOTD that look for those stamps and display an appropriate message. [1] coreos/fedora-coreos-tracker#1823 [2] coreos#3355 Closes coreos/fedora-coreos-tracker#1890

marmijo · 2025-04-24T21:37:03Z

The fix for this went into testing stream release 42.20250410.2.0. Please try out the new release and report issues.

thrix · 2025-04-30T17:44:37Z

We are not able to switch to a new derived image via bootc on Fedora CoreOS 42.20250410.3.1, and we are hitting:

sudo bootc switch --transport containers-storage localhost/tmt/bootc/52eaae98-f626-4ca0-a750-2f8408c5f645
            err: ERROR Switching: Pulling: Creating importer: failed to invoke method OpenImage: failed to invoke method OpenImage: reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/lib/containers/storage,overlay.mountopt=nodev,metacopy=on]localhost/tmt/bootc/52eaae98-f626-4ca0-a750-2f8408c5f645:latest" does not resolve to an image ID

cgwalters · 2025-04-30T18:19:49Z

Can you also add the output of e.g. podman images above there? Could this be some sort of race condition (are there concurrent mutations to the host?)

thrix · 2025-04-30T19:54:13Z

A small update, I am not able to reproduce the issue manually, so we are looking what is wrong in tmt, it might take a while, will report back.

dustymabe · 2025-05-01T14:26:32Z

The fix for this went into stable stream release 42.20250410.3.1.

In [1] we moved newly provisioned systems to be deployed via container and thus retrieving updates via zincati from the OCI registry (quay.io/fedora/fedora-coreos). This starts the migration script shipped in [2] before Zincati to set up the migration at the next update. Slightly tweak the migration script to look for a stamp to opt-out of the migration and create one to signal failure. Also add a MOTD that look for those stamps and display an appropriate message. [1] coreos/fedora-coreos-tracker#1823 [2] coreos#3355 Closes coreos/fedora-coreos-tracker#1890

jlebon added kind/enhancement area/updates component/zincati component/rpm-ostree area/bootable-containers Related to the bootable containers effort. F42 labels Oct 30, 2024

jlebon mentioned this issue Oct 30, 2024

Roadmap to Fedora Bootable Containers #1726

Open

11 tasks

This was referenced Nov 1, 2024

release: support OCI image pullspecs coreos/stream-metadata-go#75

Merged

Inject oci-image in release metadata and oci-images in release index coreos/coreos-assembler#3919

Merged

Add support for querying OCI graph coreos/fedora-coreos-cincinnati#99

Merged

This comment was marked as off-topic.

Sign in to view

jlebon mentioned this issue Nov 7, 2024

Use bootc install to-filesystem to build our bootimages #1827

Open

This comment was marked as off-topic.

Sign in to view

dustymabe mentioned this issue Apr 11, 2025

image.yaml: switch to EROFS; deploying via container coreos/fedora-coreos-config#3451

Closed

jbtrystram mentioned this issue Apr 15, 2025

overlay: migrate existing systems to OCI coreos/fedora-coreos-config#3458

Open

marmijo added status/pending-stable-release Fixed upstream and in testing. Waiting on stable release. and removed status/pending-testing-release Fixed upstream. Waiting on a testing release. labels Apr 24, 2025

dustymabe closed this as completed May 1, 2025

dustymabe removed the status/pending-stable-release Fixed upstream and in testing. Waiting on stable release. label May 1, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move from OSTree to OCI for updates of new systems #1823

Move from OSTree to OCI for updates of new systems #1823

This comment was marked as off-topic.

This comment was marked as off-topic.

Move from OSTree to OCI for updates of new systems #1823

Move from OSTree to OCI for updates of new systems #1823

Comments

This comment was marked as off-topic.

This comment was marked as off-topic.