Hi @alebastr;

So these changes are required starting from f40+ right? We need to update our SOP for future references if we are also supposed to include this for future releases! I'll bring this to our releng meeting on Tuesday.

Thanks

Hello folks,

We discussed this in our releng weekly call, before proceeding forward to make it happen, we need to loop in QE & Websites folks because this will have an impact there; And, we need to schedule this as per a particular schedule to when this is supposed to be implemented; That's why I'm inviting folks for suggestions, etc, if any!

CC: @adamwill @kevin @humaton

From the websites point of view, we use the path provided in the metadata/images.json from the compose to locate any artifact on dl.fp-o.
Ideally, if this path is updated to reflect the new location (i.e.: Atomic/aarch64/iso/Fedora-Kinoite-ostree-aarch64-39-1.5.iso or something like that), then it's all good for us.
If this path is not updated, then that would require some special handling on our side for those artifacts and Fedora version.

The path in the metadata will definitely always be accurate. However, the other metadata you use to identify the images you care about will change a bit with this.

Pungi has concepts called "variant" and "subvariant". Variants have to be explicitly 'declared', in the variants-fedora.xml file.

In a repo tree, e.g. at https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20240219.n.0/compose/ , the subdirectories (except metadata) are the variants.

In the images.json metadata, e.g. this one, the keys in the images dict are the variants.

Subvariants aren't explicitly declared anywhere. They are properties only of individual deliverables; each lowest-level dict that represents a single 'image' in the images.json file has a subvariant property. For most things pungi can produce, you can explicitly set a subvariant for an image in the appropriate place in the Pungi config - e.g. here we set the 'subvariant' of the KDE live image to "KDE". If no explicit subvariant is set, then Pungi will use the image's variant as its subvariant.

So the way we have this set up at present is basically that we have a bunch of single-purpose variants. We have a Silverblue variant that only produces images with "Silverblue" as their subvariant, a Kinoite variant that only produces images with "Kinoite" as their subvariant, and so on. So because we have Kinoite and Silverblue variants, we have Kinoite and Silverblue trees in the repos, and in the metadata, we have Kinoite and Silverblue keys in the images dict.

To change this, what we do is get rid of these single-purpose variants and create a single variant called "AtomicDesktops" (or something like that). We'd then have all these images come under that variant, and give each one the appropriate subvariant. So instead of having Kinoite images be part of the "Kinoite" variant with "Kinoite" as their subvariant, they'd be part of the "AtomicDesktops" variant with "Kinoite" as their subvariant. In the repos, we'd have an AtomicDesktops directory with the images under it, and in the metadata, there'd be an "AtomicDesktops" key in the images dict and all the images would be under that, with their appropriate subvariants.

You can see how this would look by looking at the Labs or Spins variants, where we do exactly this (create many different images under the same variant, with different subvariants).

If a tool cares about the variants when it parses the images.json metadata, then it might need to be changed to handle this properly. If it doesn't, it probably won't care. fedfind, for instance (the tool I maintain which is used by the openQA scheduler, among other things, to find and identify images) ignores variants. It simply flattens the images dict and only cares about the lowest-level dicts within it that represent the individual images. It uses the combination of subvariant, type, format and arch to "identify" images uniquely: for instance, so far as fedfind is concerned, any image with subvariant "Kinoite", type "dvd-ostree", format "iso" and arch "x86_64" 'is' the Kinoite ostree installer image for x86_64. It doesn't care about the variant at all.

This concept is actually canonicalized in productmd via the UNIQUE_IMAGE_ATTRIBUTES constant and the identify_image function (with a few additional attributes that aren't really relevant to Fedora), and composes will actually fail if they contain multiple images with the same 'unique' identity under this system.

However, I started drafting the changes to do this in practice, and I think there may be a bit of a roadblock. I'll split that into a separate comment as this is getting long.

So the problem boils down to: I don't think pungi currently allows multiple ostree builds per variant. It's just not set up to do it. You can't have a single variant which produces multiple ostrees with different subvariants. The design is strictly one ostree per variant. This is fairly evident all the way through the actual implementation of this phase in Pungi. The word 'subvariant' does not appear at all. The logic in OSTreeThread.worker() is just "we have a variant and an arch and a config, build a single ostree for that variant and arch from that config".

As a natural consequence of that, the ostree_installer phase similarly has no subvariant handling. As there's only one ostree per variant per arch, there can only be one ostree installer image per variant per arch.

The config formats for the ostree and ostree_installer phases don't accommodate multiple image definitions per variant per arch, either.

Obviously this causes a problem for this idea, because we need one ostree per desktop. I don't think we can actually achieve the goal here without a significant Pungi change to allow subvariants for the ostree and ostree_installer phases.

CC: @lsedlar

I'll file a Pungi issue, I guess, makes sense if that's what we need. edit: filed https://pagure.io/pungi/issue/1727

So, @lsedlar was very responsive on this - thanks! - and in the PR where he's looking at changing this, we wound up running into something that feels like mostly a question for the Atomic Desktop folks, so @alebastr , can you take it back to that group? See https://pagure.io/pungi/pull-request/1729#comment-198721 . The question is basically: do any of the Atomic Desktops care about shipping all the stuff under the os/ directory in their variant, e.g. https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-40/compose/Silverblue/x86_64/os/ ?

The problem is that all that stuff is generically named, we don't have a way to publish multiple os/ directories per subvariant within a single variant. If we just had a single AtomicDesktop variant, it would get the os/ stuff from one subvariant at random (whichever finished last). Or we could make it just not have the stuff at all.

My understanding is that all of that stuff is there to support different ways of reaching the installer besides just "boot from the ISO image", and the question is whether you care about supporting those paths. @kevin may be able to correct me if I'm wrong or incomplete there, though.

Thanks for the detailed explanations. I don't think we need those files that much and we certainly don't need a full copy for each variant as they are likely all the same.

If we can get them from another artifact build then we should probably completely remove them.

Guessing by the size, https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-40/compose/Silverblue/x86_64/os/images/install.img is variant-specific.
boot.iso also looks variant-specific, but... is that just a copy of the installer iso?

The installer iso in iso/ subdirectory is identical to the boot.iso (actually they are hardlinked).

We discussed this issue at the Release Engineering weekly meeting today and concluded that we will not aim primarily at the 40 release. As mentioned in previous comments this is a complex change that has to be done in Rawhide first. If we can get everything working correctly during the freeze we will make FBR to get the changes into 40.

we're past F40 now, so this could perhaps be looked at again.

I've also since discovered that we already build several atomic desktop ostrees under the Everything variant in the updates composes - https://pagure.io/fedora-infra/ansible/blob/651ba91ae0b6f379898b0bba4f0109e6b50c59cf/f/roles/bodhi2/backend/templates/pungi.rpm.conf.j2#_143 . So this does apparently work at least for building the ostrees themselves. We're currently trying to extend that to also build ostree container images, so we'll find out with that if building multiple of those under a single variant causes problems, I guess. We don't intentionally build any installer images or os/ content in those composes, though.

we're past F40 now, so this could perhaps be looked at again.

I've also since discovered that we already build several atomic desktop ostrees under the Everything variant in the updates composes - https://pagure.io/fedora-infra/ansible/blob/651ba91ae0b6f379898b0bba4f0109e6b50c59cf/f/roles/bodhi2/backend/templates/pungi.rpm.conf.j2#_143 . So this does apparently work at least for building the ostrees themselves. We're currently trying to extend that to also build ostree container images, so we'll find out with that if building multiple of those under a single variant causes problems, I guess. We don't intentionally build any installer images or os/ content in those composes, though.

so, I wound up coming back around to this via IoT bootc containers, and @siosm reminded me about all this stuff. I sent https://pagure.io/pungi/pull-request/1789 which should make it possible to build multiple ostree_containers in a variant. There is a PR for ostree_installer - https://pagure.io/pungi/pull-request/1729 - but it's been sitting around I think because the question of what to do about the os/ stuff never got settled. We should probably make a decision about that.