Note: I don't know if this is the right solution. I expect that it'll fix the upgrade issue, but will probably break installation using sdubby.

/cc @jlinton

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/3bb28e4e6ea74c7980c56b90f7faaa20

• validate-comps : SUCCESS in 3m 54s

The problem here is that it breaks the non-network systemd-boot installs because that group is being used to put packages on the install ISOs. (but not install them, thats for anaconda to pick).

OTOH, maybe just removing the mandatory option is enough to fix it? I'm not at all sure why those packages are listed as mandatory, particularly as a couple of them are being forceably ignored for the live media.

I think maybe the right fix is to take the "type=mandatory" off, although I'm not sure and will have to test if that is sufficient or it needs to be type=optional and the ISO builder tweaked to then include optional pkgs.

I updated the patch to drop type=mandatory. I didn't do any testing except make validate -j.

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/df1d16bc7c9f4825962e6ba9d75c9a11

• validate-comps : SUCCESS in 4m 09s

Yes, the more I mess with it, the more I think the right change is to make it type=optional. Then we can add an option to the image builder to pickup the optional packages, with the expectation that the grub dependencies will also be marked optional, and the live media kickstarts will pick either the grub or systemd dependencies for a particular install. That might also fix the excludes list that exists in the live media base kickstart.

So maybe let's take a step back, and consider if the whole approach can be redesigned. Why can't you make sdubby co-installable with grubby and have the tool that uses it just call a different binary? The approach with conflicting packages is causing huge problems, and we haven't event made it to a stable release yet.

Its not that stubby can't be installed alongside grubby, its that having both systemd-boot and grub installed/enabled on the same system tends to confuse tools which attempt to detect the target bootloader. So, this becomes a break early vs create a bunch of later bugs, and at the moment I think its probably a good idea to clean up some of this anaconda-tools group if for no other reason than its sorta hacky now when it comes to the live image/etc actually installing the entire group with a subset of the group manually excluded rather than treating it as the list of tools anaconda may add.

what happens if you make it type 'default' rather than 'mandatory'?

I'm about 98% sure it doesn't fix the upgrade problem. In my test env doing a system-upgrade reprocesses the comps file and causes the conflict with any state other than "type=optional".

Although I did learn the difference between "update releasever=" and system-upgrade releasever is that system-upgrade reprocesses the comps file while update doesn't, or doesn't sufficiently that the conflict is avoided.

I think i'm going to do some RFC PRs shortly which update the anaconda-tools comment, type=optional all the excluded packages and remove them from the live image exclusions then --with-opotional the rest of the images. Then the kickstarts won't need to be updated as much.

And just to clarify a bit, i've been testing just changing the sdubby package to type=optional, I don't really know what happens with the rest of them having type=mandatory removed.

Its not that stubby can't be installed alongside grubby, its that having both systemd-boot and grub installed/enabled on the same system tends to confuse tools which attempt to detect the target bootloader. So, this becomes a break early vs create a bunch of later bugs, and at the moment I think its probably a good idea to clean up some of this anaconda-tools group if for no other reason than its sorta hacky now when it comes to the live image/etc actually installing the entire group with a subset of the group manually excluded rather than treating it as the list of tools anaconda may add.

I very much disagree with this approach. Detection of the target bootloader would only be done by a few tools, and we certainly can fix them to do the detection properly. Also, when running on a system, the bootloader should be detected based on what is actually installed in /boot, and not based on the packages on the running system. In the current approach, I see the following things as mistakes in packages related to bootloaders:
- using the presence of a package to make assumptions about /boot, a resource shared between different Linux distributions and Windows and other operating systems.
- conflicts between packages, in particular for packages with tools, like grubby/sdubby.
- installation of files to /boot as part of package payload, instead of a %post scriptlet that can make that conditionally.

I also think that the evaluation that "this becomes a break early vs create a bunch of later bugs" is only correct in the first half: it broke early and it broke hard. But I'm sure we'll have many bugs later and a lot of inflexibility.

Well the current conflicting method has significantly clarified the checking in the piles of tools wishing to mutate or observe the boot state. Most recently akmods, which has a grubby check followed by a bootctl is-installed where grubby is trusted over bootctl, because bootctl is doing little more than checking for the presence of systemd-boot on the ESP. Which 100% breaks in the face of multiboot with differing linux installs because of the insistence to use the same systemd-boot path location across distros, and the variable/loaders aren't fully reliable either as the 'default' boot selections can be overridden by the user from boot to boot.

So, I don't think the current method is problematic except for this case that the implications of conflicting bootloaders in the -comps files is exposing issues with the way the images are being built. In this case the exclusions during live media and the resulting continuing pollution post install of unnecessary packages being on the disk and upgraded.

I have another PR open which AFAIK actually fixes this remaining -comps problem simply by marking sdubby as "optional" along with the other three packages excluded from the live media by the build process. This has two positive results. it allows the removal of those build specific exclusions, its still puts the listed packages on the install DVDs/repos and it avoids upgrades done to live media installed systems from pulling in the packages which were excluded during the live build. AKA it fixes more than this immediate problem.

https://pagure.io/fedora-comps/pull-request/905

And so it is IMHO a good thing this is breaking, without which its quite possible that these packages would have continued to be pulled onto peoples machines for the forseable future, so yes, break early.