@alexl ping :)

Right now, if we just revert these its not gonna work for the reason you point out. Solving that would require some development on rpm-ostree.

But, I'm not really involved in rpm-ostree and silverblue any more, so I'm not gonna be much help here.

We have two decent options:
- get the kernel to add a dependency from kernel-devel to same-version kernel; or
- add the constraint in rpm-ostree's depsolver

I'm leaning towards the second one. I spent a good amount of time already wading through that code and I feel like I could probably manage it (although I'm certainly not the most qualified person to do it). I'm gonna check with Colin to make sure it makes sense at all...

The one issue I can think of: as far as I know, there's nothing in place to ensure that we'll get the same kernel-devel as the installed kernel version, in case a newer kernel-devel is available, for example. That still needs to be addressed.

Hm, how so? I am pretty sure the base image always contains the latest available versions of both kernel and kernel-devel, and they are pushed to updates repo at the same time (they come from the same SRPM so they can't be out of sync in the updates repo).

Did you actually run into this issue somehow or is it just a thought experiment?

For what it's worth, I think (1) from the list of options is much much simpler. Maybe reach out to the kernel people and ask their opinion?

Option 1 feels simpler to me too but I'll let @walters @jlebon chime in.

Hm, how so? I am pretty sure the base image always contains the latest available versions of both kernel and kernel-devel, and they are pushed to updates repo at the same time (they come from the same SRPM so they can't be out of sync in the updates repo).

This PR is about removing kernel-devel from the base image.

The problem is that we pull the kernel from the repository at one point when composing the base image and (possibly much) later the user overlays kernel-devel. At that point, the versions available in the archive might be different.

It's probably not set up like this already in the kernel RPM because you can build modules for multiple kernel versions without actually having to install the various target kernels on the build machine. So it's unlikely they'd be willing to change that.

Hmm, one option they might be more open to would be a meta subpackage (e.g. kernel-devel-meta) which binds both kernel and kernel-devel versions together.

The problem is that we pull the kernel from the repository at one point when composing the base image and (possibly much) later the user overlays kernel-devel.

Just to note, AFAICT there's nothing actually rpm-ostree-specific with this problem. It can happen just as well on out-of-date dnf-based systems where the user wants to install kernel-devel. (Usually, dnf solves the split pkg version problem by also upgrading the base package, but if there's no link between the two like here that's not going to happen).

Seems like for now we should be able to pretty easily drop the glibc-devel and elfutils-devel deps to start with (assuming akmods already pulls those in).

This is really about kernel-devel. Dropping glibc-devel and elfutils-devel isn't a really big gain, so I'd prefer to wait.

Is this just about reducing the base image size, or did you run into some kind of issue with kernel and kernel-devel being out of sync? Just trying to understand what's going on :)

Entirely about reducing the base image size (and dropping perl and the toolchain).

This is a revert of an original patch from Alex to ensure that things would stay in sync. Removing this, will actually make matters worse in that regard.

The thing that changed is that with the new archive repo (and some kernel dependency tweaks), we can now hope to solve this problem without the crutch of having kernel-devel installed in the base image.

OK, thanks for clarifying it :)

Opened a PR for kernel-devel → kernel depend. Let's see how that goes.

https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1083

Entirely about reducing the base image size (and dropping perl and the toolchain).

I got some numbers.

With the recent samba changes, and the foomatic removal now merged, this PR alone (ie: dropping the -devel depends) will now result in 87 packages removed and ~350MB less space used (uncompressed installed).

Entirely about reducing the base image size (and dropping perl and the toolchain).

I got some numbers.

With the recent samba changes, and the foomatic removal now merged, this PR alone (ie: dropping the -devel depends) will now result in 87 packages removed and ~350MB less space used (uncompressed installed).

Nice numbers! Do you have some links to commits/bugs for the samba changes and the foomatic removal?

Samba: https://bugzilla.redhat.com/show_bug.cgi?id=1949295

foomatic: https://pagure.io/fedora-comps/pull-request/645 → https://pagure.io/workstation-ostree-config/pull-request/216

This doesn't fully make sense to me - in some future, we might want to gate Silverblue releases on "works with the NVIDIA drivers". If I understand correctly, even with the updates-archive repo, most of the ~90 packages that will be layered will still be layered as "latest stable version".

And we're also introducing quite a delta between the package set on a Silverblue system with NVIDIA drivers to one without NVIDIA drivers.

But that being said, it might be a practical step to reduce the size of the base image. Other options would be much harder to get going - e.g., building the drivers in a container.

I think we need to figure out how to get the correct version of kernel-devel installed before we land this. (If rpm-ostree effectively adds "Requires: kernel-X.Y.Z Conflicts: kernel != X.Y.Z to the transaction , then Requires: kernel-devel in a layerd package probably works?)

Assuming that that was fixed, would the instructions at https://rpmfusion.org/Howto/NVIDIA#Silverblue still work? If not, what about them would need to change?

For OpenShift the strategy is focused around building drivers (including nvidia) in a container: https://github.com/openshift/enhancements/pull/357

I think we need to figure out how to get the correct version of kernel-devel installed before we land this. (If rpm-ostree effectively adds "Requires: kernel-X.Y.Z Conflicts: kernel != X.Y.Z to the transaction , then Requires: kernel-devel in a layerd package probably works?)

That's here: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1083

I talked to Colin, and he preferred this route as a first option, and adding the hack to rpm-ostree only if necessary.

I think the Conflicts: kernel != X.Y.Z thing shouldn't be necessary because of rpm-ostree disabling installonly, which forces exactly one kernel version, which means that depending on the one version is effectively in conflict with having a different one. All that said, I don't actually know how libdnf will behave here, or if it will insist on replacing the kernel package in the base version or not. In any case, though, that would trigger just a warning that the user needs to update their system before attempting the layering, which seems pretty OK to me...

We have two decent options:
- get the kernel to add a dependency from kernel-devel to same-version kernel; or
- add the constraint in rpm-ostree's depsolver

I'm leaning towards the second one.

I attempted to get the kernel people to do the first option, but they don't think it's a good idea, so I guess we're back to the option that I liked better in the first place anyway. :)

Time to get my hands dirty...

Sort of related to this - after some weekends update:

Added:
binutils-2.35.1-41.fc34.x86_64
binutils-gold-2.35.1-41.fc34.x86_64
bison-3.7.4-2.fc34.x86_64
flex-2.6.4-7.fc34.x86_64
gcc-11.1.1-1.fc34.x86_64
m4-1.4.18-17.fc34.x86_64
make-1:4.3-5.fc34.x86_64
openssl-devel-1:1.1.1k-1.fc34.x86_64

$ rpm -q --whatrequires gcc
kernel-devel-5.12.5-300.fc34.x86_64
$ rpm -q --whatrequires kernel-devel
no package requires kernel-devel

These were added with https://src.fedoraproject.org/rpms/kernel/c/17639df044f5f81bbf267f566426eae4a5187875?branch=f34 resulting in https://src.fedoraproject.org/rpms/kernel/blob/f34/f/kernel.spec#_1062 :/

@jforbes any particular reason why the requires in kernel-devel were changed?

https://gitlab.com/cki-project/kernel-ark/-/merge_requests/972

How about a third option given that now the -devel package is pulling gcc, etc., the rpm-ostree changes are not ready yet, and the kernel is basically a special case here as all other -devel packages I could find directly require the matching base package version:

• Add a new empty sub-package kernel-core-devel-matched (name TBD) that Requires fixed versions of both kernel-core & kernel-devel. This would be essentially unused by everyone but could be added in the Silverblue/Kinoite documentations as an additional dependency for those installing external drivers, etc.

I am somewhat curious here. Pulling in gcc, etc. If you need kernel-devel, you need gcc. If you are running 3rd party modules that you are getting from somewhere else you do not need gcc, but you also do not need kernel-devel. If you are building them with dkms, you definitely do need gcc. The only Requires in that MR that are not direct Requires of dkms already are bison, flex, and openssl-devel. All of the others (gcc, make, and elfutils-libelf-devel) are already required by dkms.

The goal is to not have kernel-devel (or the new kernel-core-devel-matched) installed by default in Silverblue. kernel-devel is included right now to make sure the version is the same as the kernel one as there are no fixed Requires for the kernel on -devel.

Silverblue users wanting to install akmod modules would then install kernel-core-devel-matched which would pull the -devel package matching the version of the kernel-core package included in the image as well as the dependencies (gcc, etc.).

Created https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1195

Created https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1195

Merged! Amazing!

Now we just need to change the nvidia module dependency to use the matched package.

:thumbsup:

I really have no idea what I'm doing at this point, but I guess something like this seems vaguely appropriate: https://src.fedoraproject.org/rpms/akmods/pull-request/3

Any chance this might make it before F35? It'd sure be nice to have 76 fewer packages on my system...

I've made progress toward testing and validating that but I'm not 100% there yet. As we are near the release, I don't think we will be able to include it right now but this might happen after the release.

Thanks for the update 👍

I wrote a longer explanation of this change in: https://src.fedoraproject.org/rpms/akmods/pull-request/3#comment-90939

https://src.fedoraproject.org/rpms/akmods/pull-request/3 landed, but is only effective on fedora >= 36.

I've made this change request page to summarize the situation: https://fedoraproject.org/wiki/Changes/Binary_Kernel_Modules_rpm-ostree

We should be able to merge this for F36+ soon.

Quick poke: is this going to make it for F36 still?

Yes, I'll merge this soon but I'm holding doing too many important changes while Rawhide is broken due to the authselect change: https://bugzilla.redhat.com/show_bug.cgi?id=2034360.

(The change request has been accepted: https://pagure.io/fesco/issue/2735)

Yes, I'll merge this soon but I'm holding doing too many important changes while Rawhide is broken due to the authselect change: https://bugzilla.redhat.com/show_bug.cgi?id=2034360.

Yikes! I didn't realise. I hope you manage to get that sorted out nicely.

I don't mean to apply pressure, and if it needs to slip, it needs to slip. Just wanted to make sure it didn't fall between the cracks.

Merged https://pagure.io/workstation-ostree-config/pull-request/261 today. This one will have to wait for after the F36 branching/freeze.

Merging this one now and backporting it to f36 to be ready for the beta.