As I mentioned in the upstream issue, we should also remove systemd from the Kickstart. Adding it was a mistake that crept in during the move to build the images as part of the nightly composes.

In the past, until Fedora 36, the fedora base image used to contain systemd, but stripped it of documentation and translations. In such cases, we would dnf reinstall systemd in the fedora-toolbox images to restore the stripped out bits to avoid having an RPM that's half present half absent. Other than that there's no need for systemd in the image as of today.

As I mentioned in the upstream issue, we should also remove
systemd from the Kickstart. Adding it was a mistake that crept
in during the move to build the images as part of the nightly
composes.

To avoid future confusion: we do want to keep including weak dependencies, but having systemd included in the Kickstart by mistake led to this specific bug.

Scratch build with the proposed change:
https://koji.fedoraproject.org/koji/taskinfo?taskID=110004054

... made with:

$ cat fedora-container-toolbox-rawhide.conf 
[image-build]
name = Fedora-Container-Toolbox
version = Rawhide
target = f40
install_tree = https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/compose/Everything/$arch/os/
arches =  aarch64,ppc64le,s390x,x86_64
release = 20231207.n.0.rishi
format = docker
distro = Fedora-22
repo = https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/compose/Everything/$arch/os/
disk_size = 5
kickstart = fedora-container-toolbox.ks
ksurl = git+https://pagure.io/forks/rishi/fedora-kickstarts.git?#5a77cbf073fc7579c755a34514c5dc5d97382aa9
optional_arches = aarch64,ppc64le,s390x,x86_64

[factory-parameters]
dockerversion = 1.10.1
docker_cmd = [ "/bin/bash" ]
docker_env = [ "container=oci" ]
docker_label = { "name": "fedora", "license": "MIT", "vendor": "Fedora Project", "version": "40", "com.github.containers.toolbox": "true"}

$ koji image-build --scratch --config fedora-container-toolbox-rawhide.conf

You can download the tarball and do:

$ skopeo copy docker-archive:Fedora-Container-Toolbox-Rawhide-20231207.n.0.rishi.x86_64.tar.xz containers-storage:localhost/fedora-toolbox:40

I wonder if we can add a test to the Kickstart that ensures that /etc/resolv.conf is a regular file, and not a symbolic link. We have some tests already to assert the presence of certain files.

As I mentioned in the upstream issue, we should also remove systemd from the Kickstart.

ahh ok, maybe I misunderstood the diff. If we remove systemd, we shouldn't really need to also remove systemd-resolved right?

As I mentioned in the upstream issue, we should also remove systemd from the Kickstart.

ahh ok, maybe I misunderstood the diff.

All those dashes or - at subtly different indentation levels are hard to read. :)

If we remove systemd, we shouldn't really need to also remove systemd-resolved right?

I suppose that depends on your level of paranoia. Conceptually, we don't need systemd-resolved.service(8) inside the container, so it's not wrong to explicitly exclude systemd-resolved.

Ultimately, I think the most important part is to have a test in the Kickstart that ensures that /etc/resolv.conf is a regular file, because RPMs are always in a state of flux, and is hard to keep on top of everything.

This has now been pushed with an updated change that removes systemd as discussed

I would only exclude systemd-resolved explicitly. The systemd package seems big enough and widespread enough that something somewhere might want to pull it in. That can either break the build or lead to weird behaviour. In the past, it actually used to be part of the images that way.

On the other hand, systemd-resolved is a lot more targeted, and we really only want to ensure that it doesn't get pulled in through some weak dependencies.

As I mentioned in the upstream issue, we should also remove systemd from the Kickstart.

OK, sorry I think I'm misunderstanding this -- obviously you don't mean exclude it from this kickstart then. Where are you thinking we'd remove systemd then?

I meant a change like this:

$ git diff
diff --git a/fedora-container-toolbox.ks b/fedora-container-toolbox.ks
index 89e8ee9d38b9..5e86e5f4cba7 100644
--- a/fedora-container-toolbox.ks
+++ b/fedora-container-toolbox.ks
@@ -82,7 +82,7 @@ shadow-utils
 -shared-mime-info
 -sssd-client
 sudo
-systemd
+-systemd-resolved
 tar # https://bugzilla.redhat.com/show_bug.cgi?id=1409920
 tcpdump
 time

It seems to me that it's systemd-resolved that's causing the problem of changing /etc/resolv.conf into a symbolic link. So, that's the package that we should really exclude with a dash or - in front of it.

I don't know if we can say that we don't want systemd in the fedora-toolbox image. For a while it was being pulled in by the fedora base image, then it wasn't, and it didn't matter either way. So, we shouldn't list it as one of the packages required by the image, but I don't see a reason to explicitly exclude it either. It seems like a big enough package that some other random package might end up requiring it for one reason or another, but, regardless, it's systemd-resolved that should never get pulled in.

OK, I see now I think ... remove the systemd explicit install (but don't exclude it) and specifically exclude systemd-resolved. This is done in the latest push

(note to self; if ever designing a config file, do not have leading space/-/+ as part of the config syntax :smile: )

Yay! Thanks, @iwienand

@nirik @humaton this looks good to me! Could you please merge it to main?

(note to self; if ever designing a config file, do not have leading space/-/+ as part of the config syntax :smile: )

Totally! :)

@nirik @humaton this looks good to me! Could you please merge it to main?

Sorry, @kevin

you want it merged?

you want it merged?

Yes, that would be wonderful! :)

I realized that I had mistyped Kevin's Fedora ID.

I saw it, but am on PTO, so in and out and about. ;)

Lets merge!

Yay, thanks!

Corresponding pull request for f39: https://pagure.io/fedora-kickstarts/pull-request/1027