fedora-ci / general

#79 CI tests for kernel package PRs fail — cloud image doesn't boot using virtio drive interface
Closed by bgoncalv. Opened by dpward.

Closed
Reopen Issue

This pull request, and the failing CI logs, provide an example of the issue.

Here's what seems to be happening:

  • The "cloud-image-compose" CI stage downloads a working Fedora Rawhide cloud base image. (Note: you can download that same image on your own, and successfully boot it in QEMU using the virtio drive interface.)
  • The "cloud-image-compose" CI stage then runs virt-customize to (among other things) install the package to be tested — in this case, the new kernel package.
    To do this, virt-customize will boot the image in QEMU, without specifying the drive interface — and QEMU picks ide rather than virtio.
    After the kernel package is upgraded, dracut is run automatically to update the initramfs. Keep in mind that dracut detects the kernel modules that are currently in use, and virtio_blk is not one of them.
  • The "nvr-verify" CI stage tries to boot the now-customized image in QEMU — using the virtio drive interface — so it can test out the package. However, the virtio_blk module does not get loaded from the initramfs, so the root partition cannot be found or mounted, and it drops to an emergency shell.

The standard-test-roles' QEMU inventory was changed by in April so that it uses the virtio drive interface for RHEL 8 compatibility. (@astepano)

This problem doesn't seem to happen when building different packages using this pipeline.

(cc: kernel maintainers @jforbes, @jwboyer, @labbott)

@dpward hi, thank for very decent explanation.
I would propose that virt-customize and STR stays in sync, what is necessary is to add virtio drive interface at virt-customize step.
@bgoncalv you know that virt-customize part the best. Could you please take a look? Thank you!

from virt-customize stage I see:

libguestfs: qemu KVM: enabled
libguestfs: finished testing qemu features
/usr/bin/qemu-kvm \
    -global virtio-blk-pci.scsi=off \
    -no-user-config \
    -enable-fips \
    -nodefaults \
    -display none \
    -machine accel=kvm:tcg \
    -cpu host \
    -m 4096 \
    -no-reboot \
    -rtc driftfix=slew \
    -no-hpet \
    -global kvm-pit.lost_tick_policy=discard \
    -kernel /var/tmp/.guestfs-0/appliance.d/kernel \
    -initrd /var/tmp/.guestfs-0/appliance.d/initrd \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-pci,rng=rng0 \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/workDir/workspace/fedora-rawhide-pr-pipeline/Fedora-Rawhide.qcow2,cache=writeback,discard=unmap,id=hd0,if=none \
    -device scsi-hd,drive=hd0 \
    -drive file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none,format=raw \
    -device scsi-hd,drive=appliance \
    -device virtio-serial-pci \
    -serial stdio \
    -device sga \
    -chardev socket,path=/tmp/libguestfsXh19NT/guestfsd.sock,id=channel0 \
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -netdev user,id=usernet,net=169.254.0.0/16 \
    -device virtio-net-pci,netdev=usernet \
    -append "panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 guestfs_network=1 TERM=xterm"
libguestfs: responding to serial console Device Status Report

It looks like virt-customize is using virtio...
Edited by bgoncalv

Metadata Update from @bgoncalv:
- Issue status updated to: Closed (was: Open)

Metadata Update from @bgoncalv:
- Issue status updated to: Open (was: Closed)

It looks like virt-customize is using virtio...

Notice if=none at the end here:

    -drive file=/workDir/workspace/fedora-rawhide-pr-pipeline/Fedora-Rawhide.qcow2,cache=writeback,discard=unmap,id=hd0,if=none \

This should be if=virtio.

libguestfs does allow setting the drive interface in the library API — but I don't know if this is exposed to users of virt-customize.
Edited by dpward

It looks like virt-customize is using virtio...

Notice if=none at the end here:
-drive file=/workDir/workspace/fedora-rawhide-pr-pipeline/Fedora-Rawhide.qcow2,cache=writeback,discard=unmap,id=hd0,if=none \

This is unrelated.

The real question is: what do you actually see that makes you think that libguestfs does not use virtio? You can easily check what libguestfs does, and the internal devices in the appliance, by running libguestfs-test-tool.

Specifically:

— and QEMU picks ide rather than virtio.

Can you please explain how do you notice ide is used?

I ran the QEMU command line executed by virt-customize manually, but added
-boot menu=on,splash-timeout=5000
and in the BIOS boot menu (via VNC, which I also enabled) the drive appeared as an ATA device. After the image booted, I ran lsmod and saw that the virtio_blk kernel module was not loaded.

When I changed the command to use if=virtio, then it was listed as a Virtio drive in the BIOS boot menu instead.

Perhaps I did something wrong, but that's what led me here.
Edited by dpward

IMHO that is a "bug analysis" -- i.e. you notice an issue, and start investigating.

What I see in the output of libguestfs-test-tool is that virtio_blk is definitely in use:

$ libguestfs-test-tool
[...]
+ lsmod
Module                  Size  Used by
[...]
virtio_blk             20480  0
[...]

You can easily check which virtio-related modules were included in the initramfs:

$ guestfish -a DISK -i initrd_list /boot/initramfs.img | grep virtio

(where DISK is the disk of the guest, and /boot/initramfs.img is the path of the initramfs in the guest to inspect)

I don't see problem booting the VM when using new kernel builds...

5.4.0-0.rc5.git1.3.fc32.x86_64

https://jenkins-continuous-infra.apps.ci.centos.org/view/Fedora%20All%20Packages%20Pipeline/job/fedora-rawhide-build-pipeline/7105/artifact/nvr-verify/logs/

or

https://jenkins-continuous-infra.apps.ci.centos.org/view/Fedora%20All%20Packages%20Pipeline/job/fedora-rawhide-build-pipeline/7112/artifact/nvr-verify/logs/
Edited by bgoncalv

I don't see problem booting the VM when using new kernel builds...

This issue is still happening with new PRs (see below). In case it makes a difference, the builds you referenced are using "fedora-rawhide-build-pipeline" rather than "fedora-rawhide-pr-pipeline" (i.e. the maintainer made a commit, versus a pull request was received).

What I see in the output of libguestfs-test-tool is that virtio_blk is definitely in use

Apologies for the confusion. When I tried to run virt-customize locally, it was using the "direct" backend to libguestfs instead of the "libvirt" backend, and I believe that affected the behavior.

Let me back up here.

The image artifacts from the previous CI test have expired from the server. So, here is a new kernel PR that I created, and the failed CI test results caused by the same issue.

  • I can download the customized cloud image artifact. If I launch this image in QEMU locally using the same command that is displayed in the CI log, it fails to find the root filesystem and drops to an emergency shell in the console (as shown in the CI log).

  • If I modify the QEMU command so that it uses if=ide instead of if=virtio, the image boots to a login prompt.

  • The image above was created during the cloud-image-compose stage by first downloading an existing cloud image and then customizing it. When I replace the customized image with this existing image, and run the same QEMU command as before (with if=virtio), it boots to a login prompt.

So, what part of the cloud-image-compose stage is breaking the ability to boot this image using the virtio drive interface?

I don't see problem booting the VM when using new kernel builds...

This issue is still happening with new PRs (see below). In case it makes a difference, the builds you referenced are using "fedora-rawhide-build-pipeline" rather than "fedora-rawhide-pr-pipeline" (i.e. the maintainer made a commit, versus a pull request was received).

The problem seems to be with the scratch build the PR created, I've forced the scratch build from your PR to run on build pipeline and it also had the same problem.
Assuming the patch from the PR didn't cause this issue, I'd say it is some problem in how we create the scratch build as part of PR pipeline.

This is basically what the pipeline does to create the scratch build, can you spot anything that for kernel build should be done differently?

# git clone https://src.fedoraproject.org/rpms/kernel.git
# pushd kernel
# git checkout master
# git fetch -fu origin refs/pull/39/head:pr
# git -c 'user.name=Fedora CI' -c user.email=ci@lists.fedoraproject.org merge pr -m 'Fedora CI pipeline'
# sed -i 's/%{?dist}/%{?dist}.pr.07533bd6f6154e008b2b1ad7a1c48bea/' kernel.spec
# fedpkg --release master srpm
# kinit -k -t /workDir/workspace/fedora-rawhide-pr-pipeline/fedora.keytab bpeck/jenkins-continuous-infra.apps.ci.centos.org@FEDORAPROJECT.ORG
# koji build --wait --arch-override=x86_64 --scratch rawhide kernel-5.4.0-0.rc6.git0.1.fc32.pr.07533bd6f6154e008b2b1ad7a1c48bea.src.rpm

In the emergency shell, uname -r returns

kernel-5.4.0-0.rc6.git0.1.fc32.x86_64

But, all of the modules are installed in
/lib/modules/5.4.0-0.rc6.git0.1.fc32.pr.07533bd6f6154e008b2b1ad7a1c48bea.x86_64
— which is not /lib/modules/$(uname -r). And udev can't seem to find them.

From the SPEC file:

    # make sure EXTRAVERSION says what we want it to say
    # Trim the release if this is a CI build, since KERNELVERSION is limited to 64 characters
    ShortRel=$(perl -e "print \"%{release}\" =~ s/\.pr\.[0-9A-Fa-f]{32}//r")
    perl -p -i -e "s/^EXTRAVERSION.*/EXTRAVERSION = -${ShortRel}.%{_target_cpu}${Flav}/" Makefile

Have we actually hit this limit before? Is dropping the PR tag entirely — so that there is no sign from uname -r that this is a scratch build — actually desired? Can some shorter alternative to the 32-character hex string be used either everywhere, or in places where the kernel expects to find its version string?
Edited by dpward

@mvadkert @pingou @msrb it looks like the magic to add the commit has to nvr of the PR scratch builds is causing issues with kernel. Maybe we need another way for pagure to identify the build without need to get the information from the NVR...

There could be an easyfix: do you need the full 32 characters for the commits? Would 6 or 8 or 10 work?

For the pipeline less characters is not a problem, I think the only issue is to make sure pagure can identify the build if it has only 6 or 8 extra characters.

Is that really a commit id? I don't know how this is implemented in Fedora, but if it is the same "trick" as we use in OSCI, then it is actually a Pagure's PR id, which is usually a randomly generated UUID. OSCI uses this information so we can later map test results back to pull requests in Pagure.

@pingou is it the same in Fedora CI as well?

Looking at: https://pagure.io/Fedora-Infra/loopabull-tasks/blob/master/f/playbooks/roles/flag_ci_pr/files/flag_ci_pr.py which is the code used to flag PRs based on CI messages, it seems to be using the rev to find out which PR is concerned by a given results.

It doesn't seem to do anything with the nvr field nor does it seem to be using the PR UUID.

Note that this script is triggered on: org.centos.prod.ci.pipeline.allpackages-pr.complete and org.centos.prod.ci.pipeline.allpackages-pr.package.running.

Looking at: https://pagure.io/Fedora-Infra/loopabull-tasks/blob/master/f/playbooks/roles/flag_ci_pr/files/flag_ci_pr.py which is the code used to flag PRs based on CI messages, it seems to be using the rev to find out which PR is concerned by a given results.
It doesn't seem to do anything with the nvr field nor does it seem to be using the PR UUID.
Note that this script is triggered on: org.centos.prod.ci.pipeline.allpackages-pr.complete and org.centos.prod.ci.pipeline.allpackages-pr.package.running.

Does it mean the pipeline still listening on old message format. What do we need to update for it to process org.centos.prod.ci.dist-git-pr.test.complete and org.centos.prod.ci.dist-git-pr.test.error ?

@pingou do it mean for Fedora CI we don't need to change the nvr for the build at all? The would be great :)

@pingou do it mean for Fedora CI we don't need to change the nvr for the build at all?

That what the code seems to imply :)

What do we need to update for it to process org.centos.prod.ci.dist-git-pr.test.complete and org.centos.prod.ci.dist-git-pr.test.error ?

We need to update the playbooks in that project and potentially the script as well :)

@dpward could you try it again? It is hopefully fixed with https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/pull/199

@dpward could you try it again? It is hopefully fixed with https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/pull/199

Running now: https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2253/pipeline/

@dpward could you try it again? It is hopefully fixed with https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/pull/199

Running now: https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2253/pipeline/

Seems like it worked?

Right, the VM is able to boot up now :)

I'm trying to understand why the test is failing when we run it in the pipeline.

@dpward It took me a while to understand why the test fail.

The test uses fd3

but STR beakerlib role also uses it so it fails.

For quick fix the test could easily be rewritten to be just bash and use basic role.

@bgoncalv Thanks for the pointer — after updating the PR to improve that test case, the CI tests completed successfully.

Question: why is that test/result not appearing in Pagure? I only found it by searching Jenkins.
Edited by dpward

@dpward thanks, nice the test now passes, I'll close this issue.

For the missing result in the PR I opened issue.

Metadata Update from @bgoncalv:
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.