I'm seeing this same failure now with the lld package, so it must not be related to the package name:

https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-build-pipeline/detail/fedora-rawhide-build-pipeline/7280/pipeline/

It looks like when the playbook installs the required packages to run the test it upgrade glibc, once glibc is upgraded it is not possible to ssh to the VM anymore and the test fails.

Upgrading:
glibc                                              x86_64                                  2.30.9000-18.fc32                                  rawhide                                  3.5 M
glibc-common                                       x86_64                                  2.30.9000-18.fc32                                  rawhide                                  1.9 M
glibc-langpack-en                                  x86_64                                  2.30.9000-18.fc32                                  rawhide                                  651 k

Note that the pipeline is running an old qcow2 (Fedora-Rawhide-20190920.n.1), because the newer ones we can't use in the pipeline due to https://bugzilla.redhat.com/show_bug.cgi?id=1754471

I've open a BZ for glibc.
https://bugzilla.redhat.com/show_bug.cgi?id=1773912

this is also affecting our packages, python3 and python-pip:

https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2325/pipeline/

https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2326/pipeline/

We are blocked until we can get an working Fedora Rawhide qcow2 image.

There is a bug open for it https://bugzilla.redhat.com/show_bug.cgi?id=1754471

Once a new qcow2 that we can boot up is released it will contain the updated openssh that is needed to fix this issue.

@bookwar is there anything else we can do right now?

I don't think we can do anything here from CI side. We do need rawhide cloud image to be bootable to run tests. This is a natural thing to ask for CI pipeline.

So let's give the dbus maintainer some time to reply on the issue, which was just reassigned to him.

Maybe one more request is to research if Fedora QA have already a test to verify that qcow image is bootable. If so we need to figure out how we can run this test in the gate rather than in the post-compose setup.

@tflink @adamwill ^^ maybe you can comment on the last part?

this is what you might call an 'active area'.

there is a tool - autocloud. unfortunately it is no longer maintained and is scheduled to die quite soon, per this thread. For now it runs, and its results are forwarded to resultsdb, so at least theoretically the results are available for gating operations via Greenwave.

One of my action items for this week is to look into keeping this working until CoreOS takes over the world, whether by running the tests in openQA, making autocloud work on Fedora 30, or whatever else.

Can CI use an older working image as a temporary workaround?

The CI is using an older image, the problem happens when some package from the build or required by the test pull new glibc via dependencies.

@bookwar prefers that the official Rawhide qcow2 images gets fixed instead of us working around qcow2 images for the pipeline.

heads up, I have replicated the autocloud test in openQA now. Results are reported to resultsdb for the testcase compose.cloud_autocloud , see here:

https://taskotron.fedoraproject.org/resultsdb/results?&testcases=compose.cloud_autocloud&since=2019-10-24T00:00:00,2019-11-22T23:59:59

FWIW this seems fixed.