These problems are triggered by multi-build updates that contain a ton of builds. In this case this one: https://bodhi.fedoraproject.org/updates/FEDORA-2021-d5931ab92d (66 builds).

It's always installability that triggers these issues, but there is a reason for it. rpmdeplint is ran only once for all builds in the update, and there is only a very small fraction of packages that actually have any dist-git tests, so there is nothing to run there. rpminspect runs on all builds separately, but it doesn't require all other builds to be downloaded as well.

As for the installability test, we know what is going on here. There are multiple problems:

• 65 builds in a single Bodhi update means spawning 65 installability jobs, and each is trying to prepare environment (=download builds) with all 65 builds in a local repository. I.e. we are trying to download 65*65 fresh builds that are likely not on any mirrors yet (https://pagure.io/fedora-ci/general/issue/183)
• this means that it takes forever (hours) for all jobs to finish (I've seen jobs running for 10 hours, and still spinning)
• we still actively wait for test results
• we are quite verbose in logs when we check Testing Farm every 20 seconds for results
• Jenkins is pretty IO intensive in general and its performance degrades if it handles many long running tasks that generate a lot of output. 100 concurrent builds that we can do right now (2 slaves, 50 executors each) is not that many, but if master is not running on a node with fast storage, it quickly shows and it can kill (restart) the instance. We've seen this many times already. The storage where the Jenkins master is running right now is not exactly fast.

Upstream is aware of this issue, but I think it is unlikely it will be resolved any time soon (if ever): https://issues.jenkins.io/browse/JENKINS-38313

TL;DR: waiting hours for test results on a slow storage is kinda dumb; webhooks should solve these problems nicely: https://pagure.io/fedora-ci/general/issue/173

I already have a working pull request for the webhook support: https://github.com/fedora-ci/dist-git-pipeline/pull/29, but I want to move the logic to the library.

@msrb this seems to be solved by moving to webhooks?

Yes! Good catch.