@kevin noticed that Koji can get clogged up with scratch builds that seem to be CI systems testing dist-git pull requests. For instance right now:
[adamw@xps13a dist-git-build-pipeline (scratch-build-background)]$ koji list-tasks --arch s390x | grep OPEN | grep buildArc | egrep "(zuul|ci)" 117714081 19 bpeck/jenkins-continuous-infra.apps.ci.centos.org OPEN s390x buildArch (firefox-126.0-6.fc41.src.rpm, s390x) 117714180 19 bpeck/jenkins-continuous-infra.apps.ci.centos.org OPEN s390x buildArch (firefox-126.0-6.fc41.src.rpm, s390x) 117714187 19 bpeck/jenkins-continuous-infra.apps.ci.centos.org OPEN s390x buildArch (firefox-126.0-6.fc41.src.rpm, s390x) 117714192 19 bpeck/jenkins-continuous-infra.apps.ci.centos.org OPEN s390x buildArch (firefox-126.0-6.fc41.src.rpm, s390x) 117714437 19 zuul OPEN s390x buildArch (firefox-126.0-6.fc41.src.rpm, s390x) 117714526 19 zuul OPEN s390x buildArch (firefox-126.0-6.fc41.src.rpm, s390x) 117714527 19 zuul OPEN s390x buildArch (firefox-126.0-6.fc41.src.rpm, s390x) 117714550 19 zuul OPEN s390x buildArch (firefox-126.0-6.fc41.src.rpm, s390x) 117715078 19 bpeck/jenkins-continuous-infra.apps.ci.centos.org OPEN s390x buildArch (clang-18.1.4-3.fc41.src.rpm, s390x) 117715291 19 zuul OPEN s390x buildArch (clang-18.1.4-3.fc41.src.rpm, s390x) 117715897 19 bpeck/jenkins-continuous-infra.apps.ci.centos.org OPEN s390x buildArch (clang-18.1.4-3.fc41.src.rpm, s390x) 117716050 19 zuul OPEN s390x buildArch (clang-18.1.4-3.fc41.src.rpm, s390x) 117717103 19 bpeck/jenkins-continuous-infra.apps.ci.centos.org OPEN s390x buildArch (cmake-3.28.3-5.fc41.src.rpm, s390x)
we have 13 different scratch builds running from CI pipelines on s390x. This is blocking real builds from getting through.
I believe the two pipelines are:
This is essentially a tracker issue as infra can't do much about this directly, but we want to keep track of it.
I have sent PRs for both to run their builds with --background, which should mitigate the issue a bit at least: dist-git-build-pipeline PR, zuul-distro-jobs PR. It would also be good if we did not have two apparently-redundant CI pipelines doing the same basic job ("see if a scratch build of a dist-git PR works"). I think @mvadkert has explained why we have two of these before, but I forget. @music may also be able to help here. Also tagging @msrb for the Fedora CI pipeline and @fbo for the Software Factory one.
--background
ASAP.
other things we could potentially do here, I guess, are restrict CI scratch builds for known-heavy packages like firefox and libreoffice, perhaps rate limiting them somehow or not doing them on arches with limited capacity?
Metadata Update from @phsmoura: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
CI ticket to de-duplicate fedora-ci and zuul: https://pagure.io/fedora-ci/general/issue/476
So, both the --background pr's are merged. Do we still want to keep this open?
I realize there's a bunch of discussion about deuplicating things, but I am not sure any of that is anything infra has control over?
I've confirmed that both CI services indeed submit scratch builds with priority 25 (background). The issue of two concurrent CI systems is tracked in the Fedora CI tracker linked above. I believe that there is nothing more for Fedora Infrastructure to do at this time and we should close this ticket.
Metadata Update from @zlopez: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.