#532 PR CI has been unuseable for ~2 weeks
Closed by churchyard. Opened by churchyard.

Hello,

I know there is not much you can do about this, but I want to raise awareness. For the past approximately 2 weeks, the CI that runs on Pull Requests is pretty much unusable for packages with builds that take 1+ hour. As Koji and src.fp.o occasionally give 50x errors (see e.g. https://pagure.io/fedora-infrastructure/issue/12814) and the CI jobs heavily rely on koji scratchbuilds from dist git, every time the CI runner gets 50x it seems to abort (instead of retrying to fetch the build status, retrying to download etc.). When a job waits for a ~5 hours build to finish with polling or a blocking koji command, it's almost certain it gets a 50x during that time at least once -- effectively making the CI job very fragile.

It has been a very unfortunate experience for me, where we ned to [citest]/recheck builds multiple times in a row and it takes 2 to 3 days to successfully verify a change over CI.

Packit CI is a tad better with this, it seems, but not completely resistant.

Some examples include https://src.fedoraproject.org/rpms/python3.15/pull-request/1 to 8 or https://src.fedoraproject.org/rpms/python3.14/pull-request/112 to 122.


I tried to look for the 50X tracker for the koji issues to follow, but I couldn't find one. And didn't find one in the status page.

From the jenkins/zuul side, it could be quite tricky to design it since the jobs designs are waiting for the koji monitor. I may have a way around it for the jenkins runners by saving the taskID across the jobs, but I am blocked by an infra setup of redis server.

Packit Fedora-CI is more flexible because we own all the definitions and we can more easily save these and make it event-based, and the good folks there already have useful monitors like a babysit job to check up on stuck jobs, just need to dive into the implementation to polish it.

@churchyard with https://pagure.io/fedora-infrastructure/issue/12814 resolved, can we close this? I believe it should be a lot better now?

It is a lot better now, thanks.

Metadata Update from @churchyard:
- Issue status updated to: Closed (was: Open)

Metadata