Related: https://github.com/fedora-ci/dist-git-build-pipeline/pull/40

@adamwill I guess all these changes should go through a community review, so main users like @churchyard can provide feedback. Did not actually think this will blow up this way :(

Related: https://github.com/fedora-ci/dist-git-build-pipeline/pull/40

Looking at this change and the way it affects PR triggered CI jobs, I believe we need some more granularity. While I agree that scratch builds can and should be run at a lower priority, they shouldn't be run at the same priority as a mass rebuild.

I usually start all my Koji jobs, regular updates and scratch builds, with --background. Knowing that the mass rebuild is going on, I decided to not run with --background, because my jobs would potentially be blocked out by the mass rebuild. Ideally, I'd like to set the priority just a little higher than the mass rebuild jobs, but still lower than regular or high priority jobs.

I'm not sure what granularity Koji allows. Assuming Koji admins can set arbitrary priority for jobs triggered by the mass rebuild, the problem might be solved by using a lower priority than set by --background for mass rebuild jobs.

well, there is a lot of granularity available, ultimately at the koji level it's an integer. but the koji CLI provides --background as a convenience that (IIRC) adds 5 to the priority.

we could fairly easily extend that mechanism to add an intermediate level of priority, I guess, which would be ideal. But honestly, I think "PR CI gets stuck behind mass rebuild" is still better than "real builds get stuck behind PR CI builds".

I don't know exactly how the mass rebuild builds are triggered, if it's in a way that would allow direct setting of the priority, they could just set a higher number (IIRC, higher number == lower priority) than --background does and that would solve this. if they're using the CLI and --background, I guess we have to add a further option to the CLI. cc @kevin

But honestly, I think "PR CI gets stuck behind mass rebuild" is still better than "real builds get stuck behind PR CI builds".

In practice it means a mess with PRs. Koji has a build timeout of 48h iirc. That's beyond an acceptable threshold that package maintainers would be willing to wait for confirmation if a PR has issues or not.

I think it's not a question of choosing the lesser of two evils but rather finding a solution everybody can live with. Package maintenance goes on throughout mass rebuilds and shouldn't be impeded by it.

But honestly, I think "PR CI gets stuck behind mass rebuild" is still better than "real builds get stuck behind PR CI builds".

I agree. However I also know koji background is an integer, so this should not be an either-or question.