Currently the %limit_build and %constraint_build macros don't work properly. The goal of these macros is to limit build parallelism based on the amount of RAM available on the builder. This fails if multiple builds can be scheduled at the same time. For example:
The problem is the macros don't know about how many other concurrent builds may be scheduled. E.g. If %limit_build knew that two builds may be scheduled at the same time, it would simply half the level of parallelism and there would be no problem.
The easiest solution is to ensure no more than one build gets scheduled at a time if a package is using %limit_build or %constraint_build. Since RPM package builds generally parallelize extremely well, this possibly won't actually slow things down too much? (Builds would take longer to start, but then complete faster.) We might need to manually attach "heavy" packages to particular build channels to achieve this. Fortunately, there are probably not too many such packages.
2025/01/01
Oh, and both CentOS Stream and RHEL infrastructure will have the same problem beginning with el10 (the first release to include these macros), so we'll need to inform those teams about this problem too.
Could I get you to refile this at https://pagure.io/koji ? There may well be adjustments we can make in our deployment, but we don't want to make big downstream changes if at all possible.
I know upstream koji is reworking the scheduler right now, so this input might be useful to them to adjust things for this use case.
I think it's going to be very hard for koji to know these macros are set... it would have to unpack the rpm, get the spec, parse it, etc.
But they may have some ideas on how to better handle these cases...
Reported as https://pagure.io/koji/issue/4084
Metadata Update from @phsmoura: - Issue close_status updated to: Upstream - Issue status updated to: Closed (was: Open)
Well I think we need a Fedora infrastructure workaround, rather than completely deferring this to upstream koji. Can we have a hardcoded list of packages to receive the max build weight? I fear it will be almost impossible to make things reliable otherwise.
The goal of these macros is to limit build parallelism based on the amount of RAM available on the builder.
I've been in need of using %constrain_build lately. Looking at the macro, it determines the total amount of memory in the system, not the total amount available. That is, the next build using that macro will see exactly the same amount of total memory, no matter how much of that is currently being used by another build on the same builder.
%constrain_build
Unless I'm misunderstanding something, delaying the start of builds with the way the macro works, won't make any difference. Of course, the issue of overcommitting you describe, is still something worth taking into account.
Well, a) there's no such way to hardcode weight. koji calculates it from the build time of the package it's building. So if we wanted to adjust this at all, we would need to patch koji/write new code.
And... webkitgtk already has a 6.0 weight, so it should always get a 'full' ppc64le builder unless something is wrong in the scheduler...
It was actually aarch64 where two builds were recently running on the same builder at the same time. (I don't know about ppc64le.)
Log in to comment on this ticket.