It looks like there is only 15 GiB of RAM available on this builder. That's probably too low regardless of parallelism level; there is %limit_build in the spec file requesting 32 GB for the debuginfo stage, with a comment saying that 16 GB is not enough.

So, the build did finally finish. I was tailing the logs and it was at the end in debuginfo that it was having problems.

After freeze is over, I plan to try reinstalling some of the power9 hosts to see if I can get a more performant setup.

The ppc64le builders all have a 4.0 weight. webkitgtk has a 6.0 weight, so if its running on a builder, koji shouldn't schedule anything more on that builder (since it's already over capacity).

I am unsure what we can do downstream here to improve things aside from making builders more performant somehow.

I suggest a minimum of 32 GB of RAM for the heavybuilder channel. Pretty sure 16 GB is just not enough.

Well, I have no endless bag of memory... ;) So, the only way I can make a bigger one is by scrapping some others... which isn't great since most packages are small and having more smaller builders is better than fewer larger builders, but sure, we can look at that after freeze is over I suppose.

I'm also seeing if perhaps we could get more memory in next years budget rounds...

I've been playing with different configurations to see what I can manage to make things more performant/rebalanced.

ok, finally getting back to this.

I have setup 3 ppc64le builders with 32gb memory and put them in the heavybuilder channel.

So, please do look the next few builds and see if things are better?

I'd expect that will be enough. Will complain here if not.

Looks like the builds on the 9th went ok.

Closing this for now, feel free to re-open or file a new ticket if you see problems again.