For Fedora releases that release in April, schedule the mass rebuild in the last week of January to allow time for stabilization of GCC as well as fix up packages for compiler changes.
Even numbered Fedora releases that go out in April get a brand new GCC, which tends to have new language features, diagnostics and optimizations. Upstream GCC closes active development (stage 1) and switches to stabilization in November and then works towards making the compiler stable enough for these Fedora releases. Having the mass rebuild in the last week of January gives enough time for the GCC team to find and report package issues as well as fix compiler issues detected during a mass prebuild.
For Fedora 40, the mass rebuild was delayed (for dnf5?) because of which the problem wasn't as noticeable. Further, the GCC team tried doing a mass prebuild for the very first time, which used the delay effectively to find and fix a number of package issues before the actual mass rebuild.
Before Fedora 40, we had custom scripts to do something similar to mass prebuild by allocating several beefy boxes to isolate new failures. Additionally, some of the pain would be alleviated with other distributions (Gentoo for example) finding, reporting and helping fix compiler and packaging issues too, which continues today.
Looking at the calendar in the more recent past (2023 and 2022), it also seems coincidental that we ended up with 2 weeks after year-end holidays before the mass rebuild, giving enough time for people to fix issues before the rebuild, but that is just my speculation.
Mass rebuilds being scheduled for the last week of January gives enough time for upstream gcc to stabilize the new compiler and more importantly from the context of Fedora, allows the toolchain team to run mass prebuilds like we did for F40 to report and fix package issues before the mass rebuild. This should allow for a smoother mass rebuild. I can't be certain if we can shrink the post-mass-rebuild phase (i.e. leading up to branching) by a week to accommodate this, but maybe the way the F40 schedule turned out could give us that information.
We have been using something like mass prebuild for years before as well, often just with allocating several beefy boxes in beaker and using scripts around mock to build stuff and rebuild with older compiler what failed with the new one. Anyway, the schedule is that GCC development turns into bugfixing around mid November, first two weeks of that is really premature to try to build anything with that, the most urgent issues are being fixed at that time. After that it starts to be the time when we can prepare first test source rpms for the mass prebuild test, but with vacations over Christmas holidays and often even some days or weeks before for various people the usual thing is that the mass prebuild is run over the Christmas holidays and if the scripts do the right job, then we can start analyzing the results back in January (sometimes people are still on vacation for a few days at start of January, depending on countries etc.). The main intent of the mass prebuild is find bugs on the gcc side and if at all possible fix those or workaround, and be able to write porting_to.html guide for the new compiler on what needs to be changed in various packages. So, a mass rebuild on January 15 or 17th is pretty early for the analysis to be finished and bugs fixed. E.g. in years where as mass rebuild was done around January 25th or similar, we also had the possibility to actually change gcc in rawhide even before the mass rebuild so people had a couple of days to play with it and do voluntary rebuilds etc. This year, all we could manage was having the new gcc for around 3 days in a side-tag (with one day out of that during weekend). Note, gcc builds in koji in 18-48 hours, so any fixes have quite long latency.
Thanks, I've updated the description based on your comment.
Just a quick note, I see two potential issues with unconditionally moving the mass rebuild two weeks:
What I'm hoping is that front-loading fixes (especially to the more critical components) during the mass prebuild will help shorten the post mass-rebuild phase from ~3 weeks to, e.g. ~2 weeks.
I think "last week of January" is too much. What about moving it by one week, i.e. to January 22? This way we'd avoid overlap with FOSDEM, because the rebuild would mostly be done before people travel to Brussels on the last days of January.
I would also shorten the time allocated for the mass rebuild from 20 days to 13 days and keep the rest of schedule not changed.
I don't know if we can shorten the time for the mass build. This is usually the time where everything breaks pretty badly...
To be clear you mean reduce the post mass rebuild time (ie, time between mass rebuild and branching) by a week?
We could do that, but that time is pretty valuable to maintainers to allow them to fix any issues they saw from the mass rebuild, but also to make sure they have what they want landed so they don't have to update both branched and rawhide after branching. On the other hand, I am not sure where else we could get a week.
Have you considered trying to change the GCC release schedule? :-)
To be clear you mean reduce the post mass rebuild time (ie, time between mass rebuild and branching) by a week? We could do that, but that time is pretty valuable to maintainers to allow them to fix any issues they saw from the mass rebuild, but also to make sure they have what they want landed so they don't have to update both branched and rawhide after branching. On the other hand, I am not sure where else we could get a week.
Speaking from the perspective of ELN, this is particularly important for those releases which are also RHEL major version branch points (most recently F40). The delays in the F40 mass rebuild causes us to ask for a matching delay in the subsequent branching, so that we would have the expected amount of time to fix issues in rawhide before branching and avoid having to fix them in two places. Mind you it was a crazy few weeks and while we pulled it off, I certainly would not want to have less time to do so.
Adding to meeting agenda for tomorrow's fesco meeting. We can at least discuss it some.
17UTC in #meeting:fedoraproject.org for interested parties.
Metadata Update from @kevin: - Issue tagged with: meeting
AGREED: FESCo does not recommend adjusting the schedule due to such a change impacting too many other events. (+7,0,0)
Metadata Update from @kevin: - Issue close_status updated to: Rejected - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.