From 05d495403e3da42ce93092950790e430d290268b Mon Sep 17 00:00:00 2001 From: Samyak Jain Date: Apr 03 2024 09:34:08 +0000 Subject: Update mass-rebuild sop Signed-off-by: Samyak Jain --- diff --git a/modules/release_guide/pages/mass_rebuild.adoc b/modules/release_guide/pages/mass_rebuild.adoc index 9830af6..45020ba 100644 --- a/modules/release_guide/pages/mass_rebuild.adoc +++ b/modules/release_guide/pages/mass_rebuild.adoc @@ -1,39 +1,13 @@ include::_partials/attributes.adoc[] -= Mass rebuild information += Mass Rebuild == Description -Periodically we do mass rebuilds of rawhide during the development cycle. This SOP will outline the steps necessary to do this. - -== Assumptions - -This assumes that the mass rebuild has already been approved and scheduled via release engineering and FESCo. Coordinate with infrastructure as well for any needed koji updates. - -This also assumes that the mass rebuild does not need to be done in dependency order, and that the mass rebuild does not involve a ABI change. - -== Considerations - -The most important thing to keep in mind while doing a mass rebuild is to communicate clearly what actions are being performed and the status of the rebuild. - -Check in on scripts frequently to avoid a long stalled command from adding significant delays in completing the rebuild. - -Check with secondary arches, whether they up-to-date enough with primary, create rebuild tag and target when they are. It will then take care of rebuilds of the arch specific packages in appropriate kojis. - -== Actions - -=== Preparatory Steps - -The following steps may be completed in the weeks leading up to the scheduled mass rebuild. - -== Mass Rebuild - -=== Description - Periodically we do mass rebuilds of rawhide during the development cycle. This SOP will outline the steps necessary to do this. -=== Assumptions +== Assumptions This assumes that the mass rebuild has already been approved and scheduled via release engineering and FESCo. Coordinate with @@ -43,7 +17,7 @@ This also assumes that the mass rebuild does not need to be done in dependency order, and that the mass rebuild does not involve a ABI change. -=== Considerations +== Considerations * The most important thing to keep in mind while doing a mass rebuild is to communicate clearly what actions are being performed and the status @@ -53,6 +27,7 @@ adding significant delays in completing the rebuild. * Check with secondary arches, whether they up-to-date enough with primary, create rebuild tag and target when they are. It will then take care of rebuilds of the arch specific packages in appropriate kojis. +* Please make sure you have right ACLs to run the script mention below, one needs to be atleast in sysadmin-releng/releng-team/cvsadmin/sysadmin in order to get root access of the machine if required. == Actions @@ -77,8 +52,10 @@ ____ The mass rebuild wiki page should answer the following questions for maintainers: -* Why the mass rebuild is happening -* How to opt out of the mass rebuild +* Why the mass rebuild is happening? +* How to opt out of the mass rebuild? + * Make sure to inform people either on mail, or in the wiki page about the `PKG_SKIP_LIST` which allows packages to be skipped from rebuilding as well as `noautobuild` file that they can add it to there root of a distgit repository and the mass rebuild script will skip it. +* What changes from changesets are taken in account for F{branched}? [NOTE,subs="attributes+"] ==== @@ -90,7 +67,7 @@ ____ + ____ Send out the same information posted on the wiki to the -[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list. +[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list. Please check if the email is through, if stuck in moderation please ping respective members. [NOTE] ==== @@ -173,6 +150,7 @@ reflect the Fedora release you're building for the mass rebuild is for have landed in the build system (and a newRepo task completed with those features) * the comment which is inserted into spec changelogs +* for the references of the above mentioned changes refer to the https://pagure.io/releng/c/f8d89a58efc0b81b9358baadfeb2ba4f52edeeb4?branch=main[F40 mass rebuild PR]. === Starting the Mass Rebuild @@ -186,14 +164,15 @@ The `mass-rebuild.py` script takes care of: * git tagging the change * Submitting the build request to Koji +===== . Connect to the mass-rebuild Machine + ____ .... -$ ssh branched-composer.iad2.fedoraproject.org +$ ssh compose-branched01.iad2.fedoraproject.org .... ____ -. Start a terminal multiplexer +. Start a terminal multiplexer (this ensures if user gets interrupted due to various reasons, the script can continue in a tmux session) + ____ .... @@ -210,6 +189,7 @@ $ cd path/to/releng_repo/scripts $ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out .... ____ +===== === Monitoring Mass Rebuilds @@ -218,14 +198,15 @@ many maintainers will want to know if their build failed right away. The `find-failures.py` and `need-rebuild.py` scripts are designed to update publicly available URLs for stakeholders to monitor. -. Connect to a Compose Machine +===== +. Connect to a Compose Machine or the monitoring scripts can be run on the same machine where the mass rebuild is happening. + ____ .... -$ ssh compose-x86-02.iad2.fedoraproject.org +$ ssh compose-x86-02.iad2.fedoraproject.org / compose-branched01.iad2.fedoraproject.org:22 .... ____ -. Start a terminal multiplexer +. Start a terminal multiplexer (this ensures if user gets interrupted due to various reasons, the script can continue in a tmux session) + ____ .... @@ -259,6 +240,7 @@ Set up the Site for Packages that Need Rebuilt:: .... $ while true; do ./need-rebuild.py > f{rawhide}-need-rebuild.html && cp f{rawhide}-need-rebuild.html /mnt/koji/mass-rebuild/f{rawhide}-need-rebuild.html; sleep 600; done .... +===== === Post Mass Rebuild Tasks @@ -272,6 +254,7 @@ package * Tag remaining builds into their final destination (without generating email) +===== . Clone or checkout the latest copy of the https://pagure.io/releng[releng git repository] . Run the `mass-tag.py` script (requires koji kerberos authentication) @@ -289,3 +272,4 @@ _devel-announce@lists.fedoraproject.org_ list ____ For the contents use this https://pagure.io/releng/blob/main/f/mail-templates/02-mass-rebuild-finished.txt[template]. ____ +===== \ No newline at end of file