From 42d42428cf6ef2c8951cc58b930fd34633e6532d Mon Sep 17 00:00:00 2001 From: Pedro Moura Date: Apr 25 2022 15:54:49 +0000 Subject: [PATCH 1/3] updating koji builder sop Signed-off-by: Pedro Moura --- diff --git a/modules/sysadmin_guide/pages/koji-builder-setup.adoc b/modules/sysadmin_guide/pages/koji-builder-setup.adoc index cff98b2..81c5a5e 100644 --- a/modules/sysadmin_guide/pages/koji-builder-setup.adoc +++ b/modules/sysadmin_guide/pages/koji-builder-setup.adoc @@ -2,6 +2,38 @@ == Contents +* <<_upgrade_builders>> +** <<_staging_environment>> +** <<_production_environment>> + +== Upgrades Builders + +It is recommended to upgrade or reinstall all builders after each release. To do it, usually the builders are upgraded and virtual machines reinstalled. First they should be upgraded in staging environment to make sure everything is working properly before starting on production. + +=== Staging Environment + +Follow the process to upgrade builders in staging environment: + +. Update the virt-install path on the builders in ansible repository, like in this https://pagure.io/fedora-infra/ansible/c/af94db8ab88588aa8dc2c20893b2487afb7f7777?branch=main[example] +. Each builder virtual machine is removed, either via virt-inst-dest or manually, and the buildvm.yml playbook is run to deploy them. +. The hub is re-installed or upgraded with 'dnf --refresh --releasever distro-sync'. +. Test builds are done to make sure everything is working. + +=== Production Environment + +For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. + + +//// +From this comment below, everything is going to be deleted +Keeping all commented for reference while writing a new SOP +//// + +//// += Setup Koji Builder SOP + +== Contents + * <<_builder_setup>> ** <<_network_overview>> ** <<_setup_overview>> @@ -117,3 +149,4 @@ virsh autostart $builder_fqdn ** disable selinux in `/etc/sysconfig/selinux` ** `reboot` ** ask a koji admin to re-enable the host +//// \ No newline at end of file From b8a08cdfb59cd9891a70de48d9598a8ec056ac9c Mon Sep 17 00:00:00 2001 From: Pedro Moura Date: Apr 25 2022 15:54:49 +0000 Subject: [PATCH 2/3] added production section Signed-off-by: Pedro Moura --- diff --git a/modules/sysadmin_guide/pages/koji-builder-setup.adoc b/modules/sysadmin_guide/pages/koji-builder-setup.adoc index 81c5a5e..ab4038e 100644 --- a/modules/sysadmin_guide/pages/koji-builder-setup.adoc +++ b/modules/sysadmin_guide/pages/koji-builder-setup.adoc @@ -15,13 +15,24 @@ It is recommended to upgrade or reinstall all builders after each release. To do Follow the process to upgrade builders in staging environment: . Update the virt-install path on the builders in ansible repository, like in this https://pagure.io/fedora-infra/ansible/c/af94db8ab88588aa8dc2c20893b2487afb7f7777?branch=main[example] -. Each builder virtual machine is removed, either via virt-inst-dest or manually, and the buildvm.yml playbook is run to deploy them. -. The hub is re-installed or upgraded with 'dnf --refresh --releasever distro-sync'. +. Each builder virtual machine is removed, either via `virt-inst-dest` or manually, and the `buildvm.yml` playbook is run to deploy them. +. The hub is re-installed or upgraded with `dnf --refresh --releasever distro-sync`. . Test builds are done to make sure everything is working. === Production Environment -For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. +For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. From _buildvm-s390x-01_ to _buildvm-s390x-14_ are **zvm** instances, they have to be updated with `dnf --releasever N distro-sync`. +In production builders, are usually done per bvmhost at a time: + +. To do this process the user should be koji admin +. Find all the builders on the host looking in `/var/log/virthost-lists.out` on batcave01 or look for `vmhost` variable in `inventory/host_vars/` on https://pagure.io/fedora-infra/ansible[ansible repository] +. In a loop, disable all those builders using `koji disable-host ` +. In a loop check for running builds on them with `koji list-tasks --host ` +. Either wait for all of them to finish or optionally run `koji free-task NNNNN` to free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers +. Once they are all empty, destroy them as above and reinstall them with the `buildvm` playbook. Optionally, this is a good time to update the bvmhost and reboot it after destroying, but before installing builders +. After installed, check that each builder is checking in with `koji list-hosts | grep ` +. In a loop, reenable them + //// From 24b185baab6b0a5050004b5d7dea2dccc4123b59 Mon Sep 17 00:00:00 2001 From: Pedro Moura Date: Apr 25 2022 15:54:49 +0000 Subject: [PATCH 3/3] added code as example Signed-off-by: Pedro Moura --- diff --git a/modules/sysadmin_guide/pages/koji-builder-setup.adoc b/modules/sysadmin_guide/pages/koji-builder-setup.adoc index ab4038e..adf183c 100644 --- a/modules/sysadmin_guide/pages/koji-builder-setup.adoc +++ b/modules/sysadmin_guide/pages/koji-builder-setup.adoc @@ -21,143 +21,22 @@ Follow the process to upgrade builders in staging environment: === Production Environment -For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. From _buildvm-s390x-01_ to _buildvm-s390x-14_ are **zvm** instances, they have to be updated with `dnf --releasever N distro-sync`. +For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. From _buildvm-s390x-01_ to _buildvm-s390x-14_ are **zvm** instances, they have to be updated with `dnf --releasever NAME distro-sync`. In production builders, are usually done per bvmhost at a time: . To do this process the user should be koji admin . Find all the builders on the host looking in `/var/log/virthost-lists.out` on batcave01 or look for `vmhost` variable in `inventory/host_vars/` on https://pagure.io/fedora-infra/ansible[ansible repository] -. In a loop, disable all those builders using `koji disable-host ` +. In a loop, disable all those builders using `koji disable-host `. Like the example below: ++ +.... +for i in seq -w 01 35 +do + echo $i + koji disable-host buildvm-s390x-$i.s390.fedoraproject.org +done +.... . In a loop check for running builds on them with `koji list-tasks --host ` -. Either wait for all of them to finish or optionally run `koji free-task NNNNN` to free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers +. Either wait for all of them to finish or optionally run `koji free-task NAME` to free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers . Once they are all empty, destroy them as above and reinstall them with the `buildvm` playbook. Optionally, this is a good time to update the bvmhost and reboot it after destroying, but before installing builders . After installed, check that each builder is checking in with `koji list-hosts | grep ` . In a loop, reenable them - - - -//// -From this comment below, everything is going to be deleted -Keeping all commented for reference while writing a new SOP -//// - -//// -= Setup Koji Builder SOP - -== Contents - -* <<_builder_setup>> -** <<_network_overview>> -** <<_setup_overview>> -** <<_resettinginstalling_an_old_koji_builder>> - -== Builder Setup - -Setting up a new koji builder involves a goodly number of steps: - -=== Network Overview - -[arabic] -. First get an instance spun up following the kickstart sop. -. Define a hostname for it on the 125 network and a $hostname-nfs name -for it on the .127 network. -. make sure the instance has 2 network connections: -* eth0 should be on the .125 network -* eth1 should be on the .127 network -+ -[NOTE] -===== -For VM _eth0_ should be on _br0_, _eth1_ on _br1_ on the vmhost. -===== - -=== Setup Overview - -* install the system as normal: -+ -.... -virt-install -n $builder_fqdn -r $memsize \ --f $path_to_lvm --vcpus=$numprocs \ --l http://10.5.126.23/repo/rhel/RHEL6-x86_64/ \ --x "ksdevice=eth0 ks=http://10.5.126.23/repo/rhel/ks/kvm-rhel-6 \ - ip=$ip netmask=$netmask gateway=$gw dns=$dns \ - console=tty0 console=ttyS0" \ - --network=bridge=br0 --network=bridge=br1 \ - --vnc --noautoconsole -.... -* run python `/root/tmp/setup-nfs-network.py` this should print out the --nfs hostname that you made above -* change root pw -* disable selinux on the machine in /etc/sysconfig/selinux -* reboot -* setup ssl cert into private/builders - use fqdn of host as DN -** login to _fas01_ as root -** `cd /var/lib/fedora-ca` -** `./kojicerthelper.py normal --outdir=/tmp/ \ --name=$fqdn_of_the_new_builder --cadir=. --caname=Fedora` -** info for the cert should be like this: -+ -.... -Country Name (2 letter code) [US]: -State or Province Name (full name) [North Carolina]: -Locality Name (eg, city) [Raleigh]: -Organization Name (eg, company) [Fedora Project]: -Organizational Unit Name (eg, section) []:Fedora Builders -Common Name (eg, your name or your servers hostname) []:$fqdn_of_new_builder -Email Address []:buildsys@fedoraproject.org -.... -** scp the file in `/tmp/$\{fqdn}_key_and_cert.pem` over to _batcave01_ -** put file in the private repo under `private/builders/$dn}.pem` -** `git add` + `git commit` -** `git push` -* run `./sync-hosts` in infra-hosts repo; `git commit; git push` -* as a koji admin run: -+ -.... -koji add-host $fqdnr i386 x86_64 - -(note: those are yum basearchs on the end - season to taste) -.... - -=== Resetting/installing an old koji builder - -* disable the builder in koji (ask a koji admin) -* halt the old system (`halt -p`) -* undefine the vm instance on the buildvmhost: -+ -.... -virsh undefine $builder_fqdn -.... -* reinstall it - from the buildvmhost run: -+ -.... -virt-install -n $builder_fqdn -r $memsize \ --f $path_to_lvm --vcpus=$numprocs \ - -l http://10.5.126.23/repo/rhel/RHEL6-x86_64/ \ - -x "ksdevice=eth0 ks=http://10.5.126.23/repo/rhel/ks/kvm-rhel-6 \ - ip=$ip netmask=$netmask gateway=$gw dns=$dns \ - console=tty0 console=ttyS0" \ - --network=bridge=br0 --network=bridge=br1 \ - --vnc --noautoconsole -.... -* watch install via vnc: -+ -.... -vncviewer -via bastion.fedoraproject.org $builder_fqdn:1 -.... -* when the install finishes: -** start the instance on the buildvmhost: -+ -.... -virsh start $builder_fqdn -.... -** set it to autostart on the buildvmhost: -+ -.... -virsh autostart $builder_fqdn -.... -* when the guest comes up -** login via ssh using the temp root password -** python `/root/tmp/setup-nfs-network.py` -** change root password -** disable selinux in `/etc/sysconfig/selinux` -** `reboot` -** ask a koji admin to re-enable the host -//// \ No newline at end of file