Update Koji SOP

Great start. I like what is there so far. :)

I'd like to flesh out the prod section some more.

In prod builders are usually done per bvmhost at a time:

• Find all the builders on the host (you can look in /var/log/virthost-lists.out on batcave01, or look at their vmhost: ansible variable)
• in a loop, disable all those builders using 'koji disable-host '
  (note we should add to the top that you have to be a koji admin to do this process
• in a loop check for running builds on them with 'koji list-tasks --host '
• Either wait for all of them to finish, or optionally 'koji free-task NNNNN' to free a task and another builder will pick it up. (Note howver this restarts the build and can annoy maintainers if it's a long running build).
• Once they are all empty, destroy them as above and reinstall them with the buildvm playbook
  (optionally this is a good time to update the bvmhost and reboot it after destroying but before installing builders)
• after installed, check that each builder is checking in with koji list-hosts | grep vmname>
• in a loop reenable them

Some special info:

buildvm-s390x-01 to buildvm-s390x-14 are zvm instances, they have to be updated with dnf --releasever N distro-sync, they cannot be re-installed easily.

I think thats pretty much it, can you merge that into the pr and I'll review again and see if I can think of anything I forgot?

Hey @kevin, I updated the PR, but got some doubts:
* On 3rd and 4th step what "in a loop" means? Because in the beginning, it is said that it should be done one host at a time.

• I couldn't find buildvm playbook in ansible repository, is it somewhere else?

• On 6th step, when you say "destroy them as above and reinstall the builders" it is done like in staging? Running virt-inst-dest?

• What do you think of replacing "N" in this example dnf --releasever N distro-sync for something like and the same on 5th step with this example koji free-task NNNNN, just to keep the pattern?

Hey @kevin, I updated the PR, but got some doubts:
* On 3rd and 4th step what "in a loop" means? Because in the beginning, it is said that it should be done one host at a time.

Yeah, so by that I mean:

for i in seq -w 01 35
do
echo $i
koji disable-host buildvm-s390x-$i.s390.fedoraproject.org
done

So, it will disable each host in the range 01 to 35... one at a time.

• I couldn't find buildvm playbook in ansible repository, is it somewhere else?

Should be in playbooks/group/buildvm.yml

• On 6th step, when you say "destroy them as above and reinstall the builders" it is done like in staging? Running virt-inst-dest?

Yes. You can use virt-inst-dest.

• What do you think of replacing "N" in this example dnf --releasever N distro-sync for something like and the same on 5th step with this example koji free-task NNNNN, just to keep the pattern?

Sure, that works. or perhaps NAME to make it clear it's a vaiable?

Hey @kevin, added the changes.

Is there anything more we should add or fix in that SOP?

I think it looks good! Lets merge.