#11778 Planned Outage - koji upgrade - 2024-02-21 21:00 UTC
Closed: Fixed a year ago by kevin. Opened a year ago by kevin.

Planned Outage - koji upgrade - 2024-02-21 21:00 UTC

There will be an outage starting at 2024-02-21 21:00 UTC,
which will last approximately 3 hours.

To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:

date -d '2024-02-21 21:00UTC'

Reason for outage:

koji will be upgraded to 1.34.0, which requires a schema update that touches many rows. We estimate this will take about 45minutes to complete and during that time, koji will be completely offline.
Package maintainers are advised to not start any long term builds before the outage.

Affected Services:

  • koji
  • bodhi

Ticket Link:

https://pagure.io/fedora-infrastructure/issue/11778

Please join #fedora-admin or #fedora-noc on irc.libera.chat
or #admin:fedoraproject.org / #noc:fedoraproject.org on matrix.
Please add comments to the ticket for this outage above.

Updated status for this outage may be available at
https://www.fedorastatus.org/


FWIW, my plan for db01 is:

  • I will be making a new db02 vm. rhel9 / default postgresql
  • I will rsync all /var/lib/pgsql/data from db01 to db02 (and possibly also to some other location to have a panic backup)
  • I will rsync again before the outage window
  • at outage, take down db01's postgresql
  • Do another rsync.
  • take down db01, save it's storage and virt xml.
  • on db02 to a pg_upgrade to the default rhel9 version.
  • on db02 upgrade postgresql to 15, do another pg_upgrade
  • rename db02 to db01, switch it back to the db01 ip.
  • bring it back up.

(I did this with db-koji01 a while back just fine).

I am going to do this process now with staging.

It could be wise to take down bodhi completely before taking down Koji, as failing to tag builds will cause out of synch statuses between updates status and build tags.

I'm going to change this outage to just be the koji update.

Thinking about it we are getting some new hardware soon, and I want to wait for that to be setup for the new rhel9 db01. I could set it up on existing hardware, but then would have to move it, etc. So, just putting it on the new hardware in the first place makes more sense. ;)

Yeah, I can scale bodhi down before koji is taken down, thanks for the input!

Metadata Update from @phsmoura:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

a year ago

Outage is over.

I did try and do a vacuum full on the db, but the builrdoot table is gigantic and it was taking a really really long time, so I ended up just canceling.
We can try again sometime.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a year ago

Log in to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog