The most recent builds on have been failing on s390x with No route to host exceptions. See for example the s390x task for this build
ASAP
CCing. Still s390x builder seems unhealthy as of 2021/12/19 21:43 JST(UTC+9), e.g. https://koji.fedoraproject.org/koji/taskinfo?taskID=80184429
Would someone investigate this?
Seeing this as well...
https://koji.fedoraproject.org/koji/taskinfo?taskID=80185813
looks to me the rpm cache VM (kojipkgs-cache01.s390.fedoraproject.org) died ...
Or more precise, all the KVM guests seems to be down, last check in in koji is "2021-12-19 08:16" for builder vms 15..23
Everyone in Fedora IT is on PTO so please be patient as this may require physical fixing. The s390x is not responding and I do not have access to its x3720
Sharkcz found the hidden lever and has restarted the ancient z15.
I'm still seeing errors on s390x: https://koji.fedoraproject.org/koji/taskinfo?taskID=80192334
Turns out that there is a hardware issue with the s390x and we are having to open a ticket with Red Hat Engineering to see if they can fix it.
In the mean time I have moved the cache host to be one of the ones in the z/vm lpar.
So, builds should start working again soon here.
Builds appear to be working with the cache host moved. I don't know if you want to keep this open to track fixing the other stuff.
The kvm virthost/hypervisior is back online. :)
I'll leave this open to track moving the caching back (or making it just use them both).
Metadata Update from @mohanboddu: - Issue assigned to kevin - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble, ops
I've setup both now, this should help us anytime one of the lpars becomes unavailable for some reason.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.