There are s390x compose builders, but they are occupied by previous ELN composes, which couldn't finish because of some other issue (lost /mnt/koji?). It's a recurrent problem somewhere in the infra unfortunately.

We have been having sporadic problems with the s390x compose builders being unable to access nfs mounts. These mounts are over sshfs, which has been reliable up until the last few months. ;(

I have fixed them all so they can process the backlog, and I guess I am going to see if we can move from sshfs to a vpn and see if thats any more reliable.

In any case, they are processing now.

They have caught up on all old jobs.

There is a bigger problem about s390x builders loosing /mnt/koji randomly, @kevin is thinking of changing from sshfs to nfs and see what happens.

So, this has been stable again for a bit... I am going to close this now and we can revisit if it happens again.

We can switch to a vpn, but it's going to require opening another port on RHIT firewall and doing some anoying things in ansible, so I would prefer to avoid it if we can.