I'm writing up acls for the new rdu3 network these are going to, and I need some input from the copr team. ;)

@praiskup @frostyx

My long ago understanding of how copr hypervisors are used was that you ssh to the hypervisor with it's ipv4 address, then start / provision the build vm and then that vm comes up with a ipv6 address and you connect to it from backend and manage it via ipv6. Is that still the case? How do you assign the ipv6 addresses? Is there a pool or ?

For backups to the storinator, do you just ssh directly to it via ipv4 ? or some other path?

Once I get these acls to the networking folks and they get them setup, I can get those 3 power9's up and we can use them to test with... they are already there, just need reinstalled and setup on the right vlan.

Is that still the case?

I think so, yes

How do you assign the ipv6 addresses? Is there a pool or ?

There is some range of reserved IP addresses that Fedora Infra told us they won't use for anything else.

I think this is our IPv6 prefix 2620:52:3:1:dead:beef:cafe:c, see https://pagure.io/fedora-infra/ansible/blob/fdfb14225bddc282fb182e57c08c51fcaa72b31e/f/roles/copr/backend/templates/provision/libvirt-new#_394

For backups to the storinator, do you just ssh directly to it via ipv4 ? or some other path?

We use rsnapshot over SSH. The relevant configuration seems to be

rsnapshot_push:
  server_host: storinator01.rdu-cc.fedoraproject.org
  backup_dir: /srv/nfs/copr-be
  cases:
    copr-be-copr-user:
      user: copr
      rsync_args: -H --relative /var/lib/copr/public_html --info=progress2
      command: rsnapshot_copr_backend
      timing_plan: copr_be
      # copr-backend's copr/resalloc pubkey
      ssh_pub_key: ...

Thanks for the info!

Status update:

'fedora-isolated' network has been setup.
I sent an initial set of network acls and such to networking and they have deployed them.
I'm moving forward with trying to bring the 3 power9's online there.

Those machines should allow us to see if things are setup as needed/working.

After that, we are expecting a new machine to replace several old rdu2-cc ones, once that arrives we will get it all provisioned.
Then, we hope to migrate pagure.io at least over to it and then the actual move of hardware/outage will happen.

Currently we are looking at early december for the hardware move.

I mostly have the new copr power9's setup. Status update in the upstream issue: https://github.com/fedora-copr/copr/issues/3786#issuecomment-3403103822

Almost done with those. Then need to wait for new machine to help with the move.

new machine didn't happen, so I repurposed one from the iad2 move.

I moved stg.pagure.io to it today.
Tomorrow we have an outage to move pagure.io to it.
Next monday is the main move of hardware/outage.

pagure.io was moved yesterday.

On monday 2025-12-08 we need to:

at 12UTC:
* disable proxy03 and proxy14 in dns
* set 24h downtime? or 48h?
* virsh shutdown any of the following vm's:

vmhost-x86-cc01.rdu-cc.fedoraproject.org:proxy03.fedoraproject.org:running:1
vmhost-x86-cc01.rdu-cc.fedoraproject.org:smtp-auth-cc-rdu01.fedoraproject.org:running:1
vmhost-x86-cc03.rdu-cc.fedoraproject.org:download-cc-rdu01.fedoraproject.org:running:1
vmhost-x86-cc03.rdu-cc.fedoraproject.org:noc-cc01.rdu-cc.fedoraproject.org:running:1
vmhost-x86-cc03.rdu-cc.fedoraproject.org:proxy14.fedoraproject.org:running:1
vmhost-x86-cc03.rdu-cc.fedoraproject.org:smtp-mm-cc-rdu01.fedoraproject.org:running:1

• also 'virsh autostart --disable' them.
• shutdown -h:

inventory/hardware:vmhost-x86-cc01.rdu-cc.fedoraproject.org
inventory/hardware:vmhost-x86-cc02.rdu-cc.fedoraproject.org
inventory/hardware:vmhost-x86-cc03.rdu-cc.fedoraproject.org
inventory/hardware:vmhost-x86-copr01.rdu-cc.fedoraproject.org
inventory/hardware:vmhost-x86-copr02.rdu-cc.fedoraproject.org
inventory/hardware:vmhost-x86-copr03.rdu-cc.fedoraproject.org
inventory/hardware:vmhost-x86-copr04.rdu-cc.fedoraproject.org
inventory/hardware:vmhost-p09-copr01.rdu-cc.fedoraproject.org
inventory/hardware:storinator01.rdu-cc.fedoraproject.org
and
retrace03.rdu-cc.fedoraproject.org

• Mention in internal slack that everything is shutdown and ready to move.

All hosts down and proxies should be disabled in DNS.

ok, things are moved but there's still work to clean things up. :)

1. vmhost-x86-iso02 (formerly vmhost-x86-cc01), vmhost-x86-copr01/03 and vmhost-x86-iso03 (formerly vmhost-x86-cc02) all need 10G connections sorted out. Datacenter folks plan to work on that tomorrow morning.

2. vmhost-x86-copr02 may need a service intervention. It doesn't see it's 10G card (which I swear exists) and it won't apply firmware updates. ;(

3. storinator01 is moved, but I can't seem to login to the admin interface. ;( We may need to get someone to do a reset on it.

4. smtp-auth-iso01 (formerly smtp-auth-cc-rdu01) is redeployed, but I need to get the saslauth passwords off the old one (which is on vmhost-x86-iso02). Or we need to reissue passwords to all the users of it.

5. smtp-mm-iso01 (formerly smtp-mm-rdu01) needs to still be re deployed.

Here's today's status:

• vmhost-x86-iso02/03 are up and online. I didn't reinstall them, just reconfigured network and ran ansible over them. Just ran out of time for reinstalling.

• vmhost-x86-copr01: DC folks have 10G connections to the machine, but it shows no link still. Will need more investigation.

• vmhost-x86-copr02: lspci on the host shows a 10G card. DC ops confirm it's there and has fiber to it, but the OS does not see it at all. Might be worth trying to pxeboot and do a rhel10 install to see if it's a rhel8 driver issue somehow. (however, the drac also doesn't show it).

• vmhost-x86-copr03/04: ready to be reinstalled.

• storinator01: I can't get into mgmt interface, and there was no way DC ops could see to reset it. I guess we should see if we can work with them to boot a fedora live media and use ipmitool locally to reset the mgmt password.

• smtp-auth-iso01 is up and I copied certs and saslauthdb from the old one. I don't have a easy/good way to test this tho...

• smtp-mm-iso01 is done.

• proxy03/14 are installed and setup (proxy03 I am still ansiblizing, but should be done soon).

So the outstanding things next week would be reinstalling copr machines (work with copr team), and working with DC ops/networking to figure out storinator and those copr machines with issues above.

I've dug into the 10G links, and made no progress. The 01 links are down no matter what I try (even in the iDRAC), and the card doesn't even show up in the 02 iDRAC.

03/04 seem fine, I've reinstalled 03 with RHEL10 and working with @praiskup on setting that up.

storinator is waiting for onsite help with a livecd.

Update before I wrap:

• I worked with RDU3 to try the card from copr03 in 01 and 02. 01 came up, 02 did not - but now that the working card is back in 03 it also does not come up. Probably need to do more networking checks, and we need at least one new card.
• copr04 is done, but Pavel says it didn't reboot. However it seems fine to me, need to recheck with Copr folks
• Storinator01 had it's BC pw reset, but is getting high packet loss (40-50%) which makes the BMC near unusable.

So, current status:

• 2 of the copr machines (01/02) their network cards don't work. So, we are engaging warentee work to replace / fix them.

• The storinator turns out to be... not our storinator. ;( The wrong machine was shipped. Luckily the real machine is still in rdu2 and will move on feb 9th hopefully.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/infra/tickets/issues/12818

Please continue any further discussion there.