From what I can tell, the only recent change related to copr_all_instances_aws is:
https://pagure.io/fedora-infra/ansible/c/72f4323fa81cd8444d3ffdf39b9c36236f5e40aa

I tried to re-run the playbook, but it fails:

RUNNING HANDLER [Restart dhcpd] ************************************************************************************************************************************************************************************************************
Tuesday 16 September 2025  08:34:41 +0000 (0:00:02.085)       0:35:26.068 ***** 
Tuesday 16 September 2025  08:34:41 +0000 (0:00:02.085)       0:35:26.067 ***** 
fatal: [noc01.rdu3.fedoraproject.org]: FAILED! => {"changed": false, "msg": "Unable to restart service dhcpd: Job for dhcpd.service failed because the control process exited with error code.\nSee \"systemctl status dhcpd.service\" and \"journalctl -xeu dhcpd.service\" for details.\n"}
RUNNING HANDLER [Restart nagios] ***********************************************************************************************************************************************************************************************************
Tuesday 16 September 2025  08:34:41 +0000 (0:00:00.355)       0:35:26.423 ***** 
Tuesday 16 September 2025  08:34:41 +0000 (0:00:00.355)       0:35:26.423 ***** 
changed: [noc02.fedoraproject.org]
PLAY RECAP *********************************************************************************************************************************************************************************************************************************
noc01.rdu3.fedoraproject.org : ok=196  changed=10   unreachable=0    failed=1    skipped=213  rescued=0    ignored=0 

I'm not sure if dncpd is expected to be fixed, or if that's a mistake. But nagios itself wasn't restarted on that box.

Ok, I restarted Nagios on noc01.rdu3.fedoraproject.org manually, and it did not help.

FTR: Copr tracker: https://github.com/fedora-copr/copr/issues/3878

DHCPD failed due to a missing semi-colon in ansible/ffeea611681865a08c5803f229a801d067c42eab - I've fixed it by hand, restarted the service, and opened https://pagure.io/fedora-infra/ansible/pull-request/2842 to fix permanently.

Not sure whats going on with Nagios though, will take a look.

Hm, Copr doesn't seem to be monitored - does it affect other services?

OK, I've done some digging.... right now the copr_back_aws hostgroup is indeed empty on Nagios, as are quite a few other groups.

noc01$ cat nagios/hostgroups/all.cfg
...
define hostgroup{
  hostgroup_name copr_back_aws
  alias          copr_back_aws
}
...

I think that this commit is causing the problem - we changed the logic on how to decide if a host should be included to look at the datacenter variable

The copr hosts have datacenter: aws while noc01 has rdu3 and noc02 has ibiblio so it's not clear to me that any of the AWS hosts can be properly monitored right now. @kevin have I missed anything in that analysis?

The fix isn't clear either:
- I'm not sure what that change was trying to accomplish, so I can't determine a better way to handle it
- We're (hopefully) moving to Zabbix soon(tm) so perhaps I should prioritise AWS hosts for the next monitoring phase?

@kevin I'd appreciate your thoughts here.

Yeah, this was from when we were moving datacenters and didn't want the new noc01.rdu3 to monitor iad2/external things...

I think we can revert the part of that that added datacenter into the mix.

https://pagure.io/fedora-infra/ansible/pull-request/2855 opened for review

OK, took a little effort but I see hosts in the hostgroup again: https://nagios.fedoraproject.org/nagios/cgi-bin//status.cgi?hostgroup=copr_all_instances_aws&style=detail

I'm not clear on whether the ping failures are (a) because of the DC move, when these hosts disappeared, or (b) some misconfiguration due to the changes I made to Nagios. @praiskup @kevin any thoughts?

The issues were due to:

• needing to allow the new nagios ip in security groups
• needing to restart nftables (nagios ips updated, but nftables not restarted for some reason)
• needing to add some icmp allowed from noc to security groups

I fixed most of them, but:

copr-be-dev.aws.fedoraproject.org has a disk space alert. ;) DISK CRITICAL - free space: /var/lib/copr/public_html 8350 MiB (3.44% inode=90%):

logdetective01.fedorainfracloud.org is apparently using firewalld and doesnt have our nftables rules to allow noc

@frostyx / @praiskup could you fix those? :)

Otherwise I think we are all back on track...

copr-be-dev.aws.fedoraproject.org has a disk space alert. ;) DISK CRITICAL - free space: /var/lib/copr/public_html 8350 MiB (3.44% inode=90%):

Fixed.

logdetective01.fedorainfracloud.org is apparently using firewalld and doesnt have our nftables rules to allow noc

Reported against logdetective: https://github.com/fedora-copr/logdetective-website/issues/304

Thank you @gwmngilfen and @kevin for fixing this!