Yes, every channel is getting these. We need to figure out where it's getting crossed and fix it.

The problem is the fedmsg policy on busgateway01.stg.

However, I have no idea why it's now wrong... it has to be some change in ansible recently. ;(

This seems to be fixed in #fedora-admin, do you still have the issue in -neuro?

Seems to be fixed there too. Thanks! Closing.

Seems to have started again today:

Yeah, we need to figure out why the config is going wrong here and sending all the prod messages to staging. ;(

ok. I think I have this fixed. It was a fun road. :)

In phx2 datacenter we had a firewall rule on all prod machines that would drop everything from stg. In iad2, we didnt do this because there's firewall between the various vlans and we didn't think we needed it. However, there's some things (like the fedmsg ports) that are open accross all vlans, so fedmsg is able to send / connect accross stg and prod.
Additionally, since time began fedmsg in stg actually had all the prod servers in it too. They just didn't work because of the firewall rule. Once that was gone everything actually connected and messages started flowing.
Additionally, the fedmsg/base role was disabled for a number of staging hosts because of the sysadmin group being not there. (This was pre noggin deployment). Which meant these hosts just always kept the initial fedmsg config and running the playbooks did nothing to change it.

I fixed the ansible fedmsg filter plugin to only put staging hosts in stg policy.py files.
I fixed playbooks to run fedmsg/base on staging hosts.
I ran all the playbooks to make sure everything synced up.

I think things are back to ok now.