rabbitmq alerted this morning:
[09:23:53] <zodbot> PROBLEM - rabbitmq01.phx2.fedoraproject.org/Check queue centos-ci is WARNING: RABBITMQ_QUEUE WARNING - messages WARNING (12023), messages_ready OK (12023) messages_unacknowledged OK (0) consumers OK (0) (noc01)
So, it seems whatever processes those fedora-messaging messages, isn't doing so.
@jimbair @bgoncalv: Is jenkins-continuous-infra.apps.ci.centos.org still subscribed to this queue?
@kevin @bstinson we were trying to do some checks with the FedoraMessaging queue. @msrb is the queue configured again?
The "centos-ci" queue was configured globally in continuous-infra Jenkins. Although Jenkins was technically consuming messages from the queue, no job was actually using them as it doesn't make sense practically (one Jenkins-global queue and doing round-robin between different jobs).
I've enabled the queue in global config again so the alert should hopefully go away.
It doesn't seem to be:
RABBITMQ_QUEUE WARNING - messages WARNING (24480), messages_ready OK (24480) messages_unacknowledged OK (0) consumers OK (0)
Should we just delete this queue?
Metadata Update from @cverna: - Issue priority set to: Waiting on Assignee (was: Needs Review)
@kevin the queue is still useful for occasional experiments. I'd keep it for now, if possible.
Is it possible to set TTL on the queue so old messages would get discarded automatically after a week or so? Would that help?
I am not sure. possibly? @abompard thoughts?
Yeah the rabbit/queue role supports TTLs : https://pagure.io/fedora-infra/ansible/pull-request/88
Thanks @kevin and @abompard :-)
I'm not sure how to apply this with the IAD move. Should I run rbac-playbook on the new batcave? Probably not since it applies to the old servers?
rbac-playbook
For now, yes, use batcave01.phx2 and rbac-playbook.
We may switch to batcave01.iad2, but we will announce that...
This was done a while back.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.