@zlopez could you take a look? I don't understand why it's not unquring those after it sends... or where it keeps getting them. ;(

I restarted the workers, just on the off chance it was a sync issue between one of them and the delivery backend.

Can you let us know in a few hours if you have gotten more of these?

I just got 2 more copies of each message, so 4 total at 17:18 CST, so after the restart

Still getting them this morning, up to 274 notifications now.

After some digging, I found out the reason, why it's being resend by FMN

Jan 16 11:36:02 notifs-backend01.iad2.fedoraproject.org celery[495842]: [2023-01-16 11:36:02][ fmn.tasks    INFO] Found 2 recipients for message 2023-77eacdd6-8d8c-4d7b-8f73-b2a9f9a10f9b
Jan 16 11:36:02 notifs-backend01.iad2.fedoraproject.org celery[495842]: [2023-01-16 11:36:02][ fmn.tasks    INFO] Dispatching messages for 1 recipients for the irc backend
Jan 16 11:36:02 notifs-backend01.iad2.fedoraproject.org celery[495842]: [2023-01-16 11:36:02][ fmn.tasks    INFO] Queuing message for delivery to jforbes.id.fedoraproject.org on the irc backend
Jan 16 11:36:02 notifs-backend01.iad2.fedoraproject.org celery[495842]: [2023-01-16 11:36:02][ fmn.tasks    INFO] Dispatching messages for 1 recipients for the email backend
Jan 16 11:36:02 notifs-backend01.iad2.fedoraproject.org celery[495842]: [2023-01-16 11:36:02][celery.app.trace    INFO] Task fmn.tasks.find_recipients[3522b079-14c4-4d9c-aaf0-b573fd6966c1] retry: Retry in 3600s: ValueError('Header value may not contain linefeed or carriage return characters')

It seems there is some issue with format of the e-mail header.

Looking into FMN code, there is a hardcoded retry for 60 days. I will try to remove the message from the retry queue, obviously it's delivered. Hopefully I will find a way to remove it without clearing the whole queue.

I found out that you can delete message from the rabbitmq queue by using rabbitmq API and curl:

curl -i -u user:password -X POST http://localhost:15672/api/queues/vhost/fmb.tasks.unprocessed_messages/get -d '{"count":1,"ackmode":"ack_requeue_false","encoding":"auto","truncate":50000}'

This needs to be run on notifs-backend01.iad2.fedoraproject.org and it consumes one message from the queue without putting it back.

The issue is that I can't find the user:password for rabbitmq on notifs-backend01.iad2.fedoraproject.org.

I'm also not sure, if this queue is the right one, without user I can't check, but I know how to do it:

curl -i -u user:password -X POST http://localhost:15672/api/queues/vhost/fmb.tasks.unprocessed_messages/get -d '{"count":5,"ackmode":"ack_requeue_true","encoding":"auto","truncate":50000}'

This will just show you the messages and puts them back in queue. The important argument here is ackmode, which controls that.

@kevin Could you point me to user and password for rabbitmq user on FMN machine? I tried the default one for rabbitmq, but got Access refused.

rabbitmqctl list_users

Listing users ...
user tags
guest [administrator]

ie, we have setup no user... there's only 'guest'. It's only listening to localhost I think...

Not sure if anything new has been done, but definitely still getting the messages, up to 526 now.

@kevin I already tried the default guest login and got 401.

You need to pass "http://localhost:15672/api/queues/%2F/fmn.tasks.unprocessed_messages/get"

for / vhost (default).

If I do that I get a 200...but it returns []
;(

I tried a purge_queue on it, but it didn't work...

rabbitmqctl purge_queue fmn.tasks.unprocessed_messages

Purging queue 'fmn.tasks.unprocessed_messages' in vhost '/' ...

rabbitmqctl list_queues --vhost / | grep unp

fmn.tasks.unprocessed_messages 4

Do we need to stop backend to purge those?

I stopped backend and workers and purged the queue and restarted.

@jforbes can you let us know if it stops now?

This does seem to be resolved. Thank you!

Hurray. Sorry for all the trouble here.