@pingou can you check this, pls?

Looks like loopabull was no longer running...

It's now running again and processing its backlog which isn't small (60k+ messages), so that'll take a while before it recovers fully.

So, the thing is that the box had only 1 CPU and somehow the service had not been running for a while apparently.
It thus accumulated a backlog of messages to go through, backlog big enough than the 5 workers on 1 CPU were not able to catch up with it.

I've rebuilt the host so it has 2 CPU (woohoo!), with the assumption that rabbitmq saves its queues to disk and thus there would be no data loss.
Apparently, I've missed something as in the rebuild of the host, I've lost this queue and thus the entire backlog :(

@jbaird kindly restarted this job to see if with the new host things are still working fine and looking at the PR it seems they are.

However, in order to be more flexible in the future, I'd like to bump the box to 4 CPUs and rebuild it again.
This will cause another short outage and then I think we should be fine for a little bit.

Sorry for the inconvenience :(