See also: https://github.com/FrostyX/fedora-review-service/issues/42

I was looking at the pod today and there isn't anything helpful in the logs. So I'm not sure what actually causes bugzilla2fedmsg to stop processing.

Yeah, it's pretty odd. It used to be pretty reliable, but now it seems to get stuck pretty often. ;(

Perhaps @abompard has some ideas here?

I have restarted bugzilla2fedmsg with debug logging, hopefully it'll tell us what's going on next time it's stuck

I've had to restart it a few times... in the logs it looked like it was getting heartbeat messages back and forth, but nothing else. ;(

So, I don't think the debugging as it is now is going to tell us much.

This is happening again :-/
Can anybody poke it, please?

Done. Would it help if we lowered the threshold for the nagios alert?

I think so. I am not sure what the nagios alert threshold is but when we talked about this issue the last time, it occurred multiple times a day, so I guess nagios needs to check in the span of hours?

Right now it's set in roles/nagios_client/templates/check_datanommer_history.cfg.j2

command[check_datanommer_bugzilla]={{libdir}}/nagios/plugins/check_datanommer_timesince.py bugzilla 86400 259200

so, warning is 24 hours and critical is 72 hours.

Note that it's a balancing act because if we set it to something too small, there could be an actual gap in bug changes, but if we set it to high we miss when they stop flowing.

Do you think we can solve this caveman style and set a timer to reboot the Bugzilla2fedmsg service every hour? At this point, it sounds like the least painful solution to me

I restarted it again today, so I think the reboot every hour would be probably good solution.