This still seems to be a problem. Git activity this month hasn't showed up either.

Yeah, I'm seeing something similar...in my activity report for the last week I see things I did to PRs and issues, but not any direct commits I made. I do see commits from several weeks ago. So it seems like this is not being updated properly, somehow.

So I've spent the last day and a half setting up a dev instance that can actually test this stuff, and I've finally got one working...too well! I can't reproduce this not working. I've tried sending commits from primary and secondary emails of the single account in the dev instance, and both are showing up just fine in the activity log.

So, now trying to figure what can be different between the prod instance and my dev instance.

@puiterwijk notes that on the prod instance (this one!), pagure_logcom service started up correctly after the mass reboot infra did yesterday - at 2018-04-04 23:26:54 UTC - then appears to have been operating normally for a while, then at around 2018-04-05 09:24:09 UTC it seems to have stopped logging any activity. It still claims to be running, but isn't logging itself doing anything.

I'm going to see if I can find some commits that happened during that time and see if those show up in the relevant folks' activity logs, and if ones that happened since that time don't.

That does seem to be the case. If you look at @mohanboddu 's activity log for 2018-04-05 (on his page), it shows "mohanboddu pushed 2 commits to fedora-kickstarts". That I believe refers to https://pagure.io/fedora-kickstarts/c/5ce57cff5c70f7e4d6f7827a86608d30809dcbaf?branch=master and https://pagure.io/fedora-kickstarts/c/7a7e9ae8e3f34e7a4355e390b58ebe5c17ac63af?branch=master , both of which happened exactly during the window between the mass reboot and pagure_logcom going dark. However, if you check @dustymabe 's activity log for 2018-04-05 (on his page), it does not show any commits to fedora-kickstarts, despite the fact that he made one: https://pagure.io/fedora-kickstarts/c/13dfe736a031a6165a491d74ef5e60e393769f1c?branch=master . That one happened after pagure_logcom went dark. So the time frames seem to match up well there.

Now I guess we need to figure out why pagure_logcom suddenly stopped doing stuff...

Thanks for the investigation, maybe we should try replicating this in stg.pagure.io since that is running the beta of 4.0 which ported this service to be celery-based instead of relying on redis and trollius.

https://pagure.io/pagure/pull-request/3140 should fix this - at least the case @puiterwijk and I investigated, it is of course hard to be definitively sure that same bug caused the missing activity @andyp noticed, but it seems like a reasonable guess.

To clarify my original bug report, the use case is:

• Make a commit
• Push it to pagure
• Look at user activity page
• See no new colourful squares :(
• Push to github mirror
• See new colourful squares on github :)

The PR looks like it's related to email though, and I don't have non-ascii characters in my name so I can't see how it would be related.

@andyp the thing that calls that email sending function is the same thing that logs the commit into the database that backs the user activity page. When this bug happens, the service which does this for all commits (supposedly) silently hangs (...more or less) while trying to send out the email (for a commit which would cause unicode characters to show up in the email body). That means it never completes logging that commit to the activity database either, and also never fires again until restarted (because it's permanently 'stuck', it will never pick up another commit notification from its queue), so it will not send out email notifications or log to the database for any subsequent commit to the entire Pagure instance, until it's restarted.

Oh I see, that sounds pretty impactful! Thanks for the explanation :thumbsup:

@adamwill I wonder if we should invert the order then, log the commit before sending the notification, just to be safe(r)

For the record, I checked what happens with current git master in my dev env. Turns out that the logging/notification for that particular commit does indeed still fail, as we expected, but logcom handles it better: it doesn't get stuck, and will successfully log/notify any subsequent commits that don't have non-ASCII text. Still, we should fix this. :) #3141 is my alternative fix proposal (just fixes it a bit differently to @puiterwijk 's approach).

Commit a8fef88e fixes this issue

Note, once this is deployed to production (I think @puiterwijk may have already / be going to hotfix it, but I'm not 100% sure), please yell if you still see issues with logging of subsequent commits. Unfortunately I don't believe commits that may have missed being logged due to this bug in the last year or so will get retroactively logged, unless someone writes a script or something to do it :(