Just logging this here.
This morning I woke up with over 450 000 (no, it's not a typo) error emails from src.stg. They all concerned a celery task raising a "rpms/python2.111.git': Permission denied" from an api_new_project call apparently. I went into the host, that git repository doesn't exist. So maybe the permission denied was into trying to create it.
I ended up doing systemctl restart httpd pagure\* and that stopped the flow of emails.
systemctl restart httpd pagure\*
From a quick glance the permissions on the directory structure looked fine, but I didn't check carefully (didn't check the SELinux logs for example).
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, low-gain
Ugh. Can we do something to stop sending you those emails?
I think this has to do with work @amedvede was doing. It had been blocked until recently when I got fedora-messaging working correctly there again. See:
https://stg.pagure.io/releng/fedora-scm-requests/issues?status=Closed
so, ticket there emits message, toddler sees it and tries to do something.
It's configured in pagure's config file :)
I think this has to do with work @amedvede was doing. It had been blocked until recently when I got fedora-messaging working correctly there again. See: https://stg.pagure.io/releng/fedora-scm-requests/issues?status=Closed so, ticket there emits message, toddler sees it and tries to do something.
Yeah the timing would fit with the fix to staging starting to send messages.
There may be a real issue to be fixed in src.stg as it shouldn't throw that permission denied on trying to create a repo.
PR to adjust the emails: https://pagure.io/fedora-infra/ansible/pull-request/3046
I think the permissions thing is something that @james fixed in prod but we never fixed in stg.
The pagure user is using a local/low uid, but the files are all owned by a ipa high uid. So we need to either chown all the local files to the low uid user, or change the pagure user to use the ipa one. @james Do you recall what we did here in prod?
The pr is merged/deployed. Still need to sort the permissions issue.
This issue has been migrated to Fedora Forge: https://forge.fedoraproject.org/infra/tickets/issues/13050
Please continue any further discussion there.
Metadata Update from @ryanlerch: - Issue close_status updated to: Migrated to Fedora Forge - Issue status updated to: Closed (was: Open)