I just tried this and it worked ... but I had to create a new API key (I'm guessing my last one expired years ago).

Can you make sure your fedpkg API key is fine?
Can you see if fork button on the website works?

I just tried this and it worked ... but I had to create a new API key (I'm guessing my last one expired years ago).

I also get 404 accessing https://src.fedoraproject.org/fork/mikelo2/rpms/vultr-cli and "fatal: '/srv/git/repositories/forks/mikelo2/rpms/vultr-cli.git' does not appear to be a git repository" message using git.

Can you make sure your fedpkg API key is fine?

I tried refreshing API keys and issue persist.

Can you see if fork button on the website works?

main dist-git shows "view fork" and when clicking it reports 404 error.

Likely related, a real new package request has failed the same way:

https://src.fedoraproject.org/rpms/python-pyrankvote

https://pagure.io/releng/fedora-scm-requests/issue/77151

The only mention of it I see in the logs is:

Jul  5 09:21:24 pkgs01 celery[2178639]: 2025-07-05 09:21:24,490 [ERROR] celery.app.trace: Task pagure.lib.tasks.create_project[74e82cee-80c9-4eba-a8f9-e2b8cfcb188b] raised unexpected: DatabaseError('(psycopg2.DatabaseError) could not receive data from ser
ver: Connection timed out\n',)
Jul  5 09:21:24 pkgs01 celery[2178639]: Traceback (most recent call last):
[...SNIP...]
Jul  5 09:21:24 pkgs01 celery[2178639]: sqlalchemy.exc.DatabaseError: (psycopg2.DatabaseError) could not receive data from server: Connection timed out
Jul  5 09:21:24 pkgs01 celery[2178639]: [SQL: SELECT [...SNIP...] FROM projects WHERE projects.name = %(name_1)s AND projects.namespace = %(namespace_1)s AND projects.is_fork = false]
Jul  5 09:21:24 pkgs01 celery[2178639]: [parameters: {'name_1': 'python-pyrankvote', 'namespace_1': 'rpms'}]
Jul  5 09:21:24 pkgs01 celery[2178639]: (Background on this error at: http://sqlalche.me/e/4xp6)

Looks like a timeout reaching the database server, which is db01.rdu3.fp.o.

Same error for vultr-cli. I think it's the reason. There are 204 such tracebacks in the logs for that day (July 5th). That's quite a lot. Something happened?

There are 148 such errors yesterday, and 89 today. :-(

I just tried this and it worked ... but I had to create a new API key (I'm guessing my last one expired years ago).

Can you make sure your fedpkg API key is fine?
Can you see if fork button on the website works?

The API key is fine. The fork button doesn't exist, it's changed to "View fork". If i rerun $ fedpkg fork, I get the message that the fork already exists.

I've hacked the pagure tasks to retry on DatabaseError after 10s, for now it seems to work, the tasks are retried and succeed.
I'm still investigating the root cause with more logging on celery.

ok, how can we fix the broken forks? Looks like we can use pagure-admin to delete them and ask people to re-fork/re-request.

I fixed python-pyrankvote by doing this. (deleted, then retried the creation).

I have deleted the following forks:

--user cstratak rpms/python-rasterio
--user mikelo2 rpms/vultr-cli
--user churchyard rpms/rit-keraleeyam-fonts
--user yselkowitz rpms/NetworkManager

@cstratak @mikelo2 @churchyard @yselkowitz
Can you all refork those projects and see if they work for you now?

I am not sure how to identify the broken ones where people didn't tell us, perhaps we can come up with a way to find all forks and check that they have git repos?

Can you all refork those projects and see if they work for you now?

user churchyard rpms/rit-keraleeyam-fonts works.

Can you all refork those projects and see if they work for you now?

I've been able to fedpkg fork && git push.

Things work for me now, thanks!

OK I have a possible explanation: in pagure/lib/task_utils.py line 34, the database session pool is created with the default connection lifetime, which is 1 hour. I have forced this lifetime down to 10 minutes, and I don't see any DatabaseError anymore. I also set it to 30 minutes and there were still connection errors.
While I did check on PostgreSQL's config that no connection TTL had been set, I can only guess that something in the middle is expiring the connections after some time between 10 and 30 minutes.
I'm going to send a PR to Pagure to hopefully get this change upstream.
If anyone has a better idea of what's going on, feel free to chime in here!

I am 98.7% sure it's a setting in the firewall between vlans. CentOS folks ran into a timeout like this with rabbitmq access, and the 30min timeframe sounds right.

So, I could ask them to increase this limit? Or do we just want to adjust on our side ?

That definitely seems to have fixed it.

There are possibly more broken ones, but we are moving to forgejo soon anyhow... and as people tell us about broken ones we can fix them, so I am going to close this now.