I suspect this is because we were trying to enable in staging https pushing with pagure user tokens...

We should try removing these commit sections: https://pagure.io/fedora-infra/ansible/blob/main/f/roles/distgit/pagure/templates/pagure.cfg#_269 and https://pagure.io/fedora-infra/ansible/blob/main/f/roles/distgit/pagure/templates/pagure.cfg#_282 and see if that gets it working...

Just tried a few things:

• I was able to do a commit (via ssh) to the rawhide branch of the guake package, just now: https://src.stg.fedoraproject.org/rpms/guake/commits/rawhide

@pingou AUIU, the issue here is not being able to push to a fork, pushing to a main repo works apparently...

I can't test this out though, as every time i try to make a fork, i get the error:

Your task failed: failed to make directory '/srv/git/repositories/forks/ryanlerch': Permission denied

@zlopez this now works for me -- are you able to check and see if it is working for you now?

note that the clone url for SSH should be ssh://<username>@<restoftheuri> not ssh://git@<restoftheui> as is displayed in the UI.

filed a PR here to fix that issue too:

https://pagure.io/fedora-infra/ansible/pull-request/660

@ryanlerch I changed the url for remote
git remote set-url fork ssh://zlopez@pkgs.stg.fedoraproject.org/forks/zlopez/rpms/0ad.git

Checked if the public key is on the src.stg.fedoraproject.org and tried git push fork rawhide.

And I'm still getting

zlopez@pkgs.stg.fedoraproject.org: Permission denied (publickey).
fatal: Could not read from remote repository.

I tried to clone it again to try the clean repo and it seems that I'm no longer able to even clone it.

git clone ssh://zlopez@pkgs.stg.fedoraproject.org/forks/zlopez/rpms/0ad.git just fails with the same error as push.

Maybe just double check the SSH key is set up in:

https://src.stg.fedoraproject.org/settings#nav-ssh-tab

I just created a new fork, and cloned it successfully and pushed back to it:

$ git clone ssh://git@pkgs.stg.fedoraproject.org/forks/ryanlerch/rpms/9wm.git
Cloning into '9wm'...
git@pkgs.stg.fedoraproject.org: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

$ git clone ssh://ryanlerch@pkgs.stg.fedoraproject.org/forks/ryanlerch/rpms/9wm.git
Cloning into '9wm'...
remote: Enumerating objects: 148, done.
remote: Counting objects: 100% (148/148), done.
remote: Compressing objects: 100% (60/60), done.
remote: Total 148 (delta 81), reused 148 (delta 81), pack-reused 0
Receiving objects: 100% (148/148), 17.15 KiB | 5.72 MiB/s, done.
Resolving deltas: 100% (81/81), done.

$ cd 9wm/

$ echo "asdfsafd" >> 9wm.spec     

$ git commit -a -m"test"

$ git push 
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 287 bytes | 287.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Protected namespaces: ['rpms', 'modules', 'container']
remote: Blocking unspecified refs: False
remote: Blacklists: [re.compile('refs/heads/c[0-9]+.*'), re.compile('refs/heads/master')]
remote: User: User: 1350 - name ryanlerch
remote: User groups: {'packager'}
remote: Committer: True
remote: SIG memberships: set()
remote: RCM: False
remote: By-pass PR-only: False
remote: Committer push
remote: Protected namespaces: ['rpms', 'modules', 'container']
remote: Blocking unspecified refs: False
remote: Blacklists: [re.compile('refs/heads/c[0-9]+.*'), re.compile('refs/heads/master')]
remote: User: User: 1350 - name ryanlerch
remote: User groups: {'packager'}
remote: Committer: True
remote: SIG memberships: set()
remote: RCM: False
remote: By-pass PR-only: False
remote: Committer push
remote: Sending to redis to log activity and send commit notification emails
remote: * Publishing information for 1 commits
remote:   - to fedora-message
remote: 2021-06-22 23:09:41,706 [WARNING] pagure.lib.notify: pagure is about to send a message that has no schemas: pagure.git.receive
To ssh://pkgs.stg.fedoraproject.org/forks/ryanlerch/rpms/9wm.git
   187e95a..992ee79  rawhide -> rawhide

I checked the https://src.stg.fedoraproject.org/settings#nav-ssh-tab and the key is already there.

And here is the output:

$ git clone ssh://zlopez@pkgs.stg.fedoraproject.org/forks/zlopez/rpms/0ad.git
Cloning into '0ad'...
zlopez@pkgs.stg.fedoraproject.org: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Just to be sure, I tried to add the SSH key again and it says that the key is already added.

ah!

ll /srv/git/repositories/forks/zlopez/rpms/0ad.git/
total 12
-rw-rw-r--+ 1 pagure 1001 66 Jun 15 12:47 config
-rw-rw-r--+ 1 pagure 1001 73 Jun 15 12:47 description
-rw-rw-r--+ 1 pagure 1001  0 Jun 15 12:47 git-daemon-export-ok
...

The group is wrong for the files there, let me fix them and we can try again.

I've seen this on one when forking a project under my name. So there is something wrong at the FS level (no idea what thought).

chown pagure:packager -R /srv/git/repositories/forks/zlopez/rpms/0ad.git/

Try now?

It's possible that I'm just not in the correct groups.

These are the groups I'm part of on staging:

@pingou Tried it again, but same output.

I tried it again today and I was able to push the changes. Maybe it just took some time to reflect the changes to group ownership.

I tried it inside vagrant machine and it failed again.

Here is what I did:
1) Generate new pairs of keys ssh-keygen
2) Add the new pub key to https://src.stg.fedoraproject.org
3) Try to pull the fork git clone ssh://zlopez@pkgs.stg.fedoraproject.org/forks/zlopez/rpms/0ad.git

Failed with zlopez@pkgs.stg.fedoraproject.org: Permission denied (publickey).

I even tried direct ssh ssh -vvv pkgs.stg.fedoraproject.org to check if the key is correct and the SHA-256 is same.

I will try it again in a few hours, just to check if there isn't some delay between adding the key to src.stg.fedoraproject.org and trying to use it with pkgs.stg.fedoraproject.org.

Today I added the new key to noggin to check if this helps.

After adding the key I tried to do the same things as before:
1) Try to pull the fork git clone ssh://zlopez@pkgs.stg.fedoraproject.org/forks/zlopez/rpms/0ad.git
2) ssh zlopez@pkgs.stg.fedoraproject.org

Both failed with zlopez@pkgs.stg.fedoraproject.org: Permission denied (publickey).

This looks like there is something wrong when adding new ssh key on staging.

I did one more test in separate VM with F34.

I tried this:
1) Generate new pairs of keys ssh-keygen
2) Add the new pub key to https://src.stg.fedoraproject.org
3) Try to pull the fork git clone ssh://zlopez@pkgs.stg.fedoraproject.org/forks/zlopez/rpms/0ad.git

Failed with zlopez@pkgs.stg.fedoraproject.org: Permission denied (publickey).

So I can confirm that there is something wrong with adding new key on staging.

I will try once again with 2048 bit key instead of 3072 bit, which is generated by ssh-keygen by default now, to check if the length is not the issue.

Changing the length of the key didn't helped. I will try again after some time, so I'm sure the change was propagated.

Same output after 30 minutes.

I tried to reproduce this situation on production.

Here is what I did step by step:
1) Generate new pairs of keys ssh-keygen
2) Add the new pub key to https://accounts.fedoraproject.org (I can't see the option to add it to src.fedoraproject.org)
3) Try ssh zlopez@pkgs.fedoraproject.org

I got the same issue as on staging zlopez@pkgs.fedoraproject.org: Permission denied (publickey).

I tried the same with the key, that was added to https://accounts.fedoraproject.org previously and the authentication worked. So it seems that this affects only the newly added SSH keys, but it affects them even on production.

I tried it again now, and the new SSH key is working on both staging and production. I'm not sure where the root of this issue was, but it's now gone.