Find out why and fix that (:wink:) I can't log into 4 of the 5 x86 builders in staging. I'm in the sysadmin-releng group in staging which should let me log into Koji nodes there, and it works on buildvm-x86-01.stg.iad2 as well as the builders for other arches.
sysadmin-releng
Here's an attempt to log into buildvm-x86-02.stg.iad2 (which fails):
nils@makake:~> ssh -v buildvm-x86-02.stg.iad2.fedoraproject.org OpenSSH_8.5p1, OpenSSL 1.1.1k FIPS 25 Mar 2021 debug1: Reading configuration data /home/nils/.ssh/config debug1: /home/nils/.ssh/config line 3: Applying options for *.fedoraproject.org debug1: /home/nils/.ssh/config line 15: Applying options for *.iad2.fedoraproject.org debug1: /home/nils/.ssh/config line 88: Applying options for * debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: configuration requests final Match pass debug1: re-parsing configuration debug1: Reading configuration data /home/nils/.ssh/config debug1: /home/nils/.ssh/config line 3: Applying options for *.fedoraproject.org debug1: /home/nils/.ssh/config line 15: Applying options for *.iad2.fedoraproject.org debug1: /home/nils/.ssh/config line 88: Applying options for * debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: Executing proxy command: exec ssh -W buildvm-x86-02.stg.iad2.fedoraproject.org:22 bastion-iad01.fedoraproject.org debug1: identity file /home/nils/.ssh/id_rsa-fedora type 0 debug1: identity file /home/nils/.ssh/id_rsa-fedora-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.5 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.5 debug1: compat_banner: match: OpenSSH_8.5 pat OpenSSH* compat 0x04000000 debug1: Authenticating to buildvm-x86-02.stg.iad2.fedoraproject.org:22 as 'nphilipp' debug1: load_hostkeys: fopen /home/nils/.ssh/known_hosts2: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256@libssh.org debug1: kex: host key algorithm: ssh-ed25519-cert-v01@openssh.com debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32 debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: SSH2_MSG_KEX_ECDH_REPLY received debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com SHA256:q0E3CDWDj25/HsV9Ju91aMvmuKqTLvNwujb6VXMmHOM, serial 1604525811 ID "buildvm-x86-02.stg.iad2.fedoraproject.org" CA ssh-rsa SHA256:pmw8O+j5VVfgpVIjMvWG+u7DizO2dyzrvxHWrLUg02s valid from 2020-11-04T21:36:51 to 2021-11-03T22:36:51 debug1: load_hostkeys: fopen /home/nils/.ssh/known_hosts2: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug1: No matching CA found. Retry with plain key debug1: Host 'buildvm-x86-02.stg.iad2.fedoraproject.org' is known and matches the ED25519 host key. debug1: Found key in /home/nils/.ssh/known_hosts:1078 debug1: rekey out after 4294967296 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey in after 4294967296 blocks debug1: Will attempt key: /home/nils/.ssh/id_rsa-fedora RSA SHA256:QaASD8ZFrq/3xQvhi0jHNnfshoasyieQiVVU1UM9QQ8 explicit agent debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com> debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/nils/.ssh/id_rsa-fedora RSA SHA256:QaASD8ZFrq/3xQvhi0jHNnfshoasyieQiVVU1UM9QQ8 explicit agent debug1: Authentications that can continue: publickey debug1: No more authentication methods to try. nphilipp@buildvm-x86-02.stg.iad2.fedoraproject.org: Permission denied (publickey). nils@makake:~>
Here's an attempt to log into buildvm-x86-01.stg.iad2, which succeeds:
nils@makake:~> ssh -v buildvm-x86-01.stg.iad2.fedoraproject.org OpenSSH_8.5p1, OpenSSL 1.1.1k FIPS 25 Mar 2021 debug1: Reading configuration data /home/nils/.ssh/config debug1: /home/nils/.ssh/config line 3: Applying options for *.fedoraproject.org debug1: /home/nils/.ssh/config line 15: Applying options for *.iad2.fedoraproject.org debug1: /home/nils/.ssh/config line 88: Applying options for * debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: configuration requests final Match pass debug1: re-parsing configuration debug1: Reading configuration data /home/nils/.ssh/config debug1: /home/nils/.ssh/config line 3: Applying options for *.fedoraproject.org debug1: /home/nils/.ssh/config line 15: Applying options for *.iad2.fedoraproject.org debug1: /home/nils/.ssh/config line 88: Applying options for * debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: Executing proxy command: exec ssh -W buildvm-x86-01.stg.iad2.fedoraproject.org:22 bastion-iad01.fedoraproject.org debug1: identity file /home/nils/.ssh/id_rsa-fedora type 0 debug1: identity file /home/nils/.ssh/id_rsa-fedora-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.5 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.5 debug1: compat_banner: match: OpenSSH_8.5 pat OpenSSH* compat 0x04000000 debug1: Authenticating to buildvm-x86-01.stg.iad2.fedoraproject.org:22 as 'nphilipp' debug1: load_hostkeys: fopen /home/nils/.ssh/known_hosts2: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256@libssh.org debug1: kex: host key algorithm: ssh-ed25519-cert-v01@openssh.com debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32 debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: SSH2_MSG_KEX_ECDH_REPLY received debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com SHA256:MUAfYEI5nBHOlYkCFI9eotUWUaytZFQ0+UeDk7sNtFs, serial 1604525022 ID "buildvm-x86-01.stg.iad2.fedoraproject.org" CA ssh-rsa SHA256:pmw8O+j5VVfgpVIjMvWG+u7DizO2dyzrvxHWrLUg02s valid from 2020-11-04T21:23:42 to 2021-11-03T22:23:42 debug1: load_hostkeys: fopen /home/nils/.ssh/known_hosts2: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug1: No matching CA found. Retry with plain key debug1: Host 'buildvm-x86-01.stg.iad2.fedoraproject.org' is known and matches the ED25519 host key. debug1: Found key in /home/nils/.ssh/known_hosts:1077 debug1: rekey out after 4294967296 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey in after 4294967296 blocks debug1: Will attempt key: /home/nils/.ssh/id_rsa-fedora RSA SHA256:QaASD8ZFrq/3xQvhi0jHNnfshoasyieQiVVU1UM9QQ8 explicit agent debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com> debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/nils/.ssh/id_rsa-fedora RSA SHA256:QaASD8ZFrq/3xQvhi0jHNnfshoasyieQiVVU1UM9QQ8 explicit agent debug1: Server accepts key: /home/nils/.ssh/id_rsa-fedora RSA SHA256:QaASD8ZFrq/3xQvhi0jHNnfshoasyieQiVVU1UM9QQ8 explicit agent debug1: Authentication succeeded (publickey). Authenticated to buildvm-x86-01.stg.iad2.fedoraproject.org (via proxy). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions@openssh.com debug1: Entering interactive session. debug1: pledge: filesystem full debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0 debug1: client_input_hostkeys: searching /home/nils/.ssh/known_hosts for buildvm-x86-01.stg.iad2.fedoraproject.org / (none) debug1: client_input_hostkeys: searching /home/nils/.ssh/known_hosts2 for buildvm-x86-01.stg.iad2.fedoraproject.org / (none) debug1: client_input_hostkeys: hostkeys file /home/nils/.ssh/known_hosts2 does not exist debug1: client_input_hostkeys: no new or deprecated keys from server debug1: Remote: /usr/bin/sss_ssh_authorizedkeys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding debug1: Remote: /usr/bin/sss_ssh_authorizedkeys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding debug1: Sending environment. debug1: channel 0: setting env XMODIFIERS = "@im=ibus" debug1: channel 0: setting env LC_MONETARY = "en_GB.UTF-8" debug1: channel 0: setting env LC_PAPER = "en_GB.UTF-8" debug1: channel 0: setting env LANG = "en_US.UTF-8" debug1: channel 0: setting env LC_MEASUREMENT = "en_GB.UTF-8" debug1: channel 0: setting env LC_TIME = "en_GB.UTF-8" debug1: channel 0: setting env LC_NUMERIC = "en_GB.UTF-8" Last login: Tue Jun 1 13:41:44 2021 from 10.3.163.31 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0 logout debug1: channel 0: free: client-session, nchannels 1 Connection to buildvm-x86-01.stg.iad2.fedoraproject.org closed. Transferred: sent 3640, received 4364 bytes, in 2.2 seconds Bytes per second: sent 1688.7, received 2024.6 debug1: Exit status 0 nils@makake:~>
This is from my local laptop, SSH is configured to proxy through bastion all hosts in the .iad2 and .s390 domains (among others). I can't log into these machines from batcave though using my own user, I imagine because I don't have the SSH keys there.
I verified that IPA is configured to let me log into the hosts in question:
[nphilipp@koji01 ~][STG]$ ipa hbactest --user nphilipp --host buildvm-x86-02.stg.iad2.fedoraproject.org --service sshd --rules hostgroup/kojibuilder/shell-access -------------------- Access granted: True -------------------- Matched rules: hostgroup/kojibuilder/shell-access [nphilipp@koji01 ~][STG]$
NB: Running the ipa tool didn't work on buildvm-x86-01.stg.iad2, I get this error (which doesn't seem to affect my ability to login there):
ipa
[nphilipp@buildvm-x86-01 ~][STG]$ kdestroy [nphilipp@buildvm-x86-01 ~][STG]$ kinit Password for nphilipp@STG.FEDORAPROJECT.ORG: [nphilipp@buildvm-x86-01 ~][STG]$ ipa ping ipa: ERROR: Service 'HTTP@ipa01.stg.iad2.fedoraproject.org' not found in Kerberos database [nphilipp@buildvm-x86-01 ~][STG]$
Rather soon would be good, I need this access to deploy rpmautospec updates regularly. We want to make this available for community testing.
Metadata Update from @mohanboddu: - Issue tagged with: medium-gain, medium-trouble, ops
Metadata Update from @mohanboddu: - Issue priority set to: Waiting on Assignee (was: Needs Review)
I think these machines were installed back before the staging ipa servers were re-installed. This meant that their keytabs were against the old server and the new one denied them.
I unenrolled them and re-enrolled them and I think it's working now. at least 'id' shows you, which it did not before. :)
Re-open if you still see any issues...
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Works fine: I can log into and run sudo on all the builders in staging. Thanks!
Log in to comment on this ticket.