Currently nagios on noc02 does not work with kerberos tickets so that authorized users can not change/acknowledge items. Nothing in the logs seem to indicate why this is happening. I think this has been happening since the move to IAD2 but not sure. This is a placeholder ticket to track this as it looks to take more than 5 minutes to fix.
Odd. It works fine for me.
I'm experiencing the same issue as smooge
@puiterwijk or @simo can you take a look at this?
Thanks.
Metadata Update from @mohanboddu: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
Firefox gives this out when I try to go to the site https://nagios-external.fedoraproject.org/nagios/
[smooge@centos-8 ~]$ KRB5_TRACE=/dev/stdout firefox [13400] 1600792487.307294: ccselect module realm chose cache KCM:1000 with client principal smooge@FEDORAPROJECT.ORG for server principal HTTP/noc02.fedoraproject.org@FEDORAPROJECT.ORG [13400] 1600792487.307295: Getting credentials smooge@FEDORAPROJECT.ORG -> HTTP/noc02.fedoraproject.org@FEDORAPROJECT.ORG using ccache KCM:1000 [13400] 1600792487.307296: Retrieving smooge@FEDORAPROJECT.ORG -> HTTP/noc02.fedoraproject.org@FEDORAPROJECT.ORG from KCM:1000 with result: -1765328243/Matching credential not found [13400] 1600792487.307297: Retrieving smooge@FEDORAPROJECT.ORG -> krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG from KCM:1000 with result: 0/Success [13400] 1600792487.307298: Starting with TGT for client realm: smooge@FEDORAPROJECT.ORG -> krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG [13400] 1600792487.307299: Requesting tickets for HTTP/noc02.fedoraproject.org@FEDORAPROJECT.ORG, referrals on [13400] 1600792487.307300: Generated subkey for TGS request: aes256-cts/D6B2 [13400] 1600792487.307301: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts [13400] 1600792487.307303: Encoding request body and padata into FAST request [13400] 1600792487.307304: Sending request (1006 bytes) to FEDORAPROJECT.ORG [13400] 1600792487.307305: Resolving hostname id.fedoraproject.org [13400] 1600792487.307306: TLS certificate name matched "id.fedoraproject.org" [13400] 1600792487.307307: Sending HTTPS request to https 67.219.144.68:443 [13400] 1600792487.307308: Received answer (465 bytes) from https 67.219.144.68:443 [13400] 1600792487.307309: Terminating TCP connection to https 67.219.144.68:443 [13400] 1600792487.307310: Sending DNS URI query for _kerberos.FEDORAPROJECT.ORG. [13400] 1600792487.307311: URI answer: 10 1 "krb5srv:m:kkdcp:https://id.fedoraproject.org/KdcProxy/" [13400] 1600792487.307312: Response was from master KDC [13400] 1600792487.307313: Decoding FAST response [13400] 1600792487.307314: TGS request result: -1765328377/Server HTTP/noc02.fedoraproject.org@FEDORAPROJECT.ORG not found in Kerberos database [13400] 1600792487.307315: Requesting tickets for HTTP/noc02.fedoraproject.org@FEDORAPROJECT.ORG, referrals off [13400] 1600792487.307316: Generated subkey for TGS request: aes256-cts/150C [13400] 1600792487.307317: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts [13400] 1600792487.307319: Encoding request body and padata into FAST request [13400] 1600792487.307320: Sending request (1006 bytes) to FEDORAPROJECT.ORG [13400] 1600792487.307321: Resolving hostname id.fedoraproject.org [13400] 1600792487.307322: TLS certificate name matched "id.fedoraproject.org" [13400] 1600792487.307323: Sending HTTPS request to https 140.211.169.196:443 [13400] 1600792488.391883: Received answer (465 bytes) from https 140.211.169.196:443 [13400] 1600792488.391884: Terminating TCP connection to https 140.211.169.196:443 [13400] 1600792488.391885: Sending DNS URI query for _kerberos.FEDORAPROJECT.ORG. [13400] 1600792488.391886: URI answer: 10 1 "krb5srv:m:kkdcp:https://id.fedoraproject.org/KdcProxy/" [13400] 1600792488.391887: Response was from master KDC [13400] 1600792488.391888: Decoding FAST response [13400] 1600792488.391889: TGS request result: -1765328377/Server HTTP/noc02.fedoraproject.org@FEDORAPROJECT.ORG not found in Kerberos database
Going to the https://nagios.fedoraproject.org/nagios gives me
[13400] 1600792613.661562: ccselect module realm chose cache KCM:1000 with client principal smooge@FEDORAPROJECT.ORG for server principal HTTP/nagios.fedoraproject.org@FEDORAPROJECT.ORG [13400] 1600792613.661563: Getting credentials smooge@FEDORAPROJECT.ORG -> HTTP/nagios.fedoraproject.org@FEDORAPROJECT.ORG using ccache KCM:1000 [13400] 1600792613.661564: Retrieving smooge@FEDORAPROJECT.ORG -> HTTP/nagios.fedoraproject.org@FEDORAPROJECT.ORG from KCM:1000 with result: 0/Success [13400] 1600792613.661566: Creating authenticator for smooge@FEDORAPROJECT.ORG -> HTTP/nagios.fedoraproject.org@FEDORAPROJECT.ORG, seqnum 404185430, subkey aes256-cts/2B81, session key aes256-cts/D6D5 [13400] 1600792618.30995: ccselect module realm chose cache KCM:1000 with client principal smooge@FEDORAPROJECT.ORG for server principal HTTP/nagios.fedoraproject.org@FEDORAPROJECT.ORG [13400] 1600792618.30996: Getting credentials smooge@FEDORAPROJECT.ORG -> HTTP/nagios.fedoraproject.org@FEDORAPROJECT.ORG using ccache KCM:1000 [13400] 1600792618.30997: Retrieving smooge@FEDORAPROJECT.ORG -> HTTP/nagios.fedoraproject.org@FEDORAPROJECT.ORG from KCM:1000 with result: 0/Success [13400] 1600792618.30999: Creating authenticator for smooge@FEDORAPROJECT.ORG -> HTTP/nagios.fedoraproject.org@FEDORAPROJECT.ORG, seqnum 155401984, subkey aes256-cts/4778, session key aes256-cts/D6D5
Klist gives me
[smooge@centos-8 ~]$ klist -a Ticket cache: KCM:1000 Default principal: smooge@FEDORAPROJECT.ORG Valid starting Expires Service principal 09/22/2020 07:37:26 09/23/2020 07:37:19 krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG renew until 09/29/2020 07:37:19 Addresses: (none) 09/22/2020 12:31:06 09/23/2020 07:37:19 HTTP/nagios.fedoraproject.org@FEDORAPROJECT.ORG renew until 09/29/2020 07:37:19 Addresses: (none)
Metadata Update from @smooge: - Issue untagged with: medium-gain, medium-trouble, ops - Issue priority set to: Needs Review (was: Waiting on Assignee)
The server named noc02 exist. A principal for noc2 does not exist.
So this nagios server does not have the right keys in its keytab or has no keytab at all ?
The server has the following keytab
/etc/krb5.HTTP_nagios-external.fedoraproject.org.keytab
which seems to work for Kevin's browsers (Chrome and Firefox in Fedora) but does not work for mine (Chrome and Firefox in EL8). It used to work for me until recently so something seems to have mapped somewhere to know that noc02.fedoraproject.org was also nagios-external.fedoraproject.org . However after we moved it seems to have been lost. [This server noc02 did not move or change configs that I can tell so I am guessing this is ipa or dns side in Fedora.]
To clarify, chrome doesnt work for me either, but firefox does.
The app is https://nagios-external.fedoraproject.org/nagios/ but the hostname of that machine is 'noc02.fedoraproject.org'
$ host nagios-external.fedoraproject.org nagios-external.fedoraproject.org is an alias for noc02.fedoraproject.org. noc02.fedoraproject.org has address 152.19.134.192 noc02.fedoraproject.org has IPv6 address 2610:28:3090:3001:dead:beef:cafe:fed9
In Fedora we changed a setting now canonicalize_hostname = fallback That means that Fedora will try FIRST to get HTTP/nagios-external.fedoraproject.org and if that fails it will try to canonicalize and get HTTP/noc02.fedoraproject.org
However RHEL does not have this feature and the olf value was canonicalize_hostname = true This means on RHEL GSSPAI will stright out try HTTP/noc02.fedoraproject.org and fail.
Note that we changed this behavior in fedora because canonicalization is unsafe (easy to MITM).
In RHEL you can try to set canonicalize_hostname = false (but this may cause issues on servers that expect canonicalization).
On the server side an option is to add a key for HTTP/noc02.fedoraproject.org to the server keytab (not replace, just an additional key), and this will work with all OSs and krb5 versions until the new better canonicalization behavior is adopted across the systems.
Of course another way to solve this is to change nagios-external.fedoraproject.org to be an A name pointing directly to the right address instead of it being a CNAME.
HTH.
Cool and thank you for the detailed explanation. That makes complete sense on why it would break down and how to fix it.
Also, for info, the canonicalize_hostname=false setting being needed for nagios is documented on the Infrastructure Kerberos page: https://fedoraproject.org/wiki/Infrastructure/Kerberos#Extra_info_for_Infrastructure_people
canonicalize_hostname=false
Well I can say I missed that, but I would have probably thought it was old data since nagios.fedoraproject.org works without needing that change.
Is it ok for me to fix this by making nagios-external an A record versus a CNAME and we can remove that section?
Metadata Update from @smooge: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.