freeipa

#9134 Nightly test failure (rawhide) while establishing two-way trust
Closed: fixed by frenaud. Opened by frenaud.

Closed: fixed
Reopen Issue

The nightly tests are failing on rawhide when establishing a two-way trust, see for instance PR #1602 with the test test_smb (Logs, report):

INFO     ipatests.pytest_ipa.integration.host.Host.master.IPAOpenSSHTransport:transport.py:391 RUN ['ipa', 'trust-add', '--type', 'ad', 'ad.test', '--two-way=true', '--admin', 'Administrator@ad.test', '--password']
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd100:transport.py:513 RUN ['ipa', 'trust-add', '--type', 'ad', 'ad.test', '--two-way=true', '--admin', 'Administrator@ad.test', '--password']
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd100:transport.py:557 ipa: ERROR: Insufficient access: IPA master denied trust validation requests from AD DC 10 times. Most likely AD DC contacted a replica that has no trust information replicated yet. Additionally, please check that AD DNS is able to resolve _ldap._tcp.ipa.test, _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs.ipa.test SRV records to the correct IPA server.

Packages:
samba-4.16.0-6.fc37.x86_64

Manually reproduced, httpd error_log:

s4_tevent: Run immediate event "tevent_req_trigger": 0x7f05a0094c00
     netr_LogonControl2Ex: struct netr_LogonControl2Ex
        out: struct netr_LogonControl2Ex
            query                    : *
                query                    : union netr_CONTROL_QUERY_INFORMATION(case 2)
                info2                    : *
                    info2: struct netr_NETLOGON_INFO_2
                        flags                    : 0x00000080 (128)
                               0: NETLOGON_REPLICATION_NEEDED
                               0: NETLOGON_REPLICATION_IN_PROGRESS
                               0: NETLOGON_FULL_SYNC_REPLICATION
                               0: NETLOGON_REDO_NEEDED
                               0: NETLOGON_HAS_IP
                               0: NETLOGON_HAS_TIMESERV
                               0: NETLOGON_DNS_UPDATE_FAILURE
                               1: NETLOGON_VERIFY_STATUS_RETURNED
                        pdc_connection_status    : WERR_ACCESS_DENIED
                        trusted_dc_name          : *
                            trusted_dc_name          : ''
                        tc_connection_status     : WERR_ACCESS_DENIED
            result                   : WERR_OK
rpc reply data:
[0000] 02 00 00 00 00 00 02 00   80 00 00 00 05 00 00 00   ........ ........
[0010] 04 00 02 00 05 00 00 00   01 00 00 00 00 00 00 00   ........ ........
[0020] 01 00 00 00 00 00 00 00   00 00 00 00               ........ ....
s4_tevent: Added timed event "tevent_req_timedout": 0x7f05a0098a60
smb2_signing_encrypt_pdu: Encrypted SMB2 message
s4_tevent: Schedule immediate event "tevent_req_trigger": 0x7f05a0094c00
s4_tevent: Cancel immediate event 0x7f05a0094c00 "tevent_req_trigger"
s4_tevent: Schedule immediate event "tevent_req_trigger": 0x7f05a0094c00
s4_tevent: Destroying timer event 0x7f05a0098a60 "tevent_req_timedout"
s4_tevent: Cancel immediate event 0x7f05a0094c00 "tevent_req_trigger"
[Mon Mar 28 14:14:57.652676 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454] ipa: DEBUG: WSGI wsgi_execute PublicError: Traceback (most recent call last):
[Mon Mar 28 14:14:57.652732 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]   File "/usr/lib/python3.10/site-packages/ipaserver/rpcserver.py", line 407, in wsgi_execute
[Mon Mar 28 14:14:57.652756 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]     result = command(*args, **options)
[Mon Mar 28 14:14:57.652778 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]   File "/usr/lib/python3.10/site-packages/ipalib/frontend.py", line 471, in __call__
[Mon Mar 28 14:14:57.652800 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]     return self.__do_call(*args, **options)
[Mon Mar 28 14:14:57.652822 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]   File "/usr/lib/python3.10/site-packages/ipalib/frontend.py", line 499, in __do_call
[Mon Mar 28 14:14:57.652844 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]     ret = self.run(*args, **options)
[Mon Mar 28 14:14:57.652866 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]   File "/usr/lib/python3.10/site-packages/ipalib/frontend.py", line 816, in run
[Mon Mar 28 14:14:57.652887 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]     return self.execute(*args, **options)
[Mon Mar 28 14:14:57.652909 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]   File "/usr/lib/python3.10/site-packages/ipaserver/plugins/trust.py", line 762, in execute
[Mon Mar 28 14:14:57.652943 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]     result = self.execute_ad(full_join, *keys, **options)
[Mon Mar 28 14:14:57.652981 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]   File "/usr/lib/python3.10/site-packages/ipaserver/plugins/trust.py", line 1018, in execute_ad
[Mon Mar 28 14:14:57.653006 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]     result = self.trustinstance.join_ad_full_credentials(
[Mon Mar 28 14:14:57.653048 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]   File "/usr/lib/python3.10/site-packages/ipaserver/dcerpc.py", line 1867, in join_ad_full_credentials
[Mon Mar 28 14:14:57.653116 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]     result = self.remote_domain.verify_trust(self.local_domain)
[Mon Mar 28 14:14:57.653142 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]   File "/usr/lib/python3.10/site-packages/ipaserver/dcerpc.py", line 1488, in verify_trust
[Mon Mar 28 14:14:57.653176 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]     return self.verify_trust(another_domain)
[Mon Mar 28 14:14:57.653217 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]   File "/usr/lib/python3.10/site-packages/ipaserver/dcerpc.py", line 1488, in verify_trust
[Mon Mar 28 14:14:57.653241 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]     return self.verify_trust(another_domain)
[Mon Mar 28 14:14:57.653264 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]   File "/usr/lib/python3.10/site-packages/ipaserver/dcerpc.py", line 1488, in verify_trust
[Mon Mar 28 14:14:57.653286 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]     return self.verify_trust(another_domain)
[Mon Mar 28 14:14:57.653361 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]   [Previous line repeated 6 more times]
[Mon Mar 28 14:14:57.653384 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]   File "/usr/lib/python3.10/site-packages/ipaserver/dcerpc.py", line 1510, in verify_trust
[Mon Mar 28 14:14:57.653407 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]     raise errors.ACIError(info=error_message)
[Mon Mar 28 14:14:57.653434 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454] ipalib.errors.ACIError: Insufficient access: IPA master denied trust validation requests from AD DC 10 times. Most likely AD DC contacted a replica that has no trust information replicated yet. Additionally, please check that AD DNS is able to resolve _ldap._tcp.testrelm.test, _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs.testrelm.test SRV records to the correct IPA server.
[Mon Mar 28 14:14:57.653458 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454]
[Mon Mar 28 14:14:57.653719 2022] [wsgi:error] [pid 24943:tid 25283] [remote 10.0.169.59:33454] ipa: INFO: [jsonserver_session] admin@TESTRELM.TEST: trust_add/1('ad.test', trust_type='ad', realm_admin='Administrator', realm_passwd='********', bidirectional=True, version='2.246'): ACIError

We need samba debug logs as WERR_ACCESS_DENIED is coming from AD DC when it attempst to talk to IPA DC (e.g. Samba).

@frenaud gave me logs with more details and basically this is a problem since we added commit b5fbbd1957873207afc2542a9d32246bccca0556 in 2019. Its logic allowed to add RC4-HMAC keys for cifs/.. service principal but it didn't account for the case when cifs/.. principal initiates the request. The latter is when ipasam PDB module for Samba attempts to create a trust account. We rely on system-wide crypto policy to disallow clients using RC4-HMAC but we need this key to be able to retrieve NTLM hash on-demand for the trusted domain objects in ipasam.
Trust validation process triggers a request ServerAuthenticate3 from the AD DC side which relies on NTLM hash availability to mutually authenticate both parties (https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-nrpc/3a9ed16f-8014-45ae-80af-c0ecb06e2db9):

The server MUST compute or retrieve the NTOWFv1 (as specified in NTLM v1 Authentication in [MS-NLMP] section 3.3.1) of the client computer password and use it to compute a session key, as specified in section 3.1.4.3. If the server cannot compute or retrieve the NTOWFv1 of the client computer password, it MUST return STATUS_NO_TRUST_SAM_ACCOUNT.

The server MUST compute the client Netlogon credential as specified in section 3.1.4.4 and compare the result with the client Netlogon credential passed from the client for verification. The computation is performed using the ClientChallenge from the ChallengeTable. If the comparison fails, session-key negotiation fails, and the server MUST return STATUS_ACCESS_DENIED.

The part that needs change is https://pagure.io/freeipa/blob/master/f/daemons/ipa-slapi-plugins/ipa-pwd-extop/ipa_pwd_extop.c#_1712, where we define if NTLM hashes are allowed for the getkeytab control. Note that we have the same check in writekeytab control as well but that one should be left unchanged. We only need to extend is_nthash_allowed to verify if bindDN is actually a krbprincipalname=cifs/this-master@REALM which is what ipasam module uses.

PR: https://github.com/freeipa/freeipa/pull/6232

Metadata Update from @abbra:
- Issue assigned to abbra

master:

  • 5638bdcb854b317f5bd890dd3fc225277246b67a ipa-pwd-extop: allow ipasam to request RC4-HMAC in Kerberos keys for trusted domain objects
  • 4a95661686544bbe3a242e99e6afbd63d5fcc19a ipa-sam: retrieve trusted domain account credential from the TDO itself
  • 599792e86341478913060ac404c1bc062de3da9e ipatests: collect samba logs when setting up trust to AD

ipa-4-9:

  • 710314a794eb3446f0467d33133d70d2425fbf65 ipa-pwd-extop: allow ipasam to request RC4-HMAC in Kerberos keys for trusted domain objects
  • 91d083c36e1daf88686bf8096691b3913d2ad23c ipa-sam: retrieve trusted domain account credential from the TDO itself
  • ee6472cee20ff99d16cde1a97c3dd5167b7cd893 ipatests: collect samba logs when setting up trust to AD

Metadata Update from @frenaud:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)
Metadata

test-failure

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.