In environments where 389ds is overwhelmed, KDC crashes are happening as part of ipadb_simple_search(), more precisely when calling the openldap API, which is surprising because the connection with 389ds is supposed to be checked before that point:
ipadb_simple_search()
krb5_error_code ipadb_simple_search(struct ipadb_context *ipactx, char *basedn, int scope, char *filter, char **attrs, LDAPMessage **res) { int ret; ret = ipadb_check_connection(ipactx); if (ret != 0) return ipadb_simple_ldap_to_kerr(ret); ret = ldap_search_ext_s(ipactx->lcontext, basedn, scope, /* KDC crash here */ filter, attrs, 0, NULL, NULL, &std_timeout, LDAP_NO_LIMIT, res);
The connection check function tries to re-establish the connection if the LDAP context (ipactx->lcontext) is NULL. Error handling seems to be done properly in all the functions mentioned so far, which pushes me to think there is likely a problem with the ipadb_get_connection() function.
ipactx->lcontext
ipadb_get_connection()
KDC crashes occur just after a failed attempt to configure the PAC generator.
krb5kdc[xxxxx]: MS-PAC generator: Failed to fetch trusted domains information systemd[1]: Started Process Core Dump (PID xxxxx/UID 0).
This kind of messages are returned by the ipadb_reinit_mspac() function, which may call ipadb_mspac_check_trusted_domains() where ipadb_simple_search() is called too, which might result in a ipadb_get_connection() call, if a connection failure happens at this point, the LDAP context will be reset to NULL:
ipadb_reinit_mspac()
ipadb_mspac_check_trusted_domains()
ipadb_reinit_mspac():
err = ipadb_mspac_check_trusted_domains(ipactx); if (err) { if (err == KRB5_KDB_NOENTRY) { /* SKIP */ err = 0; } else { in_stmsg = "Failed to fetch trusted domains information"; /* message just before KDC crash */ } goto end; } {code} ``ipadb_mspac_check_trusted_domains()``: ```c ret = ipadb_simple_search(ipactx, base, LDAP_SCOPE_SUBTREE, filter, attrs, &result); done: ldap_msgfree(result); free(base); return ret;
The problem is that the final step in ipadb_get_connection() is also calling ipadb_reinit_mspac(). There are clearly recursion and error handling problems here:
ret = ipadb_reinit_mspac(ipactx, false, &stmsg); /* connection lost, ipactx->lcontext is NULL */ if (ret && stmsg) krb5_klog_syslog(LOG_WARNING, "MS-PAC generator: %s", stmsg); ret = 0; /* connection error lost here */ done: ldap_msgfree(res); if (ret) { /* ret == 0, skipped */ if (ipactx->lcontext) { ldap_unbind_ext_s(ipactx->lcontext, NULL, NULL); ipactx->lcontext = NULL; } if (ret == LDAP_SERVER_DOWN) { return ETIMEDOUT; } return EIO; } return 0; /* no error returned, ipactx->lcontext still NULL */
Non-zero return codes from ipadb_reinit_mspac() are not treated as fatal error because the absence of the LDAP entries needed to configure the PAC generator are not strictly required for the KDC to function. However, a connection lost error in this context is fatal because it will be ignored, and the final part of the function returning an error code if the LDAP context is NULL will not be executed.
Since no error was raised, ipadb_simple_search() will call ldap_search_ext_s() with a NULL LDAP context.
ldap_search_ext_s()
Metadata Update from @jrische: - Custom field rhbz adjusted to https://issues.redhat.com/browse/RHEL-88833 https://issues.redhat.com/browse/RHEL-89145 https://issues.redhat.com/browse/RHEL-88834 https://issues.redhat.com/browse/RHEL-89144 https://issues.redhat.com/browse/RHEL-89143 https://issues.redhat.com/browse/RHEL-89142 https://issues.redhat.com/browse/RHEL-89141 https://issues.redhat.com/browse/RHEL-89140 https://issues.redhat.com/browse/RHEL-58453 https://issues.redhat.com/browse/RHEL-89149 https://issues.redhat.com/browse/RHEL-89148 https://issues.redhat.com/browse/RHEL-89147 https://issues.redhat.com/browse/RHEL-89146
Metadata Update from @jrische: - Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/7790
Metadata Update from @jrische: - Custom field rhbz adjusted to https://issues.redhat.com/browse/RHEL-88833 https://issues.redhat.com/browse/RHEL-89145 https://issues.redhat.com/browse/RHEL-88834 https://issues.redhat.com/browse/RHEL-89144 https://issues.redhat.com/browse/RHEL-89143 https://issues.redhat.com/browse/RHEL-89142 https://issues.redhat.com/browse/RHEL-89141 https://issues.redhat.com/browse/RHEL-89140 https://issues.redhat.com/browse/RHEL-58453 https://issues.redhat.com/browse/RHEL-89149 https://issues.redhat.com/browse/RHEL-89148 (was: https://issues.redhat.com/browse/RHEL-88833 https://issues.redhat.com/browse/RHEL-89145 https://issues.redhat.com/browse/RHEL-88834 https://issues.redhat.com/browse/RHEL-89144 https://issues.redhat.com/browse/RHEL-89143 https://issues.redhat.com/browse/RHEL-89142 https://issues.redhat.com/browse/RHEL-89141 https://issues.redhat.com/browse/RHEL-89140 https://issues.redhat.com/browse/RHEL-58453 https://issues.redhat.com/browse/RHEL-89149 https://issues.redhat.com/browse/RHEL-89148 https://issues.redhat.com/browse/RHEL-89147 https://issues.redhat.com/browse/RHEL-89146)
master:
ipa-4-9:
ipa-4-10:
ipa-4-11:
ipa-4-12:
Closing the ticket.
Metadata Update from @sumenon: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)