#9481 Retry "storms" with sssd
Closed: invalid by trimbolim. Opened by trimbolim.

Issue

I have a recurring problem where sssd seems to DDOS my IPA servers. It seems to only happen when there is a disturbance to the normal demand placed on the system.

The last time it occurred was when I brought 6 of 9 IPA servers down for network maintenance.

My baseline kerb request per IPA server per second is 5.6. After my 3-server shutdown it was almost immediately 8.6, as expected. But 2.5 hours after the shutdown, it sharply increased to 21.6. At this point, attempts to make tcp connections on port 88 were timing out. A sampling of sssd logs showed that "no KDC was available."

My baseline kerb request per IPA server per second is 5.6.

After 3 more hours, I lost my nerve to wait for things to rebalance themselves and restarted the 3 servers I had shut down. 1 hour after that, kerberos requests sharply dropped back to their baseline rate and service was fully restored.

I would like some assistance in evaluating the sanity of my sssd.conf. For an idea of the scale, I have 15k clients pointed at these 9 IPA servers.

I have my doubts about the sssd.conf because it has "evolved" over the years with many fingers in it. Nothing among the various timer settings is speaking to me, so I humbly submit to public review.

For your perusal:
```
[domain/edge.company.com]

cache_credentials = True
krb5_store_password_if_offline = True
krb5_realm = AUTH.COMPANY.COM
ipa_domain = edge.company.com
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ldap_tls_cacert = /etc/ipa/ca.crt
ipa_hostname = qa1-server.edge.company.com
chpass_provider = ipa
ipa_server = srv, ipa102.auth.company.com
dns_discovery_domain = edge.company.com
refresh_expired_interval=45038
entry_cache_timeout=56238
entry_cache_sudo_timeout=56238
entry_cache_netgroup_timeout=604800
krb5_auth_timeout=30
ldap_opt_timeout=30
subdomains_provider=none
selinux_provider=none
session_provider=none
autofs_provider=none
ipa_hbac_refresh=600
[sssd]
services = nss, pam, ssh, sudo

domains = edge.company.com
[nss]
homedir_substring = /home
cache_first=True
[pam]

cache_first=True
pam_id_timeout=600
[sudo]

cache_first=True
[autofs]

[ssh]

cache_first=True
[pac]

[ifp]

[secrets]

[session_recording]

Actual behavior

Overloaded krb5kdc processes.

Expected behavior

Smoother rebalancing with sssd.

Version/Release/Distribution

Client:
$ rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server
ipa-client-4.6.8-5.el7.centos.15.x86_64
Server:
package freeipa-server is not installed
package freeipa-client is not installed
ipa-server-4.9.11-6.module+el8.8.0+19022+e8902f4b.x86_64
ipa-client-4.9.11-6.module+el8.8.0+19022+e8902f4b.x86_64
389-ds-base-1.4.3.35-1.module+el8.8.0+19157+dcce0882.x86_64
package pki-ca is not installed
krb5-server-1.18.2-25.el8_8.x86_64

Genericized_IPA_traffic.png


Since this is an SSSD-specific issue I think you should open an issue with that project directly, https://github.com/SSSD/sssd/issues/

Since this is an SSSD-specific issue I think you should open an issue with that project directly, https://github.com/SSSD/sssd/issues/

Thank you. Will do.

Metadata Update from @trimbolim:
- Issue close_status updated to: invalid
- Issue status updated to: Closed (was: Open)

Metadata