#47896 DS dump a core at shutdown in persistent search thread (sync plugin)
Closed: wontfix Opened by tbordaz.

Test with freeipa:
freeipa-server-4.0.1-1
389-ds-base-1.3.2.23

When DS stops it does not wait for the persistent search threads of the sync plugin.
It frees the connection table and a persistent search thread may access the connection after it has been freed.

test case is:
install freeipa+dns
ipa-server-install -a -p --hostname=vm-001.example.com -r EXAMPLE.COM -n example.com -U
ipa-dns-install -p --no-forwarders --no-reverse

Enables core dumps (because DS stops smoothly, it is the only way to detect the crash at shutdown)

loop: ipactl restart

The crash occurs rarely (1 times out of 20 restart on my VM)


{{{
(gdb) info threads
Id Target Id Frame
2 Thread 0x7f7d70e81840 (LWP 20170) 0x00007f7d6e78bd20 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1 Thread 0x7f7d3efe5700 (LWP 20239) slapi_connection_remove_operation (pb=0x7f7d38004c60, conn=0x7f7d5c4a3800,
op=0x7f7d734daf40, release=1) at ldap/servers/slapd/operation.c:592
(gdb) where

0 slapi_connection_remove_operation (pb=0x7f7d38004c60, conn=0x7f7d5c4a3800, op=0x7f7d734daf40, release=1)

at ldap/servers/slapd/operation.c:592

1 0x00007f7d673a3dee in sync_release_connection (release=1, op=, conn=, pb=)

at ldap/servers/plugins/sync/sync_persist.c:522

2 sync_send_results (arg=) at ldap/servers/plugins/sync/sync_persist.c:664

3 0x00007f7d6ede7e5b in _pt_root () from /lib64/libnspr4.so

4 0x00007f7d6e787f33 in start_thread () from /lib64/libpthread.so.0

5 0x00007f7d6e4b5ded in clone () from /lib64/libc.so.6

(gdb) print *conn
Cannot access memory at address 0x7f7d5c4a3800

}}}

Per suggestion by Thierry, setting the milestone to 1.3.4.

deferred cloning (not sure if we'll get to this).

I'm running this command line for setting up the debug env.

ipa-server-install -a passwordA -p paawordP --hostname=ibm-x3650m4-01-vm-16.lab.eng.bos.redhat.com -r EXAMPLE.COM -n example.com -U

Then, it sopped here:
{{{
Configuring Kerberos KDC (krb5kdc). Estimated time: 30 seconds
[1/10]: adding sasl mappings to the directory
[2/10]: adding kerberos container to the directory
[3/10]: configuring KDC
[4/10]: initialize kerberos container
WARNING: Your system is running out of entropy, you may experience long delays
}}}
Is this some resource issue? I'm curious what is the entropy? Is it some notion in kerberos or IPA?

I'm assuming you are running in a VM? How are you creating the VM? You will need to use a hardware PRNG in the VM in order to provide enough entropy (randomness for the random number generator, used to generate crypto keys).

For example, with virt-install, I use --rng /dev/random. Then, inside the VM, I do this very early, before running ipa-server-install:
{{{
# Set up entropy source for IPA installer
rngd -r /dev/hwrng
}}}

Thierry, I'm running the scenario on a VM.

ipa-server-install -a -p --hostname=vm-001.example.com -r EXAMPLE.COM -n example.com -U
ipa-dns-install -p --no-forwarders --no-reverse
loop: ipactl restart

So far, it restarted 100 times without a crash...

Does it require more powerful configuration? Or it was already fixed with some other bug fix?

I continue running the test overnight, but if you could rerun the test and tell us the result, I'd appreciate it. Thanks!

BTW, DS has no problem, but at one point (precisely, from the 88th attempt), httpd started failing to restart

Starting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting ipa_memcached Service
Restarting httpd Service
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
Failed to restart httpd Service
Shutting down
Aborting ipactl

Do we care?

I repeated ipactl restart over night to see no crash.
{{{
7020th
Starting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting ipa_memcached Service
Starting httpd Service
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
Failed to start httpd Service
Shutting down
Aborting ipactl
ls: cannot access /var/log/dirsrv/slapd-EXAMPLE-COM/core*: No such file or directory
}}}

Let me close this ticket for now. Please feel free to reopen it when the crash occurred again.

Note: tested with 389-ds-base-1.3.4.4.

Metadata Update from @rmeggins:
- Issue set to the milestone: 1.3.4 backlog

Metadata Update from @vashirov:
- Custom field reviewstatus adjusted to None (was: needinfo)
- Custom field rhbz reset (from todo)
- Issue close_status updated to: None (was: Invalid)
- Issue set to the milestone: None (was: 1.3.4 backlog)

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/1227

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix

Metadata