freeipa

#9638 mod_wsgi can't close cleany
Opened by et7f3. Modified

Open
Close Issue

Request for enhancement

As a sysadmin , I want to be able to reload httpd/ipa so that I can change the log/configuration.

Issue

We have found that during reload of httpd.service triggered by our logrotate configuration we had leftover process from apache. They stacked and we arrived to the failure "scoreboard is full, not at MaxRequestWorkers.Increase Server".

With this thread we were able to get a reproducer: https://www.apachelounge.com/viewtopic.php?t=9178

Steps to Reproduce

  1. Open a webpage in freeipa that call python rpc (for instance user details)
  2. You can spam request to simulate a high traffic insance (and hit the multiple worker)
  3. pgrep httpd | wc -l
  4. reload the apache service systemctl reload httpd
  5. pgrep httpd | wc -l

Actual behavior

If we inspect stack trace of old gen process we can see a dead lock:

$ gstack 3785
#0  0x00007f27e088679a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
#1  0x00007f27e0891c48 in __new_sem_wait_slow64.constprop.0 () from target:/lib64/libc.so.6
#2  0x00007f27deefe2da in PyThread_acquire_lock_timed () from target:/lib64/libpython3.9.so.1.0
#3  0x00007f27def8e461 in acquire_timed () from target:/lib64/libpython3.9.so.1.0
#4  0x00007f27def8e26f in lock_PyThread_acquire_lock () from target:/lib64/libpython3.9.so.1.0
#5  0x00007f27def1ce51 in method_vectorcall_VARARGS_KEYWORDS () from target:/lib64/libpython3.9.so.1.0
#6  0x00007f27def0fe37 in _PyEval_EvalFrameDefault () from target:/lib64/libpython3.9.so.1.0
#7  0x00007f27def1c223 in function_code_fastcall () from target:/lib64/libpython3.9.so.1.0
#8  0x00007f27df6d017b in ShutdownInterpreter_call () from target:/etc/httpd/modules/mod_wsgi_python3.so
#9  0x00007f27def17f24 in _PyObject_MakeTpCall () from target:/lib64/libpython3.9.so.1.0
#10 0x00007f27def2dd41 in PyObject_VectorcallMethod () from target:/lib64/libpython3.9.so.1.0
#11 0x00007f27defb618c in wait_for_thread_shutdown () from target:/lib64/libpython3.9.so.1.0
#12 0x00007f27deef3d94 in Py_EndInterpreter () from target:/lib64/libpython3.9.so.1.0
#13 0x00007f27df6cef1d in Interpreter_dealloc () from target:/etc/httpd/modules/mod_wsgi_python3.so
#14 0x00007f27def0c318 in free_keys_object () from target:/lib64/libpython3.9.so.1.0
#15 0x00007f27df6d711c in wsgi_python_child_cleanup () from target:/etc/httpd/modules/mod_wsgi_python3.so
#16 0x00007f27e0b07eae in apr_pool_destroy () from target:/lib64/libapr-1.so.0
#17 0x00007f27e040776f in clean_child_exit () from target:/etc/httpd/modules/mod_mpm_event.so
#18 0x00007f27e0409c4e in child_main () from target:/etc/httpd/modules/mod_mpm_event.so
#19 0x00007f27e040a57a in make_child () from target:/etc/httpd/modules/mod_mpm_event.so
#20 0x00007f27e040b318 in event_run () from target:/etc/httpd/modules/mod_mpm_event.so
#21 0x00005631d27dc6d8 in ap_run_mpm ()
#22 0x00005631d27ca64f in main ()

Expected behavior

process should exit without trouble

Version/Release/Distribution

$ rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server python3-mod_wsgi
attention : Signature not supported. Hash algorithm SHA1 not available.
le paquet freeipa-server n'est pas installé
le paquet freeipa-client n'est pas installé
ipa-server-4.11.0-15.el9_4.x86_64
ipa-client-4.11.0-15.el9_4.x86_64
389-ds-base-2.4.5-8.el9_4.x86_64
le paquet pki-ca n'est pas installé
krb5-server-1.21.1-1.el9.x86_64
python3-mod_wsgi-4.7.1-11.el9.x86_64

IPA configures WSGI with a display name so the processes should appear as "wsgi:ipa" and "wsgi:kdcproxy". Do those remain after the restart?

 
$ systemctl restart httpd && systemctl status | grep -e ipa -e httpd
           │ ├─httpd.service
           │ │ ├─64454 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─64456 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─64459 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─64460 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─64461 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─64462 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─64463 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─64464 /usr/sbin/httpd -DFOREGROUND
           │ │ └─64465 /usr/sbin/httpd -DFOREGROUND
           │ ├─ipa-custodia.service
           │ │ └─14660 /usr/bin/python3 -I /usr/libexec/ipa/ipa-custodia /etc/ipa/custodia/custodia.conf
               │ └─64692 grep --color=auto -e ipa -e httpd

it seems yes.

This is only relevant in context. You need to get the current process list, then restart and compare the old to the new. In my testing they are all new.

Sorry after I posted I saw I was dumb (for not providing old value).
So I redo again with httpd status page:

$ systemctl stop httpd && systemctl start httpd && sleep 3 && systemctl status | grep -e ipa -e httpd -e wsgi && pgrep httpd | wc -l
           │ ├─httpd.service
           │ │ ├─38414 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38416 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38417 "(wsgi:kdcproxy)" -DFOREGROUND
           │ │ ├─38419 "(wsgi:kdcproxy)" -DFOREGROUND
           │ │ ├─38420 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38421 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38422 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38423 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38424 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38425 /usr/sbin/httpd -DFOREGROUND
           │ │ └─38426 /usr/sbin/httpd -DFOREGROUND
           │ ├─ipa-custodia.service
           │ │ └─9825 /usr/bin/python3 -I /usr/libexec/ipa/ipa-custodia /etc/ipa/custodia/custodia.conf
               │ └─38671 grep --color=auto -e ipa -e httpd -e wsgi
11

here we have the base case.

FREEIPA_wsgi_base.png

I opened a freeipa page (so did a rpc call):

$ systemctl status | grep -e ipa -e httpd -e wsgi && pgrep httpd | wc -l
           │ ├─httpd.service
           │ │ ├─38414 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38416 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38417 "(wsgi:kdcproxy)" -DFOREGROUND
           │ │ ├─38419 "(wsgi:kdcproxy)" -DFOREGROUND
           │ │ ├─38420 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38421 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38422 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38423 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38424 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38425 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38426 /usr/sbin/httpd -DFOREGROUND
           │ │ └─38690 /usr/sbin/httpd -DFOREGROUND
           │ ├─ipa-custodia.service
           │ │ └─9825 /usr/bin/python3 -I /usr/libexec/ipa/ipa-custodia /etc/ipa/custodia/custodia.conf
               │ └─38830 grep --color=auto -e ipa -e httpd -e wsgi
12

process 38690 /usr/sbin/httpd -DFOREGROUND is created (last in the dashboard)

systemctl reload httpd && sleep 3 && systemctl status | grep -e ipa -e httpd -e wsgi && pgrep httpd | wc -l
           │ ├─httpd.service
           │ │ ├─38414 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38425 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38847 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38848 "(wsgi:kdcproxy)" -DFOREGROUND
           │ │ ├─38849 "(wsgi:kdcproxy)" -DFOREGROUND
           │ │ ├─38850 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38851 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38852 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38853 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38854 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38855 /usr/sbin/httpd -DFOREGROUND
           │ │ └─38856 /usr/sbin/httpd -DFOREGROUND
           │ ├─ipa-custodia.service
           │ │ └─9825 /usr/bin/python3 -I /usr/libexec/ipa/ipa-custodia /etc/ipa/custodia/custodia.conf
               │ └─39071 grep --color=auto -e ipa -e httpd -e wsgi
12

Then I open directly the dashboard.

FREEIPA_wsgi_first_zombie.png

after first request I got:

FREEIPA_wsgi_first_request_after_zombie.png

and process wise:

systemctl status | grep -e ipa -e httpd -e wsgi && pgrep httpd | wc -l
           │ ├─httpd.service
           │ │ ├─38414 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38425 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38847 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38848 "(wsgi:kdcproxy)" -DFOREGROUND
           │ │ ├─38849 "(wsgi:kdcproxy)" -DFOREGROUND
           │ │ ├─38850 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38851 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38852 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38853 "(wsgi:ipa)     " -DFOREGROUND
           │ │ ├─38854 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38855 /usr/sbin/httpd -DFOREGROUND
           │ │ ├─38856 /usr/sbin/httpd -DFOREGROUND
           │ │ └─39091 /usr/sbin/httpd -DFOREGROUND
           │ ├─ipa-custodia.service
           │ │ └─9825 /usr/bin/python3 -I /usr/libexec/ipa/ipa-custodia /etc/ipa/custodia/custodia.conf
               │ └─39147 grep --color=auto -e ipa -e httpd -e wsgi
13
Metadata
None
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.