#9479 Nightly test failure: healthcheck test_ipahealthcheck_hidden_replica fails with ods-exporter not running
Closed: duplicate by frenaud. Opened by rcritten.

Issue

test_integration.test_replica_promotion.py::TestHiddenReplicaPromotion::test_ipahealthcheck_hidden_replica is failing because ods-exporter is not running.

INFO ipatests.pytest_ipa.integration.host.Host.master.IPAOpenSSHTransport:transport.py:391 RUN ['ipa-healthcheck', '--output-type', 'json', '--failures-only']
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:513 RUN ['ipa-healthcheck', '--output-type', 'json', '--failures-only']
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 Invalid IP address fe80::cf8a:9b86:eb87:e467 for master.ipa.test.: cannot use link-local IP address fe80::cf8a:9b86:eb87:e467
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 [
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 {
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 "source": "ipahealthcheck.meta.services",
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 "check": "ipa_ods_exporter",
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 "result": "ERROR",
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 "uuid": "00f9ae40-b2a1-43ef-8667-428b52fed713",
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 "when": "20231113171849Z",
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 "duration": "0.013054",
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 "kw": {
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 "status": false,
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 "msg": "ipa-ods-exporter: not running"
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 }
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 }
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:557 ]
DEBUG ipatests.pytest_ipa.integration.host.Host.master.cmd83:transport.py:217 Exit code: 1


A clearer version of the failure:

ipa-healthcheck --output-type json --failures-only
[ 
  { 
    "source": "ipahealthcheck.meta.services",
    "check": "ipa_ods_exporter",
    "result": "ERROR",
    "uuid": "00f9ae40-b2a1-43ef-8667-428b52fed713",
    "when": "20231113171849Z",
    "duration": "0.013054",
    "kw": {
      "status": false,
      "msg": "ipa-ods-exporter: not running"
    }
  }
]

This is completely unrelated to what is being tested and is a side-effect of validating more IPA services.

I can reproduce this pretty easily . i added DNSSEC post-install with: ipa-dns-install --dnssec-master

I then got busy and when I came back, healthcheck fails. Since this is a socket I think the check needs to be removed altogether.

ipa-ods-exporter is also crashing which may be the root cause. Looks like same traceback as https://pagure.io/freeipa/issue/9057

# systemctl status ipa-ods-exporter
○ ipa-ods-exporter.service - IPA OpenDNSSEC Signer replacement
Loaded: loaded (/usr/lib/systemd/system/ipa-ods-exporter.service; disabled; preset: disabled)
Active: inactive (dead) since Mon 2023-11-13 15:24:34 EST; 17min ago
Duration: 2.216s
TriggeredBy: ● ipa-ods-exporter.socket
Process: 2806401 ExecStart=/usr/libexec/ipa/ipa-ods-exporter (code=exited, status=0/SUCCESS)
Main PID: 2806401 (code=exited, status=0/SUCCESS)
CPU: 2.064s

Nov 13 15:24:34 ipa.example.test ipa-ods-exporter[2806401]: ipa-ods-exporter: INFO master2ldap_zone_keys: keys in local HSM & LDAP: set()
Nov 13 15:24:34 ipa.example.test ipa-ods-exporter[2806401]: ipa-ods-exporter: INFO master2ldap_zone_keys: keys in local HSM & LDAP: set()
Nov 13 15:24:34 ipa.example.test ipa-ods-exporter[2806401]: ipa-ods-exporter: INFO HSM synchronization finished, skipping zone synchronization.
Nov 13 15:24:34 ipa.example.test ipa-ods-exporter[2806401]: Exception ignored in:
Nov 13 15:24:34 ipa.example.test ipa-ods-exporter[2806401]: Traceback (most recent call last):
Nov 13 15:24:34 ipa.example.test ipa-ods-exporter[2806401]: File "/usr/lib/python3.11/site-packages/ipaserver/dnssec/localhsm.py", line 99, in del
Nov 13 15:24:34 ipa.example.test ipa-ods-exporter[2806401]: File "/usr/lib/python3.11/site-packages/ipaserver/p11helper.py", line 939, in finalize
Nov 13 15:24:34 ipa.example.test ipa-ods-exporter[2806401]: TypeError: 'NoneType' object is not callable
Nov 13 15:24:34 ipa.example.test systemd[1]: ipa-ods-exporter.service: Deactivated successfully.
Nov 13 15:24:34 ipa.example.test systemd[1]: ipa-ods-exporter.service: Consumed 2.064s CPU time.

Metadata Update from @frenaud:
- Issue tagged with: test-failure

Closing as duplicate of #9463

Metadata Update from @frenaud:
- Issue close_status updated to: duplicate
- Issue status updated to: Closed (was: Open)

Metadata