Can you provide more details on what is in the certificate and its request?

The selftest checks the following certificates:

kra.cert.list=transport,storage,sslserver,subsystem,audit_signing

The logs can be found on our internal artifacts server, below idm-ci/freeipa_upstream_nightly/Nightly-latest-fips/master/2024-11-19_04-33/latest/test_hsm_TestHSMcertRenewal/4/

The same issue can be seen with RHEL 10 after renewal of the auditSigningCert cert-pki-kra. The renewal produces a new certificate but the KRA self tests fail with:

SystemCertsVerification: system certs verification failure: Invalid certificate: (-8101) Certificate type not approved for application.

It makes me wonder if we are renewing with the wrong profile. certmonger is using caAuditSigningCert

No, it looks like it used the right profile:

Nov 19 15:18:29 master.ufreeipa.test certmonger[20493]: 2024-11-19 15:18:29 [20493] Setting "CERTMONGER_LOCAL_CA_DIR" to "/var/lib/certmonger/local" for child.
Nov 19 15:18:29 master.ufreeipa.test certmonger[20493]: 2024-11-19 15:18:29 [20493] Setting "CERTMONGER_KEY_TYPE" to "RSA" for child.
Nov 19 15:18:29 master.ufreeipa.test certmonger[20493]: 2024-11-19 15:18:29 [20493] Setting "CERTMONGER_CA_COOKIE" to "{"cookie": "request:{\"cookie\": \"state=retrieve&requestId=18\", \"profile\": \"caAuditSigningCert\"}", "profile": "caAuditSigningCert"}" for child.
Nov 19 15:18:29 master.ufreeipa.test certmonger[20493]: 2024-11-19 15:18:29 [20493] Setting "CERTMONGER_CA_NICKNAME" to "dogtag-ipa-ca-renew-agent" for child.
Nov 19 15:18:29 master.ufreeipa.test certmonger[20493]: 2024-11-19 15:18:29 [20493] Setting "CERTMONGER_CA_PROFILE" to "caAuditSigningCert" for child.

However, there is a difference between common constants and what the HSM renewal test uses:
in ipalib/constants.py:

# moved from ipaserver/install/krainstance.py::KRAInstance to avoid duplication
# as per https://pagure.io/freeipa/issue/8795
KRA_TRACKING_REQS = {
    'auditSigningCert cert-pki-kra': 'caAuditSigningCert',
    'transportCert cert-pki-kra': 'caTransportCert',
    'storageCert cert-pki-kra': 'caStorageCert',
}   

and in ipatests/test_integration/test_hsm.py:

        CA_TRACKING_REQS = {
            'ocspSigningCert cert-pki-ca': 'caocspSigningCert',
            'subsystemCert cert-pki-ca': 'casubsystemCert',
            'auditSigningCert cert-pki-ca': 'caauditSigningCert'
        }

note the case and overall difference difference. My understanding is that Dogtag cares about the case sensitivity.

so at least 'auditSigningCert cert-pki-ca': 'caauditSigningCert' needs to be 'auditSigningCert cert-pki-ca': 'caAuditSigningCert' because the profile in Dogtag was a file on a case-sensitive file system.

When we import them in LDAP, that might change (cn and description are case insensitive) but Dogtag's profile system uses java.util.LinkedHashMap<String, Profile> which treats the key case-sensitive.

So I suspect fixing the case in the test should be enough?

Changing the case in the test wouldn't fix the issue (the dictionnary CA_TRACKING_REQS is used only in one place:

for nickname in CA_TRACKING_REQS:

and only the keys are used in the test.

I think the issue is rather related to the trust flags on the updated cert. The script renew_ca_cert should update the trust flags to u,u,Pu for auditSigningCert cert-pki-kra but the content of the NSS database does not reflect that:

certutil -L -d /etc/pki/pki-tomcat/alias/ -h YEOjWmXjSP
Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI
Enter Password or Pin for "YEOjWmXjSP":
YEOjWmXjSP:auditSigningCert cert-pki-kra                     u,u,u        <<<<< HERE
YEOjWmXjSP:auditSigningCert cert-pki-ca                      u,u,u
YEOjWmXjSP:auditSigningCert cert-pki-kra                     u,u,Pu     <<<<< HERE
YEOjWmXjSP:transportCert cert-pki-kra                        u,u,u
YEOjWmXjSP:ocspSigningCert cert-pki-ca                       u,u,u
YEOjWmXjSP:subsystemCert cert-pki-ca                         u,u,u
YEOjWmXjSP:auditSigningCert cert-pki-ca                      u,u,Pu
YEOjWmXjSP:caSigningCert cert-pki-ca                         CTu,Cu,Cu
YEOjWmXjSP:storageCert cert-pki-kra                          u,u,u
YEOjWmXjSP:ocspSigningCert cert-pki-ca                       u,u,u
YEOjWmXjSP:subsystemCert cert-pki-ca                         u,u,u

There are 2 lines for auditSigningCert cert-pki-kra, one with u,u,u and the other with u,u,Pu. That's probably the reason for selftests failure as the most recent cert doesn't have the expected trust flags.

I can add to your analysis. I tested on F41.

certmonger will normally try to remove an existing cert when renewing it. It tries to do that in this case and fails (in simplified output):

2024-12-02 16:26:42 [25144] Imported to token, adding to internal
2024-12-02 16:26:42 [25144] Imported certificate with nickname "auditSigningCert cert-pki-kra" to "NSS FIPS 140-2 Certificate DB".
2024-12-02 16:26:42 [25144] Imported certificate with nickname "auditSigningCert cert-pki-kra" to "ipa_token".
2024-12-02 16:26:42 [25144] Error setting trust on certificate "auditSigningCert cert-pki-kra": SEC_ERROR_TOKEN_NOT_LOGGED_IN.
2024-12-02 16:26:42 [25144] Looking to remove "auditSigningCert cert-pki-kra" from slot "NSS FIPS 140-2 Certificate DB"
2024-12-02 16:26:42 [25144] Certificate not found in "NSS FIPS 140-2 Certificate DB"
2024-12-02 16:26:42 [25144] No certs deleted

I believe that the renew_ca_cert.sh did actually try to modify the trust but it applied to the original cert and was done with it. I ran with a reverse trust, -Pu,u,u, and the trust on the original cert was updated.

In non-FIPS mode the certificate is also not deleted but the trust is correct even though there is the same "Error setting trust...: TOKEN_NOT_LOGGED_IN" message.

Trust is stored in the NSS database on the softoken (either NSS Certificate DB or NSS FIPS 140-2 Certificate DB). It appears that modifying the trust requires the internal token password as well. I don't know if this is new behavior or I've simply forgotten.

So I think this issue may be split between certmonger and ipa. I think that renew_ca_cert.sh should be using the token name along with the nickname (YEOjWmXjSP:auditSigningCert cert-pki-kra) when modifying the trust.

certmonger currently only tracks a single token PIN, the one the certificate is stored on.

Looking through the NSS token code:
- in FIPS mode if you are not logged into the token, you are in FIPS 140-1, else you are in FIPS 140-2 (or later)
- by default the token code assumes you are using FIPS 140-2 if in FIPS at all, so any write operation (e.g. C_SetAttributeValue which is used for flags as well) to a private or a secret key must be logged in.

This is the state since 2017...

master:

• 1e5eb442adb9b6630b95eaf118e65f110d2087ac Dogtag instance: add method to create temp password file
• 7ec0cb4ced0fe5118077a4804a70b928b2a9f442 Cert renewal: update the trust flags for audit cert
• 4146d77d2547160df2df31665dc201a7d3118173 Temp commit: move to fedora 41

ipa-4-12:

• 83d47a2cd88b7ef84509117b7b8ff28cfdb7376e Dogtag instance: add method to create temp password file
• d556ac4611998ead16be4b93ba081b9f48ddb5ee Cert renewal: update the trust flags for audit cert