#6342 Multiple CA Certificates after mistaken ipa-cacert-manage
Closed: Invalid Opened by andybrucenet.

Overview

Two CA certs show up in the Web UI after I mistakenly used {{{ipa-cacert-manage renew}}} on a domain controller. I cannot edit the second CA cert, and when I try to join a client to the domain I receive an error because {{{ipa-client-install}} attempts to add both CA certs to the NSS database(s) and the second CA cert fails.

Desired Outcome

Instructions on how to delete the second CA cert from Dogtag and from LDAP.

Context

FreeIPA Version (client and server): {{{4.2.0-15.0.1.el7.centos.19}}}

Operating System: {{{CentOS Linux release 7.2.1511 (Core)}}}

{{{uname -a}}} output: {{{Linux [HOSTNAME] 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux}}}

Environment: KVM virtual machines (both domain controllers and clients)

FreeIPA history: Originally had 3.0.0 (CentOS 6); created new VM with CentOS 7 and latest FreeIPA - used {{{ipa-replica-install}}} as documented https://www.freeipa.org/page/Howto/Migration . Then added second domain controller and decommissioned the original 3.0 controllers. The domain is working well - all policies work, users can login to hosts, LDAP authentication from other applications (e.g. Redmine) works fine.

I think I caused this situation; I mistakenly ran {{{ipa-cacert-manage renew}}} and ended up with two valid CA certificates in LDAP and Dogtag.

Description

There are two problems:

  • Blocker: Unable to add new clients to the domain. Error is:

    2016-09-20T15:48:40Z DEBUG trying to retrieve CA cert via LDAP from lvinfripa100.hlsdev.local
    2016-09-20T15:48:40Z DEBUG flushing ldap://lvinfripa100.hlsdev.local:389 from SchemaCache
    2016-09-20T15:48:40Z DEBUG retrieving schema for SchemaCache url=ldap://lvinfripa100.hlsdev.local:389 conn=
    2016-09-20T15:48:41Z INFO Successfully retrieved CA cert
    Subject: CN=Certificate Authority,O=HLSDEV.LOCAL
    Issuer: CN=Certificate Authority,O=HLSDEV.LOCAL
    Valid From: Wed Nov 19 18:13:47 2014 UTC
    Valid Until: Sun Nov 19 18:13:47 2034 UTC

    Subject:     CN=Certificate Authority,O=HLSDEV.LOCAL
    Issuer:      CN=Certificate Authority,O=HLSDEV.LOCAL
    Valid From:  Wed Sep 07 00:32:08 2016 UTC
    Valid Until: Sun Sep 07 00:32:08 2036 UTC
    

    2016-09-20T15:48:41Z DEBUG Starting external process
    2016-09-20T15:48:41Z DEBUG args='/usr/sbin/ipa-join' '-s' 'lvinfripa100.hlsdev.local' '-b' 'dc=hlsdev,dc=local' '-h' 'c07.hlsdev.local' '-f'

    [...time passes...]

    2016-09-20T15:48:46Z DEBUG retrieving schema for SchemaCache url=ldap://lvinfripa100.hlsdev.local:389 conn=
    2016-09-20T15:48:46Z DEBUG Adding CA certificates to the IPA NSS database.
    2016-09-20T15:48:46Z DEBUG Starting external process
    2016-09-20T15:48:46Z DEBUG args='/usr/bin/certutil' '-d' '/etc/ipa/nssdb' '-A' '-n' 'HLSDEV.LOCAL IPA CA' '-t' 'CT,C,C'
    2016-09-20T15:48:46Z DEBUG Process finished, return code=0
    2016-09-20T15:48:46Z DEBUG stdout=
    2016-09-20T15:48:46Z DEBUG stderr=
    2016-09-20T15:48:46Z DEBUG Starting external process
    2016-09-20T15:48:46Z DEBUG args='/usr/bin/certutil' '-d' '/etc/ipa/nssdb' '-A' '-n' 'HLSDEV.LOCAL IPA CA' '-t' 'CT,C,C'
    2016-09-20T15:48:46Z DEBUG Process finished, return code=255
    2016-09-20T15:48:46Z DEBUG stdout=
    2016-09-20T15:48:46Z DEBUG stderr=certutil: could not add certificate to token or database: SEC_ERROR_ADDING_CERT: Error adding certificate to database.

    2016-09-20T15:48:46Z ERROR Failed to add HLSDEV.LOCAL IPA CA to the IPA NSS database.

  • Unable to view / edit the second CA cert in the Web UI. The following attached files demonstrate the problem:

  • FreeIPA-2-CA-Certs-10.png - Shows the two certificates
  • FreeIPA-2-CA-Certs-20.png - Error popup dialog after I click on the second certificate
  • FreeIPA-2-CA-Certs-30.png - Error displayed after clearing the popup.

I can view the initial (older) certificate in the Web UI without a problem.

I can see both certificates in LDAP:

-bash-4.2$ ldapsearch -x -b "cn=certificates,cn=ipa,cn=etc,dc=hlsdev,dc=local" 
# extended LDIF
#
# LDAPv3
# base <cn=certificates,cn=ipa,cn=etc,dc=hlsdev,dc=local> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# certificates, ipa, etc, hlsdev.local
dn: cn=certificates,cn=ipa,cn=etc,dc=hlsdev,dc=local
cn: certificates
objectClass: nsContainer
objectClass: top
# HLSDEV.LOCAL IPA CA, certificates, ipa, etc, hlsdev.local
dn: cn=HLSDEV.LOCAL IPA CA,cn=certificates,cn=ipa,cn=etc,dc=hlsdev,dc=local
ipaConfigString: ipaCa
ipaCertIssuerSerial: CN=Certificate Authority,O=HLSDEV.LOCAL;1
ipaCertIssuerSerial: CN=Certificate Authority,O=HLSDEV.LOCAL;536739841
ipaKeyTrust: trusted
cACertificate;binary:: MIIDlzCCAn+gAwIBAgIBATANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQ
 KEwxITFNERVYuTE9DQUwxHjAcBgNVBAMTFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNDExMTkx
 ODEzNDdaFw0zNDExMTkxODEzNDdaMDcxFTATBgNVBAoTDEhMU0RFVi5MT0NBTDEeMBwGA1UEAxMVQ
 2VydGlmaWNhdGUgQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA+IE4Vv
 jAdFUkKdv7eQjhD8R/USVZb/cWgGCDIKN7+iF1cmM68/Cr/9Z/dI0eCf83sZwzL2i6UVt54ezr0UF
 ULKCVU3xolmEBt8bZVIT59/3M2ycgi27o+z7jL4AJWS14Ch6QxNSLY01y7/Pky+iQx/P14QNts57t
 clarfDUXOHuKQukcUIYx6ADoTGTiiVuu4mYIZpFWz8szuLpXAEb26U0YipA285gZXPBW9nYz7hIan
 y1DIIak5sohBlsZL/50rzpjKvS40y0CcrlTeqXCDqtq2AcJToSqt0MBcja/FHfi608ZEUOdY+BASQ
 MjHC3jKJYEK4S0qxug5IoTdZcFdwIDAQABo4GtMIGqMB8GA1UdIwQYMBaAFOVisidTbxSPSUcptad
 UA6leroN8MA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgHGMB0GA1UdDgQWBBTlYrInU28U
 j0lHKbWnVAOpXq6DfDBHBggrBgEFBQcBAQQ7MDkwNwYIKwYBBQUHMAGGK2h0dHA6Ly9sdmluZnJpc
 GExMDAuaGxzZGV2LmxvY2FsOjgwL2NhL29jc3AwDQYJKoZIhvcNAQELBQADggEBAAybcGZxx5WfSj
 wLKGZ3FV8WBVlkBexbeORgOYc7YuXa1jW97BsbnEh/O/vImFFIeTnj9wm2u8R0HUJtf9nV0A+/iNb
 PmGpAfUia9Bu4/QJSrInnUbltDExRrhdJPAiQgoxskyptPaf2+PchW3jvTjXLyuBIPPqprOrbZq3o
 dU6imqXeVUtyE1s6ppVfygag2LPIq10Q/mKhMR+DZcZy/ZTKgsYgBipsaiR1YBBPmKmyU49lkCnRJ
 pCbzpwaWSjBNN96UBivwlJ/2cjrXfSHai4IyNykYBjUQcRFy4pKIskFcKSSEWvFHdG33ZMVPzsSfV
 1QE0UleyAb8tyonQcMung=
cACertificate;binary:: MIIDmjCCAoKgAwIBAgIEH/4AATANBgkqhkiG9w0BAQsFADA3MRUwEwY
 DVQQKEwxITFNERVYuTE9DQUwxHjAcBgNVBAMTFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNjA5
 MDcwMDMyMDhaFw0zNjA5MDcwMDMyMDhaMDcxFTATBgNVBAoMDEhMU0RFVi5MT0NBTDEeMBwGA1UEA
 wwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA+I
 E4VvjAdFUkKdv7eQjhD8R/USVZb/cWgGCDIKN7+iF1cmM68/Cr/9Z/dI0eCf83sZwzL2i6UVt54ez
 r0UFULKCVU3xolmEBt8bZVIT59/3M2ycgi27o+z7jL4AJWS14Ch6QxNSLY01y7/Pky+iQx/P14QNt
 s57tclarfDUXOHuKQukcUIYx6ADoTGTiiVuu4mYIZpFWz8szuLpXAEb26U0YipA285gZXPBW9nYz7
 hIany1DIIak5sohBlsZL/50rzpjKvS40y0CcrlTeqXCDqtq2AcJToSqt0MBcja/FHfi608ZEUOdY+
 BASQMjHC3jKJYEK4S0qxug5IoTdZcFdwIDAQABo4GtMIGqMB8GA1UdIwQYMBaAFOVisidTbxSPSUc
 ptadUA6leroN8MB0GA1UdDgQWBBTlYrInU28Uj0lHKbWnVAOpXq6DfDAPBgNVHRMBAf8EBTADAQH/
 MA4GA1UdDwEB/wQEAwIBxjBHBggrBgEFBQcBAQQ7MDkwNwYIKwYBBQUHMAGGK2h0dHA6Ly9sdmluZ
 nJpcGExMDAuaGxzZGV2LmxvY2FsOjgwL2NhL29jc3AwDQYJKoZIhvcNAQELBQADggEBACPBAoaX2w
 /rYFn4TLmiyNCngLZrkvfbldIsuFUIwvzExUqYLasAZpMhphB2digy24k8XAAsnQ/1hAZ7N/r8tRN
 +MRToMeWMmrZj8Ca5Lv6OyD/iVslJTk3wwSE0gXyjSH4epbCG8/FUKFlbaOBM3XgDZizFYATBpbA3
 aGayR4jITMzNNHcGDrSUdXwjqGcJKiTGfBOjthPRRdmJXWki8dXCmRxDgIg4eUs1mY0cmAwFJz9rm
 y+iJzphWz3QemPg32DAWo/jZ3RNwvexT21Rda0LjxDoez1rC6ZG8TASdgGY9oJCPjy91LMgS+gr4H
 3zK9LNtQvicRsAbWttKpxKYuU=
ipaPublicKey:: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA+IE4VvjAdFUkKdv7eQj
 hD8R/USVZb/cWgGCDIKN7+iF1cmM68/Cr/9Z/dI0eCf83sZwzL2i6UVt54ezr0UFULKCVU3xolmEB
 t8bZVIT59/3M2ycgi27o+z7jL4AJWS14Ch6QxNSLY01y7/Pky+iQx/P14QNts57tclarfDUXOHuKQ
 ukcUIYx6ADoTGTiiVuu4mYIZpFWz8szuLpXAEb26U0YipA285gZXPBW9nYz7hIany1DIIak5sohBl
 sZL/50rzpjKvS40y0CcrlTeqXCDqtq2AcJToSqt0MBcja/FHfi608ZEUOdY+BASQMjHC3jKJYEK4S
 0qxug5IoTdZcFdwIDAQAB
ipaCertSubject: CN=Certificate Authority,O=HLSDEV.LOCAL
objectClass: ipaCertificate
objectClass: pkiCA
objectClass: ipaKeyPolicy
objectClass: top
cn: HLSDEV.LOCAL IPA CA
ipaKeyExtUsage: 1.3.6.1.5.5.7.3.1
ipaKeyExtUsage: 1.3.6.1.5.5.7.3.2
ipaKeyExtUsage: 1.3.6.1.5.5.7.3.3
ipaKeyExtUsage: 1.3.6.1.5.5.7.3.4
# search result
search: 2
result: 0 Success
# numResponses: 3
# numEntries: 2

It is certificate {{{536739841}}} that is the problematic one I wish to remove.

Workaround

Modifying {{{ipa-client-install}} to avoid failure:

[root@c07 nssdb]# diff ~/ipa-client-install-orig $(which ipa-client-install)
2862c2862
<             return CLIENT_INSTALL_ERROR
---
>             #return CLIENT_INSTALL_ERROR
2877c2877
<             return CLIENT_INSTALL_ERROR
---
>             #return CLIENT_INSTALL_ERROR

After making the above patch - on the newly-added client here is the CA stored:

[root@c07 nssdb]# certutil -L -d /etc/ipa/nssdb 'HLSDEV.LOCAL IPA CA'
Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI
HLSDEV.LOCAL IPA CA                                          CT,C,C

This is the original CA cert (not the newer one created by {{{ipa-cacert-manage renew}}}


Screenshots of CA Cert editing problems
FreeIPA-2-CA-Certs.tar

I was not able to reproduce the issue. My steps:

  • install ipa using ipa-server-install on CentOS Linux release 7.2.1511
  • prepare a replica file using ipa-replica-prepare
  • install ipa on the replica using ipa-replica-install (same CentOS)
  • run ipa-cacert-manage renew on the master

This creates a new CA cert and we can see 2 certs in the GUI, as well as in LDAP

  • install a client

The client installation succeeds.

I suspect that the issue comes from the content of your certificate with serial number 536739841 because trying to add this specific CA with ipa-cacert-manage install gives the following error:

Installing CA certificate, please wait
Not a valid CA certificate: (SEC_ERROR_UNKNOWN_ISSUER) Peer's Certificate issuer is not recognized. (visit http://www.freeipa.org/page/Troubleshooting for troubleshooting guide)

while it succeeds for the first one.
We need to understand what's wrong with this cert.

Closing due to inactivity.

Metadata Update from @andybrucenet:
- Issue assigned to someone
- Issue set to the milestone: 0.0 NEEDS_TRIAGE

Metadata