freeipa

#9327 Issue when upgrading from freeIPA 4.9.8 to freeIPA 4.9.10 under RockyLinux 8
Closed: invalid by nunohiggs. Opened by nunohiggs.

Closed: invalid
Reopen Issue

Issue

Cannot upgrade from freeIPA 4.9.8 to freeIPA 4.9.10

Steps to Reproduce

  1. Do a dnf update under rocky 8.6 to 8.7
  2. Wait until ipa-server-upgrade runs
  3. Fails

Actual behavior

IPA upgrade script fails do upgrade with error in log

Expected behavior

Upgrade to run completly

Version/Release/Distribution

$ rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server

package freeipa-server is not installed
package freeipa-client is not installed
ipa-server-4.9.10-9.module+el8.7.0+1119+c1de6b59.x86_64
ipa-client-4.9.10-9.module+el8.7.0+1119+c1de6b59.x86_64
389-ds-base-1.4.3.30-6.module+el8.7.0+1060+852806e7.x86_64
package pki-ca is not installed
krb5-server-1.18.2-22.el8_7.x86_64

Additional info:

Fails under Executing upgrade plugin: fix_kra_people_entry.
I think i might have a KRA broken install. How can i uninstall KRA to make sure that this is not the culprit?

Logs:

2023-02-01T18:28:58Z DEBUG raw: fix_kra_people_entry
2023-02-01T18:28:58Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:28:58Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
2023-02-01T18:28:58Z DEBUG Starting external process
2023-02-01T18:28:58Z DEBUG args=['pki-server', 'subsystem-show', 'kra']
2023-02-01T18:28:58Z DEBUG Process finished, return code=0
2023-02-01T18:28:58Z DEBUG stdout= Subsystem ID: kra
Instance ID: pki-tomcat
Enabled: True

2023-02-01T18:28:58Z DEBUG stderr=
2023-02-01T18:28:58Z DEBUG Destroyed connection context.ldap2_140239616011288
2023-02-01T18:28:58Z ERROR Upgrade failed with no such entry
2023-02-01T18:28:58Z DEBUG Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1085, in error_handler
yield
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1585, in find_entries
raise e
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1545, in find_entries
result = self.conn.result3(id, 0)
File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 767, in result3
resp_ctrl_classes=resp_ctrl_classes
File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 774, in result4
ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 340, in _ldap_call
reraise(exc_type, exc_value, exc_traceback)
File "/usr/lib64/python3.6/site-packages/ldap/compat.py", line 46, in reraise
raise exc_value
File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 324, in _ldap_call
result = func(args,*kwargs)
ldap.NO_SUCH_OBJECT: {'msgtype': 101, 'msgid': 484, 'result': 32, 'desc': 'No such object', 'ctrls': [], 'matched': 'o=ipaca'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipaserver/install/upgradeinstance.py", line 291, in upgrade
self.modified = (ld.update(self.files) or self.modified)
File "/usr/lib/python3.6/site-packages/ipaserver/install/ldapupdate.py", line 998, in update
self._run_updates(all_updates)
File "/usr/lib/python3.6/site-packages/ipaserver/install/ldapupdate.py", line 961, in _run_updates
self._run_update_plugin(update['plugin'])
File "/usr/lib/python3.6/site-packages/ipaserver/install/ldapupdate.py", line 938, in _run_update_plugin
restart_ds, updates = self.api.Updaterplugin_name
File "/usr/lib/python3.6/site-packages/ipalib/frontend.py", line 1495, in __call
return self.execute(options)
File "/usr/lib/python3.6/site-packages/ipaserver/install/plugins/fix_kra_people_entry.py", line 33, in execute
entry = self.api.Backend.ldap2.get_entry(krainstance.KRA_AGENT_DN)
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1941, in get_entry
dn, attrs_list, time_limit, size_limit, get_effective_rights
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1642, in get_entry
size_limit=size_limit, get_effective_rights=get_effective_rights,
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1454, in get_entries
kwargs)
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1592, in find_entries
break
File "/usr/lib64/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1095, in error_handler
raise errors.NotFound(reason=arg_desc or 'no such entry')
ipalib.errors.NotFound: no such entry

2023-02-01T18:28:58Z DEBUG Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1085, in error_handler
yield
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1585, in find_entries
raise e
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1545, in find_entries
result = self.conn.result3(id, 0)
File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 767, in result3
resp_ctrl_classes=resp_ctrl_classes
File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 774, in result4
ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 340, in _ldap_call
reraise(exc_type, exc_value, exc_traceback)
File "/usr/lib64/python3.6/site-packages/ldap/compat.py", line 46, in reraise
raise exc_value
File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 324, in _ldap_call
result = func(args,*kwargs)
ldap.NO_SUCH_OBJECT: {'msgtype': 101, 'msgid': 484, 'result': 32, 'desc': 'No such object', 'ctrls': [], 'matched': 'o=ipaca'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipaserver/install/upgradeinstance.py", line 291, in upgrade
self.modified = (ld.update(self.files) or self.modified)
File "/usr/lib/python3.6/site-packages/ipaserver/install/ldapupdate.py", line 998, in update
self._run_updates(all_updates)
File "/usr/lib/python3.6/site-packages/ipaserver/install/ldapupdate.py", line 961, in _run_updates
self._run_update_plugin(update['plugin'])
File "/usr/lib/python3.6/site-packages/ipaserver/install/ldapupdate.py", line 938, in _run_update_plugin
restart_ds, updates = self.api.Updaterplugin_name
File "/usr/lib/python3.6/site-packages/ipalib/frontend.py", line 1495, in __call
return self.execute(options)
File "/usr/lib/python3.6/site-packages/ipaserver/install/plugins/fix_kra_people_entry.py", line 33, in execute
entry = self.api.Backend.ldap2.get_entry(krainstance.KRA_AGENT_DN)
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1941, in get_entry
dn, attrs_list, time_limit, size_limit, get_effective_rights
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1642, in get_entry
size_limit=size_limit, get_effective_rights=get_effective_rights,
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1454, in get_entries
kwargs)
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1592, in find_entries
break
File "/usr/lib64/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1095, in error_handler
raise errors.NotFound(reason=arg_desc or 'no such entry')
ipalib.errors.NotFound: no such entry

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py", line 635, in start_creation
run_step(full_msg, method)
File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py", line 621, in run_step
method()
File "/usr/lib/python3.6/site-packages/ipaserver/install/upgradeinstance.py", line 299, in __upgrade
raise RuntimeError(e)
RuntimeError: no such entry

2023-02-01T18:28:58Z DEBUG [error] RuntimeError: no such entry
2023-02-01T18:28:58Z DEBUG [cleanup]: stopping directory server
2023-02-01T18:28:58Z DEBUG Destroyed connection context.ldap2_140239648239856
2023-02-01T18:28:58Z DEBUG Starting external process
2023-02-01T18:28:58Z DEBUG args=['/bin/systemctl', 'stop', 'dirsrv@NET-XPTO.service']
2023-02-01T18:29:04Z DEBUG Process finished, return code=0
2023-02-01T18:29:04Z DEBUG stdout=
2023-02-01T18:29:04Z DEBUG stderr=
2023-02-01T18:29:04Z DEBUG Stop of dirsrv@NET-XPTO.service complete
2023-02-01T18:29:04Z DEBUG step duration: dirsrv __stop_instance 6.38 sec
2023-02-01T18:29:04Z DEBUG [cleanup]: restoring configuration
2023-02-01T18:29:04Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:04Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:04Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:04Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2023-02-01T18:29:05Z DEBUG step duration: dirsrv __restore_config 0.05 sec
2023-02-01T18:29:05Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2023-02-01T18:29:05Z DEBUG File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 180, in execute
return_value = self.run()
File "/usr/lib/python3.6/site-packages/ipaserver/install/ipa_server_upgrade.py", line 56, in run
raise admintool.ScriptError(str(e))

2023-02-01T18:29:05Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: ('IPA upgrade failed.', 1)
2023-02-01T18:29:05Z ERROR ('IPA upgrade failed.', 1)
2023-02-01T18:29:05Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Hi @nunohiggs
your server is missing the entry uid=ipakra,ou=people,o=kra,o=ipaca.

You can manually re-create it with the following ldif file:

$ ldapmodify -D cn=directory\ manager -W <<EOF
dn: uid=ipakra,ou=people,o=kra,o=ipaca
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: cmsuser
uid: ipakra
sn: IPA KRA User
cn: IPA KRA User
usertype: undefined
usercertificate::MII...Jw==
EOF

You need to replace the usercertificate value with the one from /var/lib/ipa/ra-agent.pem (without the header and footer, and everything concatenated in a single line).

Then, calling ipa-server-upgrade should fill the missing description attribute.

Hi @frenaud

Thanks for your quick reply.
I think it might be deeper than that. The entire o=kra is missing:

Capture.JPG

As expected when i run the ldiff i get stuck with:

Enter LDAP Password:
adding new entry "uid=ipakra,ou=people,o=kra,o=ipaca"
ldap_add: No such object (32)
matched DN: o=ipaca

Defaulting to the question, can i force-reinstall KRA without having to recreate the entire ipa environment?

@nunohiggs
unfortunately there is no uninstaller for the KRA only.
What is your exact situation? Is this server your only IPA server? Or do you have other replicas that may have the missing data?

Hi @frenaud

This is 1/3 of my ipa cluster. All of the remaining 2/3 do not have any information regarding KRA in that dirserv and they all have the same issue, cannot update due to the error that i pasted at the start of the thread.
Edited by nunohiggs

Hi @nunohiggs
I can give you some hints but the procedure is definitely not supported. Under your own responsibility...
Start by making a backup (please refer to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/preparing_for_disaster_recovery_with_identity_management/index)

  • Delete the entry cn=KRA,cn=hostname_fqdn,cn=masters,cn=ipa,cn=etc,dc=basedn (replace hostname_fqdn and basedn with the right values).

  • Stop tracking the KRA-related certificates (storageCert cert-pki-kra, transportCert cert-pki-kra and auditSigningCert cert-pki-kra):
    getcert stop-tracking -d /etc/pki/pki-tomcat/alias -n 'storageCert cert-pki-kra'
    getcert stop-tracking -d /etc/pki/pki-tomcat/alias -n 'transportCert cert-pki-kra'
    getcert stop-tracking -d /etc/pki/pki-tomcat/alias -n 'auditSigningCert cert-pki-kra'

  • Call pkidestroy -i pki-tomcat -s KRA

  • Delete the entry cn=kra,dc=basedn (replace basedn with the right value) and its subentries (use ldapdelete -r for recursive delete). Important: this will remove any secret stored in the vaults

With those steps you may be able to re-run ipa-server-upgrade but there is no guarantee that you would be able to reinstall a KRA later on.

Thanks! Will try. I was thinking of something even jankier....

Hi @frenaud

You were right. I was able to upgrade. But KRA is broken and i cannot reinstall it.
I found this https://www.freeipa.org/page/V4/FreeIPA_to_FreeIPA_Migration#UI

Is it an on-going project or do you know a better way to do this?
I think that this ipa cluster is done for and its running on borrowed time.

Thanks so much for your time!
Edited by nunohiggs

Hi @nunohiggs
FreeIPA already provides a CLI (ipa migrate-ds) allowing to migrate users and groups from any LDAP server to FreeIPA (see this doc).
It doesn't migrate the full content of IPA (for instance hbac rules, sudo rules... would not be migrated) but you can export the data from your existing server into a ldif and import it into the new server. Some LDIF editing may be required, the clients need to be re-enrolled but it may help you in the process.

The FreeIPA-to-FreeIPA migration project is still at an early stage, there is a proposed design available at https://freeipa.readthedocs.io/en/latest/designs/ipa_to_ipa_migration.html but no code yet.

Hi @frenaud,

Thanks so much for your guidance and help. This will work perfectly. I will start to construct a new ipa cluster, and then do a scripted unattended un-enrollment/re-enrollment of the clients.

Again, thanks for your time.

Metadata Update from @nunohiggs:
- Issue close_status updated to: invalid
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.