In a process of upgrading from F29 to F30 to F31, I found that ipactl start fails with a message
ipactl start
# ipactl start Existing service file detected! Assuming stale, cleaning and proceeding Starting Directory Service Failed to read data from service file: Failed to get list of services to probe status! Configured hostname 'ipa.some.zone' does not match any master server in LDAP: ipa.some.zone Shutting down
Apparently, it is an issue with newer 389-ds version that treats unindexed search filters with empty result set, even for searches as cn=Directory Manager. When looking at logconf.pl output, the following search is unindexed:
cn=Directory Manager
logconf.pl
- Unindexed Filter: (&(objectclass=ipaconfigobject)(|(ipaconfigstring=enabledservice)(ipaconfigstring=hiddenservice))) (occurrances 2)
I thought it is a problem with ipaConfigString index. However, when I ran
ipaConfigString
dsconf SOME-ZONE backend index reindex userRoot
there were no problems with this search anymore.
Either we need to make sure we are reindexing userRoot during upgrade or see what else should be done during the upgrade.
@abbra, are ipaConfigString and objectclass indexed in 'eq' ? I am surprise it is unindexed (notes=A) as I thought objtectclass was indexed 'eq'. Is ipaConfigString defined in the schema ?
Yes, ipaConfigString is indexed. The index on the attribute is fairly new. I added it in in bug https://pagure.io/freeipa/issue/7792 about a year ago.
Before indexing the result looks like this:
[26/Nov/2019:08:47:13.991076428 +0100] conn=49 op=1 SRCH base="cn=ipa.some.zone,cn=masters,cn=ipa,cn=etc,dc=some,dc=zone" scope=2 filter="(&(objectClass=ipaConfigObject)(|(ipaConfigString=enabledSe rvice)(ipaConfigString=hiddenService)))" attrs="ipaConfigString cn" [26/Nov/2019:08:47:13.991899596 +0100] conn=49 op=1 RESULT err=0 tag=101 nentries=11 etime=0.001133041 notes=U details="Partially Unindexed Filter"
after indexing it looks like this:
[26/Nov/2019:08:47:50.801728372 +0100] conn=4 op=1 SRCH base="cn=ipa.some.zone,cn=masters,cn=ipa,cn=etc,dc=some,dc=zone" scope=2 filter="(&(objectClass=ipaConfigObject)(|(ipaConfigString=enabledService)(ipaConfigString=hiddenService)))" attrs="ipaConfigString cn" [26/Nov/2019:08:47:50.804219567 +0100] conn=4 op=1 RESULT err=0 tag=101 nentries=11 etime=0.002662216
Is ipaConfigString defined in the schema ? Are you testing DS 1.4.2 or 1.4.3 ?
I think the problem is a side effect of https://pagure.io/389-ds-base/issue/50349 (filter component validation against schema).
in DS 1.4.3, a filter component containing an unknown (not defined in the schema) attribute matches 0 candidates. The returned entries can be a partial set of the matching entries. A search result text and a message in the access log are provided to warn the client to fix the filter.
In DS 1.4.2 (since https://pagure.io/389-ds-base/c/32be56b5), the default behavior is softer as it returns the complete set of matching entries (in addition to result text/access log).
If ipaConfigString is not defined then the solution is to define it. You may softer the filter validation with the 1.4.2 dn: cn=config nsslapd-verify-filter-schema: warn-invalid
Yes, it is defined in the IPA schema (60ipaconfig.ldif) since IPA v2 times (years ago). The index for it was added relatively recently but IPA upgrade code should have run indexing. Only after I did an explicit index task, it stopped showing unindexed search.
However, the note is 'note=U', not 'note=A' here. And I don't think the ticket 50349 applies here because the attribute is known in the schema since the original installation of this deployment.
note=U means the server managed to do an indexed search even if the original filter was unindexed. We can see that with/without the index the response time is fast. For what reason do you think the original failure is related to that search ?
Because that's exactly the search failing. It is ipactl tool, it does search for enabled or hidden searches on the system and it only triggers to show the message
ipactl
Configured hostname 'ipa.some.zone' does not match any master server in LDAP: ipa.some.zone
if there is an exception errors.NotFound returned from the LDAP search I listed.
errors.NotFound
@abbra, I fail to reproduce the failure. Do you have a way to reproduce it so that I can connect to the machine ?
Something I do not understand, the following message is logged on errors.NotFound
But the suspected SRCH gets err=0 (both before and after reindex) so why is it related to this message ?
Because that's exactly the message displayed in ipactl tool, as I said in the previous comment (I looked at the code ;) ).
I cannot reproduce it anymore, sadly.
I'm closing it as there is no reproducer anymore.
Metadata Update from @abbra: - Issue close_status updated to: insufficientinfo - Issue status updated to: Closed (was: Open)