Adding info from BZ:

I see an issue in

    @property
    def kra_host(self):
        """
        :return:   host
                   as str
        Select our KRA host.
        """
        ldap2 = self.api.Backend.ldap2
        if host_has_service(api.env.ca_host, ldap2, "KRA"):
            return api.env.ca_host
        if api.env.host != api.env.ca_host:
            if host_has_service(api.env.host, ldap2, "KRA"):
                return api.env.host
        host = select_any_master(ldap2, "KRA")
        if host:
            return host
        else:
            return api.env.ca_host

Where at the end, when it doesn't find a suitable KRA server it returns a CA host. It might be the reason why it contacts itself. But that doesn't answer why it doesn't find a suitable server when there are 5. It may be another issue.

We need to log exceptions in:

def select_any_master(ldap2, service='CA'):
    """
    :param ldap2: connection to the local database
    :param service: The service for which we're looking for a master.
    :return:   host as str
    Select any host which is a master for a specified service.
    """
    base_dn = DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'),
                  api.env.basedn)
    filter_attrs = {
         'objectClass': 'ipaConfigObject',
         'cn': service,
         'ipaConfigString': 'enabledService',}
    query_filter = ldap2.make_filter(filter_attrs, rules='&')
    try:
        ent, _trunc = ldap2.find_entries(filter=query_filter, base_dn=base_dn)
        if len(ent):
            entry = random.choice(ent)
            return entry.dn[1].value
    except Exception:
        pass
    return None

As it probably hide the root cause.

The reproduction steps mention that the vault-retrieve is run by another user. I assume it is a standard user without any special permissions. In this case, the root cause is probably the fact the normal user cannot read Enabled services of IPA masters. IPA API has the same permissions as the user thus a search in select_any_master will return 0 entries. Thus it will wrongly default to master defined by api.env.ca_host which is a current master without KRA.

A workaround is to add permission "System: Read Status of Services on IPA Servers" to users which should be able to use Vault. Or a more specific one to not disclose some information.

@abbra api.Command.kra_is_enabled() will return True for KRA-enabled clusters and False for KRA-less clusters.

kra_is_enabled does:

(...)
        base_dn = DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'),
                     self.api.env.basedn)
        filter = '(&(objectClass=ipaConfigObject)(cn=KRA))'
(...)

which builds a non-host specific ldap query like:

# ldapsearch -Y GSSAPI -b cn=masters,cn=ipa,cn=etc,dc=laptop,dc=example,dc=org '(&(objectClass=ipaConfigObject)(cn=KRA))'

which will find KRA hosts if they exist in LDAP, even if the host this is run on is not a KRA master.

Right, so we need to review all code using kra_is_enabled() to see what they expect to receive. If they are checking for the KRA enabled in a cluster, that's one thing. If they expect KRA on the same machine, that's another one. We definitely need to make both queries possible and switch to use the one that is needed in a more explicit way.

So this is indeed an ACI issue.

Changing "pass" from select_any_master(ldap2, service='CA') into:
raise errors.RemoteRetrieveError(reason=_('Unable to determine master for {0}'.format(service)))"
generates as non-admin user:

# ipa vault-retrieve admin_vault --shared --out secret_exported.txt
ipa: ERROR: Unable to determine master for KRA

I tried the following search which is what select_any_master() does as a standard user:

#  ldapsearch -Y GSSAPI -b cn=masters,cn=ipa,cn=etc,dc=laptop,dc=example,dc=org '(&(objectClass=ipaConfigObject)(ipaConfigString=enabledService)(cn=KRA))'
SASL/GSSAPI authentication started
SASL username: testuser@LAPTOP.EXAMPLE.ORG
SASL SSF: 256
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <cn=masters,cn=ipa,cn=etc,dc=laptop,dc=example,dc=org> with scope subtree
# filter: (&(objectClass=ipaConfigObject)(ipaConfigString=enabledService)(cn=KRA))
# requesting: ALL
#
# search result
search: 4
result: 0 Success
# numResponses: 1

yet removing (ipaConfigString=enabledService) returns the only valid KRA server:

#  ldapsearch -Y GSSAPI -b cn=masters,cn=ipa,cn=etc,dc=laptop,dc=example,dc=org '(&(objectClass=ipaConfigObject)(cn=KRA))'
SASL/GSSAPI authentication started
SASL username: testuser@LAPTOP.EXAMPLE.ORG
SASL SSF: 256
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <cn=masters,cn=ipa,cn=etc,dc=laptop,dc=example,dc=org> with scope subtree
# filter: (&(objectClass=ipaConfigObject)(cn=KRA))
# requesting: ALL
#
# KRA, idm0.laptop.example.org, masters, ipa, etc, laptop.example.org
dn: cn=KRA,cn=idm0.laptop.example.org,cn=masters,cn=ipa,cn=etc,dc=laptop,dc=ex
 ample,dc=org
objectClass: nsContainer
objectClass: ipaConfigObject
objectClass: top
cn: KRA
# search result
search: 4
result: 0 Success
# numResponses: 2
# numEntries: 1

Will revisit after https://github.com/freeipa/freeipa/pull/2144 is merged.
I'll check the implications of adding "System: Read Status of Services on IPA Servers" to all users. However, as noted above, the information is already available either through a custom LDAP search or by using DNS SRV records.

master:

• a709da674833a0c44cdca0c3431313a44bacb9dd Add a shared-vault-retrieve test
• dd0490e1d8f2021b3fad8f283a73bd7e63e2178c Add a "Find enabled services" ACI in 20-aci.update so that all users can find IPA servers and services. ACI suggested by Christian Heimes.

master:

• 58053b27475a6fe6f84b7cb85229938e7839af62 TestBackupAndRestoreWithReplica needs 2 replicas

ipa-4-6:

• e0207fad2239227775ea3e914bdf71c58af6fd45 Add a shared-vault-retrieve test
• 212215c810a9ed52276c94f44a98f20cb94b01e7 Add a "Find enabled services" ACI in 20-aci.update so that all users can find IPA servers and services. ACI suggested by Christian Heimes.

ipa-4-7:

• 858859187a1353cbaa893642cc7b27f9f644b18b Add a shared-vault-retrieve test
• d57d97ea7f911e18ac75d532e19833c4efaafa96 Add a "Find enabled services" ACI in 20-aci.update so that all users can find IPA servers and services. ACI suggested by Christian Heimes.