#8275 Support systemd-resolved
Closed: fixed by cheimes. Opened by abbra.

Fedora 33 is planning to enable systemd-resolved by default. The change is described in detail in https://fedoraproject.org/wiki/Changes/systemd-resolved

FreeIPA needs to be ready for systemd-resolved operation by Fedora 33. FreeIPA does modification of resolv.conf when installed with DNS and it also expects to fix resolv.conf on the test systems in own integration tests. Since resolv.conf would become a symlink some of the expectations might not be true anymore.

We need to find out what actual tasks need to be done to preserve current FreeIPA behavior with regards to DNS and DNSSEC operations.


Metadata Update from @abbra:
- Issue set to the milestone: FreeIPA 4.8.7

It might be sufficient to drop in a config file and reload systemd-resolved

# /etc/systemd/resolved.conf.d/ipa.conf
[Resolve]
DNS=127.0.0.1
Domains=~.

It might also be advisable to configure dnssec trust anchors and add our DNSSEC KSK.

Metadata Update from @cheimes:
- Issue set to the milestone: None (was: FreeIPA 4.8.7)

Metadata Update from @cheimes:
- Issue priority set to: important
- Issue set to the milestone: FreeIPA 4.8.7

Rawhide doesn't use systemd resolved yet. We have to wait until the new feature is implemented.

See #8375 for a related issue

Metadata Update from @cheimes:
- Issue set to the milestone: None (was: FreeIPA 4.8.7)

Metadata Update from @abbra:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1880628

Fedora 33 now uses systemd resolved and we've got bug https://bugzilla.redhat.com/show_bug.cgi?id=1880628 as accepted exception to fix the behavior.

Metadata Update from @abbra:
- Issue set to the milestone: FreeIPA 4.8

--auto-forwarders is not compatible with systemd-resolved yet. Internally the flag uses dnspython to get a list of name servers with dns.resolver.get_default_resolver().nameservers / get_ipa_resolver() since #8383. On Fedora 33 this returns the stub resolver ['127.0.0.53']. FreeIPA either needs to parse the output of resolvectl dns (ugly) or use D-Bus to query the DNS resolver information for global scope and all link scopes.

code to detect if systemd-resolved manages /etc/resolv.conf

import os
from ipaplatform.paths import paths
def detect_resolve1_resolv_conf():
    """Detect if /etc/resolv.conf is managed by systemd-resolved
    See man(5) NetworkManager.conf
    """
    systemd_resolv_conf = {
        "/run/systemd/resolve/stub-resolv.conf",
        "/run/systemd/resolve/resolv.conf",
        "/lib/systemd/resolv.conf",
        "/usr/lib/systemd/resolv.conf",
    }
    try:
        dest = os.readlink(paths.RESOLV_CONF)
    except OSError:
        # not a link
        return False
    # convert path relative to /etc/resolv.conf to abs path
    dest = os.path.normpath(
        os.path.join(os.path.dirname(paths.RESOLV_CONF), dest)
    )
    return dest in systemd_resolv_conf

code to get upstream nameservers from systemd-resolved's D-Bus interface

import socket
import dbus
DBUS_RESOLVE1_NAME = "org.freedesktop.resolve1"
DBUS_RESOLVE1_PATH = "/org/freedesktop/resolve1"
DBUS_RESOLVE1_MANAGER_IF = 'org.freedesktop.resolve1.Manager'
DBUS_PROPERTY_IF = 'org.freedesktop.DBus.Properties'
# netlink interface index for resolve1 global settings and loopback
IFINDEX_GLOBAL = 0
IFINDEX_LOOPBACK = 1
def get_resolve1_nameservers():
    """Get list of DNS forwarders from systemd-resolved
    :return: list of tuples (ifindex, ipaddress)
    """
    bus = dbus.SystemBus()
    try:
        resolve1 = bus.get_object(DBUS_RESOLVE1_NAME, DBUS_RESOLVE1_PATH)
        prop_if = dbus.Interface(resolve1, DBUS_PROPERTY_IF)
        dns_prop = prop_if.Get(DBUS_RESOLVE1_MANAGER_IF, "DNSEx")
    finally:
        bus.close()
    results = []
    for ifindex, af, dns_arr, port, sniname in dns_prop:
        if af not in {socket.AF_INET, socket.AF_INET6}:
            # neither IPv4 nor IPv6
            continue
        if port not in {0, 53} or sniname:
            # non-default port, non-standard port, or SNI name configuration
            # for DNS over TLS, e.g. 1.2.3.4:9953#example.com
            continue
        # netlink interface index, see socket.if_nameindex()
        ifindex = int(ifindex)
        dnsip = socket.inet_ntop(af, bytes(dns_arr))
        results.append((ifindex, dnsip))
    return results

```

Using socket.if_indextoname() you can get back the interface name:

....
     results.append((ifindex, socket.if_indextoname(ifindex), dnsip))

Metadata Update from @cheimes:
- Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/5125
- Issue assigned to cheimes
- Issue priority set to: critical (was: important)

master:

  • 96edff0b8c526763d75dc45d26841dc2cd1a70cf Add helpers for resolve1 and nameservers
  • e64f27fdf8970c22052ec010f3b7bada084c743b Configure NetworkManager to use systemd-resolved
  • 528c519cb5a30df691f00edba6ef6f3a6ab7df56 Use new API for auto-forwarders
  • d12f1b4b39ef514c2bcb75b55c3f300767aca64d Configure systemd-resolved to use IPA's BIND
  • 79b9982b86b68c708275afaaf1927c24db7c2eb1 Create systemd-resolved configuration on update

ipa-4-8:

  • 489ddc6d872b00fe5cddd1e9234fbb3e26f4aa0f Add helpers for resolve1 and nameservers
  • d6827f52b629d2a6afdd1b60ad190efae0d55a3e Configure NetworkManager to use systemd-resolved
  • 6dc5566c7b1edfd40722a5740e9bf9f33d74a609 Use new API for auto-forwarders
  • c67aba230fafd1ad9aded64fdac25081b4cd532d Configure systemd-resolved to use IPA's BIND
  • 3b3cb99dc15d826b825701fd04b00d74617e526e Create systemd-resolved configuration on update

Metadata Update from @abbra:
- Custom field changelog adjusted to FreeIPA DNS servers now detect systemd-resolved and configure it to pass through itself.

Upgrade does not yet work well, I filed a separate issue https://pagure.io/freeipa/issue/8518 for this problem

master:

  • 34e47778b47748e94792455699b35e8c126c5db7 Ensure that resolved.conf.d is accessible
  • ced1dcb1d9343a6fd0b021551eaf0b2c1d92a2c9 Also backup DNS config drop-ins

ipa-4-8:

  • e9824b51493bf64894c548f28099acffa0902ca9 Ensure that resolved.conf.d is accessible
  • 2f44efca70a91d15fb5d792283baf2524094388b Also backup DNS config drop-ins

Metadata Update from @cheimes:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

I found a potential issue with my code. If the server has multiple NICs with multiple IP addresses than the new code might add "127.0.0.1" and "::1" to /etc/resolv.conf multiple times.

The get_server_ip_address(host, True, False, []) should be equivalent to resolve_ip_addresses_nss(host).

$ git diff
diff --git a/ipaserver/install/bindinstance.py b/ipaserver/install/bindinstance.py
index aae2fe603..3b446ce76 100644
--- a/ipaserver/install/bindinstance.py
+++ b/ipaserver/install/bindinstance.py
@@ -1121,18 +1121,18 @@ class BindInstance(service.Service):
     def setup_resolv_conf(self):
         searchdomains = [self.domain]
-        nameservers = []
+        nameservers = set()
         resolve1_enabled = dnsforwarders.detect_resolve1_resolv_conf()
         for ip_address in self.ip_addresses:
             if ip_address.version == 4:
-                nameservers.append("127.0.0.1")
+                nameservers.add("127.0.0.1")
             elif ip_address.version == 6:
-                nameservers.append("::1")
+                nameservers.add("::1")
         try:
             tasks.configure_dns_resolver(
-                nameservers, searchdomains,
+                sorted(nameservers), searchdomains,
                 resolve1_enabled=resolve1_enabled, fstore=self.fstore
             )
         except IOError as e:
diff --git a/ipaserver/install/server/upgrade.py b/ipaserver/install/server/upgrade.py
index d1d8b3a64..ef58fd24c 100644
--- a/ipaserver/install/server/upgrade.py
+++ b/ipaserver/install/server/upgrade.py
@@ -1437,8 +1437,9 @@ def upgrade_bind(fstore):
     # resolve1's stub resolver config file.
     has_resolved_ipa_conf = os.path.isfile(paths.SYSTEMD_RESOLVED_IPA_CONF)
     if not has_resolved_ipa_conf and detect_resolve1_resolv_conf():
-        ip_addresses = installutils.get_server_ip_address(
-            api.env.host, True, False, [])
+        ip_addresses = installutils.resolve_ip_addresses_nss(
+            api.env.host
+        )
         bind.ip_addresses = ip_addresses
         bind.setup_resolv_conf()
         logger.info("Updated systemd-resolved configuration")

Metadata Update from @cheimes:
- Issue status updated to: Open (was: Closed)

master:

  • 6ba5a6a46e8f9e47cad82efa37a4ea79e5428dd6 Require(post) systemd with resolved enabled on F33
  • 814328ea3c9173c5e89c1b8370f2459fa67ae763 Don't add 127.0.0.1 to resolv.conf twice
  • a3abae825ce09ac7976969f61fb41ab448f1699f Simplify update code

ipa-4-8:

  • da301c25e69d741bfb672340f5ffeca49a1cabea Require(post) systemd with resolved enabled on F33
  • d1c43a9ce6038b29403f0b99aeacc312665b8510 Don't add 127.0.0.1 to resolv.conf twice
  • 01b5904e2dbee627d34e22ed766c85bf9758b3a5 Simplify update code

Metadata Update from @cheimes:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

so. I installed F36 as a KVM. My first guest is Freeipa. my second guest is CentOS 8 Stream .
I have tried every fix I can find to get freeipa-dns to work. it seems that systemd-resolved doesn't respect secondary nameservers i.e. it should by default try the primary then when it doesn't respond try the secondary ( at least that is how I learned) to make matters worse. it randomly changes which is the primary. it would seem to me that code changed discussed here would have either disabled this feature from systemd-resolved or wrote the correct switches \ parameters to the correct conf files to overcome this.
after two days of trying to get my CentOS 8 box to work. I deleted the FreeIPA instance and built another on CentOS 8 (just to make sure I understood how to configure it) and it works perfectly. I added three more servers 1) CentOS 9 1) Alma 1) Rocky all configured simply to use the FreeIPA servers DNS service and they work fine.
the only system I can't get to reliably resolve is my F36 Laptop. and for the same reason the FreeIPA server didn't work. systemd-resolved keeps flipping the primary DNS and doesn't care about the secondary...
the work around seems to be to set my global primary to the external DNS and the links primary to the internal
f36_DNS_Conf_2022-07-20_17-43-33.png

so. I installed F36 as a KVM. My first guest is Freeipa. my second guest is CentOS 8 Stream .
I have tried every fix I can find to get freeipa-dns to work. it seems that systemd-resolved doesn't respect secondary nameservers i.e. it should by default try the primary then when it doesn't respond try the secondary ( at least that is how I learned) to make matters worse. it randomly changes which is the primary. it would seem to me that code changed discussed here would have either disabled this feature from systemd-resolved or wrote the correct switches \ parameters to the correct conf files to overcome this.
after two days of trying to get my CentOS 8 box to work. I deleted the FreeIPA instance and built another on CentOS 8 (just to make sure I understood how to configure it) and it works perfectly. I added three more servers 1) CentOS 9 1) Alma 1) Rocky all configured simply to use the FreeIPA servers DNS service and they work fine.
the only system I can't get to reliably resolve is my F36 Laptop. and for the same reason the FreeIPA server didn't work. systemd-resolved keeps flipping the primary DNS and doesn't care about the secondary...
the work around seems to be to set my global primary to the external DNS and the links primary to the internal
f36_DNS_Conf_2022-07-20_17-43-33.png

I have a similar issue, if not the same issue actually. on F36 this doesn't work

Please do not use this issue to report problems with systemd-resolved operations because they are problems on systemd-resolved side. Please use systemd issue tracker if there are real issues.

FreeIPA servers will set up a configuration snippet that forces systemd-resolved to use IPA DNS server for IPA domains as /etc/systemd/resolved.conf/zzz-ipa.conf. This only happens on the IPA servers, not clients, because clients supposed to talk to their upstream DNS servers and IPA DNS servers should be serving a zone that is accessible to upstream DNS servers of the network.
You can add a similar snippet yourself on the IPA clients, if needed:

        # generated by IPA tests
        [Resolve]
        DNS={nameservers}
        Domains=~. {searchdomains}

where {nameservers} is the list of IPA DNS servers and {searchdomains} is the list of IPA DNS domains. See man page for resolved.conf for details.

As far as I can see, recent upstream CI tests which use these settings on the clients do succeed against Fedora 36 with systemd-resolved. This can be seen at http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/7f73a058-201e-11ed-8f94-fa163ea04eff/report.html (expand all tests to see details).

Metadata