#8666 Nightly test failure (fed33) in test_simple_replication.py::TestSimpleReplication::test_replica_manage
Closed: fixed by frenaud. Opened by frenaud.

The nightly test test_simple_replication.py::TestSimpleReplication::test_replica_manage is failing in fedora 33. See PR #647 with the following logs and report:

self = <ipatests.test_integration.test_simple_replication.TestSimpleReplication object at 0x7f80a27b0520>
    def test_replica_manage(self):
        """Test ipa-replica-manage list
        Ensure that ipa-replica-manage list -v <node> does not print
        last init status: None
        last init ended: 1970-01-01 00:00:00+00:00
        when the node never had any total update.
        Test for ticket 7716.
        """
        msg1 = "last init ended: 1970-01-01 00:00:00+00:00"
        msg2 = "last init status: None"
>       result = self.master.run_command(
            ["ipa-replica-manage", "list", "-v", self.replicas[0].hostname])
test_integration/test_simple_replication.py:87: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <ipatests.pytest_ipa.integration.host.Host master.ipa.test (master)>
argv = ['ipa-replica-manage', 'list', '-v', 'replica0.ipa.test'], set_env = True
stdin_text = None, log_stdout = True, raiseonerr = True, cwd = None, bg = False
encoding = 'utf-8', ok_returncode = 0
    def run_command(self, argv, set_env=True, stdin_text=None,
                    log_stdout=True, raiseonerr=True,
                    cwd=None, bg=False, encoding='utf-8', ok_returncode=0):
        """Wrapper around run_command to log stderr on raiseonerr=True
        :param ok_returncode: return code considered to be correct,
                              you can pass an integer or sequence of integers
        """
        result = super().run_command(
            argv, set_env=set_env, stdin_text=stdin_text,
            log_stdout=log_stdout, raiseonerr=False, cwd=cwd, bg=bg,
            encoding=encoding
        )
        # in FIPS mode SSH may print noise to stderr, remove the string
        # "FIPS mode initialized" + optional newline.
        result.stderr_bytes = FIPS_NOISE_RE.sub(b'', result.stderr_bytes)
        try:
            result_ok = result.returncode in ok_returncode
        except TypeError:
            result_ok = result.returncode == ok_returncode
        if not result_ok and raiseonerr:
            result.log.error('stderr: %s', result.stderr_text)
>           raise subprocess.CalledProcessError(
                result.returncode, argv,
                result.stdout_text, result.stderr_text
            )
E           subprocess.CalledProcessError: Command '['ipa-replica-manage', 'list', '-v', 'replica0.ipa.test']' returned non-zero exit status 1.
pytest_ipa/integration/host.py:200: CalledProcessError
 ------------------------------Captured stderr call------------------------------ 
ipa: ERROR: stderr: Unknown host replica0.ipa.test: Host 'replica0.ipa.test' does not have corresponding DNS A/AAAA record

Another failure: PR648, report

The problem is blocking https://pagure.io/freeipa/issue/8544

Metadata Update from @cheimes:
- Custom field blocking adjusted to 8544
- Issue marked as blocking: #8544

I added a DNS lookup before the failing test. Turns out master cannot resolve the replica hostname:

:transport.py:513 RUN ['dig', 'replica0.ipa.test']
:transport.py:557 
:transport.py:557 ; <<>> DiG 9.11.27-RedHat-9.11.27-1.fc33 <<>> replica0.ipa.test
:transport.py:557 ;; global options: +cmd
:transport.py:557 ;; Got answer:
:transport.py:557 ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 27926
:transport.py:557 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
:transport.py:557 
:transport.py:557 ;; OPT PSEUDOSECTION:
:transport.py:557 ; EDNS: version: 0, flags:; udp: 1232
:transport.py:557 ; COOKIE: fcc42f05843bef4827c08aad601801dfbd1377485560a286 (good)
:transport.py:557 ;; QUESTION SECTION:
:transport.py:557 ;replica0.ipa.test.       IN  A
:transport.py:557 
:transport.py:557 ;; AUTHORITY SECTION:
:transport.py:557 ipa.test.     1   IN  SOA master.ipa.test. hostmaster.ipa.test. 1612186046 3600 900 1209600 3600
:transport.py:557 
:transport.py:557 ;; Query time: 0 msec
:transport.py:557 ;; SERVER: 127.0.0.1#53(127.0.0.1)
:transport.py:557 ;; WHEN: Mon Feb 01 13:27:59 UTC 2021
:transport.py:557 ;; MSG SIZE  rcvd: 128
:transport.py:557 
:transport.py:217 Exit code: 0
:transport.py:391 RUN ['ipa-replica-manage', 'list', '-v', 'replica0.ipa.test']
:transport.py:513 RUN ['ipa-replica-manage', 'list', '-v', 'replica0.ipa.test']
:transport.py:557 Unknown host replica0.ipa.test: Host 'replica0.ipa.test' does not have corresponding DNS A/AAAA record
:transport.py:217 Exit code: 1

The root cause has been identified. PRCI is creating /etc/hosts file on the master/replica/client machines with a line for each test machine.
In fedora33, systemd-resolved is used and "dig@127.0.0.53 client.ipa.test" on the client.ipa.test machine is able to answer (because systemd-resolved is configured by default with ReadEtcHosts=true, it also gets answers from the local /etc/hosts content).
As a consequence, ipa-client-install is tricked into believing that an A/AAAA record is already defined for the client machine and does not create the record (see https://github.com/freeipa/freeipa/blob/3a584803da697b80d49590223be632a7451e1b7e/ipaclient/install/client.py#L1564).

On fedora32, the client installer correctly sees that no DNS record exists for the client and adds the record in IPA DNS. That explains the difference between fedora32 and fodora33.

Another failure: PR680, report

Fixed on the latest nightlies:
- testing_master_previous: PR 780, Report
- testing_master_latest: PR 779, Report
- testing_ipa-4-9_latest: PR 771, Report
- testing_ipa-4-9_previous: PR 773, Report
- testing_ipa-4.6: PR 776, Report

thanks to PR https://github.com/freeipa/freeipa/pull/5474 DNS resolvers management

Metadata Update from @frenaud:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

Metadata