So far 4 failures happen randomly:
- test_command[0006: dnszone_add: Try to create a zone with nonexistent NS entry]

AssertionError: Expected 'dnszone_add' to raise NotFound, but caught different.   args = ['zone2.test']   options = {'idnssoamname': 'ns1.zone2.test.', 'idnssoarname': 'root.zone2.test.', 'version': '2.234'}   expected = NotFound: Nameserver 'ns1.zone2.test.' does not have a corresponding A/AAAA record   got = DNSResolverError: All nameservers failed to answer the query ns1.zone2.test. IN A: Server 127.0.0.1 UDP port 53 answered SERVFAIL

This response happens when the forwarder is not answering. The test is setup with --auto-forwarders and configures the DNS server found in /etc/resolv.conf as a forwarder.
The call to ipa dnszone-add tries to resolve ns1.zone2.test. but this domain is not managed by IPA DNS server, hence the DNS forwarder comes into play and doesn't respond properly.

Same type of issue for the other failing tests:
- test_command[0109: dnszone_add: Try to create zone 'zone3.test' with relative nameserver]

AssertionError: Expected 'dnszone_add' to raise NotFound, but caught different.   args = ['zone3.test']   options = {'idnssoamname': 'ns', 'idnssoarname': 'root.zone3.test.', 'version': '2.234'}   expected = NotFound: Nameserver 'ns.zone3.test.' does not have a corresponding A/AAAA record   got = DNSResolverError: All nameservers failed to answer the query ns.zone3.test. IN A: Server 127.0.0.1 UDP port 53 answered SERVFAIL

• test_command[0110: dnszone_add: Try to create zone 'zone3.test' with nameserver in the zone itself]

AssertionError: Expected 'dnszone_add' to raise NotFound, but caught different.   args = ['zone3.test']   options = {'idnssoamname': 'zone3.test.', 'idnssoarname': 'root.zone3.test.', 'version': '2.234'}   expected = NotFound: Nameserver 'zone3.test.' does not have a corresponding A/AAAA record   got = DNSResolverError: All nameservers failed to answer the query zone3.test. IN A: Server 127.0.0.1 UDP port 53 answered SERVFAIL

The failure happens in Azure env but not in Travis. As we have no control over the DNS server that Azure configures it will be difficult to fix the issue.

@frenaud, do you think that self-hosting Azure agents could fix that?
https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops

I have no idea if name resolution would behave differently inside another infrastructure.

Currently we have one hosted agent configured (I guess it was set-up by @abbra), ~~I wonder if when the test passes it is being executed by that agent, which might explain the inconsistency of results.~~ (Edit: there are successful runs on other agents)

The issue may have been fixed by commit feae9de which is setting the resolver as 8.8.8.8 instead of 168.63.129.16 in the azure tests (as a result, the forwarder is 8.8.8.8). Let's wait for a few days and if the issue does not appear any more I will close this ticket.

Closing the issue as it didn't happen any more since last comment.