This can be observed in nightly runs of test_smb: http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/42b1a2da-e36d-11eb-b29e-fa163e472ef6

There are 4 test cases failing in test report:
* test_samba_reinstall
* test_ntlm_authentication_with_auto_domain
* test_ntlm_authentication_with_upn_with_lowercase_domain
* test_ntlm_authentication_with_upn_with_uppercase_domain

What happens here is that test_ntlm* test cases are executed after test_samba_reinstall. The latter is failing due to a known problem and it leaves system without a working Samba which causes test_ntlm* to fail as well.

ipatests are using pytest_sourceorder plugin to execute tests cases in the same order as they are defined in the code. The test_smb was designed to avoid aforementioned failure by placing test_ntlm* test cases before samba uninstaltion: https://github.com/freeipa/freeipa/blob/b43ac4fdd20a0c1b122d6a58cea4a5f91e498fe6/ipatests/test_integration/test_smb.py#L404

But the plugin does not properly handle decorated methods and those test cases are wrapped using a decorator.

I have raised a PR for improving the plugin: https://pagure.io/python-pytest-sourceorder/pull-request/2

When it is merged we need to update the package in PR-CI