Thanks for the report. I'm going to increase the timeout for the job rpm-test for this repository (https://pagure.io/fedora-zuul-jobs-config/pull-request/132)

https://src.fedoraproject.org/rpms/selinux-policy/pull-request/220

The job is running on a 2vcpu/8GB-Ram.

What is the test node spec @mvadkert you are using in Fedora CI ?

I can probably enroll a beefy node in our setup and set the job to run there.

CC: @zpytela

by default, we run a 2vCPU/4GiB RAM, 50GB drive, so I guess we are even lower on specs :(

The latest builds fail again in rpm-test with FAILURE or RETRY_LIMIT result, I can't find any logs to check though, neither am I able to assess if these problems are related.

https://src.fedoraproject.org/rpms/selinux-policy/pull-request/223
https://fedora.softwarefactory-project.io/zuul/buildset/cda31ac4ebb146fa9ba81d58f7bb4d9a
https://src.fedoraproject.org/rpms/selinux-policy/pull-request/222
https://fedora.softwarefactory-project.io/zuul/buildset/b7be5fd984624db2823cb2a36c4568b6
https://src.fedoraproject.org/rpms/selinux-policy/pull-request/221
https://fedora.softwarefactory-project.io/zuul/buildset/6e2e527ec78e4fa8b82c20001ca9443a

Hi @zpytela,

I see that the test playbook restart the test node. Unfortunately, the Zuul CI cannot support this.
The error we see is directly related to that.
See line 474: https://fedora.softwarefactory-project.io/zuul/build/3865b7d14e5844deb2fefa5a429a8e95/log/job-output.txt#474

Is the restart host mandatory for the test ?

Hmm, good to know. Is this something planned to be supported? If no we would need to find some alternatives (like running STI tests via Testing Farm), once we would migrate completely off Fedora CI for PR testing.

I'm not aware of any plan to support this. I guess the solution is, onto the test node, to spawn a VM and run the ansible-playbook again that nested VM.

So it seems we could:
- Check is the restart is mandatory here for the test
- Evaluate how to provide a role that spawn a VM inside the test node
- Update the project pipeline for that package to run the STI test on Testing Farm
- Or we can remove the run of rpm-test for this packages and assume we rely only on Fedora CI.

• Check is the restart is mandatory here for the test

It might not be mandatory for this particular test, but there's a test in systemd which requires reboot in order to confirm that the new systemd is able to boot with the latest selinux policy

@fbo so after more thinking, seems the easiest would be just to delegate STI tests to Testing Farm, as we do with TMT. Required role changes should be very easy. We want to get rid of Fedora CI for PRs in the long term.

Seems other reasonable option:

• Evaluate how to provide a role that spawn a VM inside the test node

Would be a lot more work.

WDYT @fbo?

@fbo, the latest selinux-policy builds fail in rpm tests after 5 minutes with a traceback:

Traceback (most recent call last):
  File "<stdin>", line 102, in <module>
  File "<stdin>", line 94, in _ansiballz_main
  File "<stdin>", line 37, in invoke_module
  File "/tmp/ansible_command_payload_6l_n4ir_/ansible_command_payload.zip/ansible/module_utils/basic.py", line 82, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 945, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1430, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1402, in _get_spec
  File "<frozen zipimport>", line 170, in find_spec
  File "<frozen importlib._bootstrap>", line 431, in spec_from_loader
  File "<frozen importlib._bootstrap_external>", line 741, in spec_from_file_location
  File "<frozen zipimport>", line 229, in get_filename
  File "<frozen zipimport>", line 752, in _get_module_code
  File "<frozen zipimport>", line 586, in _get_data
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ansible_command_payload_6l_n4ir_/ansible_command_payload.zip'

https://fedora.softwarefactory-project.io/zuul/build/4c739acc99914f33816ac99724d84051
https://fedora.softwarefactory-project.io/zuul/build/1ea726d2e1214ee486512de1d53ad16a
can you take a look?

I think the tests which restart the system are no longer in place.

@zpytela I still see two problematic tasks https://fedora.softwarefactory-project.io/zuul/build/1ea726d2e1214ee486512de1d53ad16a/log/job-output.txt#470

• Restart host
• Wait for host to come back

@mvadkert, is FMF a replacement for STI ? Could these tests that require a test node restart could be rewritten using FMF and then being handled by testing farm ?

@fbo yes, I believe for exactly this test migrating to TMT could be an option, and would resolve the problem with reboot.

@psss FTR ... seems with tmt we got to the same conclusion.

@zpytela so ... we propose to close this issue. If the test needs reboot it should be migrated to tmt

Was this changed announced? Was approved? Is it documented? I mean it's something that used to work and we are supposed to migrate to other technology even though STI is claimed as supported - https://docs.fedoraproject.org/en-US/ci/

@plautrba you are right, I am sorry, the other feasible option is to move STI to testing farm, so I guess we should rather to do that seems. I agree it is a regression ... we should do better

Will be tracked under https://issues.redhat.com/browse/TFT-788, @guyinger will work on it

@mvadkert I see the task referred to was closed, anything to be done here in this ticket?

@zpytela thanks for following up, indeed this is resolved for a long time, the STI tests run for at least a year via Testing Farm.

Closing