Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/9e1dbc6c53654ec5898ba1ad0de53f84

• validate : FAILURE in 2m 48s
• compose-base : SUCCESS in 12m 08s

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/2be65c1fb4344cd192597186f2220ac6

• validate : FAILURE in 2m 52s
• compose-base : SUCCESS in 8m 57s

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/f98d0dff146e456c98c254f853f7a7f9

• validate : FAILURE in 3m 12s
• compose-base : SUCCESS in 10m 22s

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/4fa0a1e1ec264a07b772407b2d401d5a

• validate : SUCCESS in 11m 55s

OK, this does not significantly increase the time to run the tests and combines them into a single job, avoiding an additional VM spawn so should be worth it.

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/faf0f20a2c544b178600853a95487871

• validate : SUCCESS in 11m 10s

Would just compose-legacy base ever pass if just compose-dry-run failed? If so, you might want to check whether it still runs in that scenario now, and if not, see if you can change things so it does.

Yes, it's possible that compose-legacy base could pass when compose-dry-run would fail as base is a smaller set.

In general if either fail, we want to fail CI here.

If so, you might want to check whether it still runs in that scenario now, and if not, see if you can change things so it does.

Not sure I understand what you mean?

I'm thinking about the case where just compose-dry-run fails. Does the system just stop running the tests at that point? If so, you don't know whether just compose-legacy base will pass or fail, which is still useful information.

I see this pattern quite often in CI setups with multiple tests and linters. If the run stops on the first failure, you only know about the first failure; so you fix it, run again, find that something else fails, fix it, run again...it's more efficient if you can configure it to always run the steps that are independent, so you know everything that's broken on a single run.

Yes, I though about that as well. Unfortunately, doing two separated steps means spawning two VMs and fetching the repo metadata twice. Not horrible but not really efficient, given that most of the time neither will fail.

I'm not against reverting back to two jobs. I'm not sure we'll get much out of it.

I didn't necessarily mean that, just that sometimes there's a setting you can use or something to say "do this step of the flow even if the previous step failed". In Github Actions for e.g. you can do if: success() || failure() to run a step of the workflow even if a previous one failed. WIth tox you can set the confusingly-named testenv setting ignore_errors = true, which doesn't actually ignore errors - tox still exits non-zero if there are any - but does not die on the first error it encounters. So I just wondered if there might be something like that here. It's a minor point.

Ah, I understand now. Zuul uses Ansible playbooks so maybe https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_error_handling.html ?

Looks like ignore_errors: true could do it. I'll give it a try.

Made: https://pagure.io/workstation-ostree-config/pull-request/495