Hi. I do not have strong opinion on this.

Just a few points to consider.

• Invocation order. Invocation order in CI-pipelines and by user must be the same. In this case we must retract ability to call ansible-playbook directly. And provide some tool, that controls order.
• Reset TEST_SUBJECT for each test_xxx.yaml. There is a way how to reset qcow2. But, how do we reset test-subject when user runs test locally?

Invocation order. Invocation order in CI-pipelines and by user must be the same. In this case we must retract ability to call ansible-playbook directly. And provide some tool, that controls order.

Isn't the test system supposed to provide a clean environment for each of the playbooks? The spec isn't particularly clear on this, but it would make a lot of sense, I think. In that case you don't care about the order, because the playbooks can't affect each other (and you can run them in parallel as a bonus).

@kparal please check reference STI : https://fedoraproject.org/wiki/CI/Standard_Test_Interface

1. To avoid any misunderstanding, please use exact naming. Not test system, but rather testing system.

2. Isn't the test system supposed to provide a clean environment for each of the playbooks?
   Again confusion. What is clean environment ?

Ok, now some thoughts:

Testing System does stage Test environment

@kparal your proposal sounds reasonable.
We need define strict run-order.
Define strict workflow between:

1. 1 piece Test Runner
2. 1 piece Testing System
3. And numerous running Test environment (this is from @kparal proposal)
4. Take into consideration that some tests require multi test-environment at the same time.

So using the STI terminology I would say Test Runner receives the list of ansible playbooks to be executed from the Testing System and prepares a clean Test Environment for each playbook. Finally, after the testing is finished aggregates results from individual playbooks. Thus there would be no collisions as @kparal suggested above. Does that make sense? Would something like this work with the pipeline, @bgoncalv, @mvadkert?

yes, I think this is correct.

Yep, we are doing +- something like this in Bci, I guess it is also doable in the pipeline.

To summarize what we've agreed on:

• We should support multiple playbooks, that is tests/tests*.yml
• STI should be updated to explicitly require clean environment for each such playbook

Which also leads to:

• Fedora docs should be updated accordingly (e.g. the tests page)
• Stats generating scripts should not rely on tests/tests.yml (as there can be none)

Here's a recent real-life example of a tests/tests_python.yml playbook.

STI should be updated to explicitly require clean environment for each such playbook

Specification updated.

Fedora docs should be updated accordingly (e.g. the tests page)

Docs changes proposed.

Stats generating scripts should not rely on tests/tests.yml (as there can be none)

Issue #48 created to track this.