CC @happz @psss

Thanks Miro. I've added a short introduction to the main README and extended the ci metadata README with more detailed comments and examples. I also propose to use L1 metadata and L2 metadata terms to prevent possible confusions (test metadata sounds often too generic and could be used for both). Using L1 & L2 could help us to make our terminology more distinct. What do you say?

L1/L2 sounds good to me.

I'll check the README in the evening. Today we had a chat with @lzachar, discussing L2 metadata for the "test the test" project, we also touched interaction with L1 and get an agreement of how a prototype could look like. It opened a few questions and shuffled a few issues I had with steps' responsibilities. TL;DR version, I might be mixing specification and implementation a bit too much, bluring the borders. I suppose careful wording will be necessary to prevent others failing for the same trap, maybe more based on "promises" made by each stage, not sure yet. Nothing groundbreaking or changing a direction, but worth a chat.

I would even mention, that these steps can be also virtual only, just a placeholder and can serve just to discover the metadata ...

Maybe I would be more verbose with (distro from build)

use Flexible Metadata Format filters to get the list of test cases?

and also the list of test cases?

I'd maybe explicitly mention that FMF is "just" a YAML with special treatment of keys/values if written in a specific way (like /build:). I.e. "don't be afraid, it's still YAML but with FMF, you get these features for free".

I'd maybe use one or two properties to illustrate how each of default, detect and define affects them. E.g. compose or RAM size: by default, CI system is configured to use compose C1 and R1 GB. In detect, CI system can apply its wisdom to deduce that for this particular build target, compose C2 is a much better choice, and RAM is fine as it is. And in define, user explicitly says C3 and R2 are the best.

I'm not sure whether only from the test case metadata - wouldn't DDD (default/detect/define) would play a role as well? I mean, defined compose (or, generaly, environment, limit arches to just x86_64) would affect the discovery as well (via relevancy).

I'd rather avoid using this form of "provision" - suggests the provisioning happens in this stage, which doesn't have to be true, and it would be an implementation detail anyway. I'd prefer something along "provisioning-related knobs" or "describes what and how to provision things". More in a way what user can affect in this section than describing what CI system does.

This I'd drop: there's no "Output" in other stages, comes closer to the implementation side. Instead, I'd add a paragraph before the stages to describe what's the goal, that we want to make a clear separation of stages and areas, giving user control over their aspects, making clear what and how they can influence in each virtual stage of the process, and that we want to make clear to CI system developers that test case and CI metadata are important :) We don't have to specify the actual output of each stage (actually, given how our CI works, most of the "outputs" would come from discovery stage anyway).

I'd also mention that we (or I do, at least) want to provide a generic and extensible way to write down HW requirements - it'd be awful if user had to specify "at least 2GB of RAM" five different ways, once for each provisioner their tests may run on.

Isn't it before/after artifact installation?

I'd drop the "output" reference - "execute the tests on provisioned boxes" could be clear enough. Or something like that, "output" again suggests the stages had to exchange some data.

While chatting with @lzachar - we defined "prepare" stage, but its "teardown" counterpart is missing, and there seems to be an interest in this step, to e.g. unregister machines from some CND or upload artifacts to users' community storage somewhere in Amazon's cloud. You know, "teardown" stuff, as compared to our "setup"-like "prepare". I was thinking about adding this kind of stage, but maybe we could hijack "prepare" and make take care of both setup and teardown?

+1 for examples

podman ... create a container using podman
qemu-kvm ... run vm using qemu-kvm

yep, +1 for adding another stage, teardown or maybe destroy :)

Thanks much for the review, guys! All comments should now be addressed. Here's summary of the changes:

• Added a short "FMF is just an improved YAML" sentence
• Steps extended with discover-only and execute-only example
• Included DDD example with memory and compose
• Extended steps section description
• Added a new step finish for cleanup actions
• Plus several other minor changes and typo fixes

Regarding the finish step naming: I think it would be good to have it as a verb to be consistent with the rest. Destroy sounds also tempting but I guess there could be some positive actions performed as well ;-)

This needs an update, following our discussion on IRC:

report:
  email:
    - email@address.org

Last comment from my side, to include the recent change of report stage, otherwise LGTM. Nice work!

+1 to happz comment, and otherwise also LGTM

Thanks for the review. Commits squashed and pushed to master as b2debc4.