Trying this out here: https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=b916554254b6efc92daa168f78034ec75a186dba

LGTM, :thumbsup:

Assuming I'm reading this correctly we'll likely need to tweak this some more.

I'll try to post a longer comment on how the CI pipeline works and what GW should consider as pass

How does the CI pipeline work?

Basically the CI pipeline runs in multiple steps:

• Check if the commit concerns a package of interest (one of the package in
  Fedora Atomic Host)
• Build the rpm corresponding to this commit
• Run tests on this RPM
• Compose an OSTree with this new RPM
• Create an image from this compose
• Run tests on this image
• Run tests on this compose

So if the package is not in the list the pipeline sends an package.ignore fedmsg.
Otherwise, we need to look for .compose.test.integration.complete results.

I need to double check that failing one step will stop the entire pipeline
(ie: that we do not have a .compose.test.integration.complete results if the
tests are failing on the RPM built).

Hm, thinking more about this, we need to consider more messages since a missing compose.test.integration.complete results message could mean that either the step hasn't been run yet or that the pipeline failed in an earlier step.

So we need to consider all the messages for every step to find if one failed.

So after discussing on #fedora-ci it turns out the CI pipeline is growing a new fedmsg message/topic: ci.pipeline.complete (cf fedmsg_meta PR: https://github.com/fedora-infra/fedmsg_meta_fedora_infrastructure/pull/441) which will in one message reflect the entire pipeline run.
If the pipeline fails at any step, the message's status will be FAILED, if any step is aborted, that message will be sent with a status ABORTED and if the pipeline runs until the end without any problem, it will have a status SUCCESS.

Ari has agreed that we can rely on this message for our purpose, this should make our logic much easier, there will either be an ci.pipeline.package.ignore or a ci.pipeline.complete message with the status of the entire build in it.

FYI we had a similar issue with "ignoring" some builds in one of our internal testing tools, RPMDiff. It will run its tests if it finds a valid baseline build to compare, else it skips the comparison tests.

After some discussion with the Fedora QA folks on https://phab.qa.fedoraproject.org/D1218 we came to the conclusion that in this case, the testing tool should report PASSED with a note explaining that it decided there was nothing to test.

So maybe we could be doing something similar with the CI results? If the Jenkins job decides it doesn't want to test the package it could report PASSED with a note saying why it didn't do any tests.

On the other hand... this means that the logic for deciding which packages will be tested will live in the Jenkins job. But maybe Greenwave is really the place we should be expressing that instead? For example we have on our list to do per-package policies so that we can, for example, enforce more checks on packages in the critpath list. Maybe this is a similar situation?

If the Jenkins job decides it doesn't want to test the package it could report PASSED

That's what it does but it uses the ignore message for that.

this means that the logic for deciding which packages will be tested will live in the Jenkins job. But maybe Greenwave is really the place we should be expressing that instead?

I think GW is the place where we decide: do we have enough information to go further not the place where we should start tests (ie: not the place that the CI pipeline should/would query to decide if a package should be tested or not).

Tbh, I really think GW shouldn't have to worry about this, it has a set of rules and applies them. If something doesn't it's not GW's problem :)
As they say in the hitch hiker's guide to the galaxy: SEP* :)

• Somebody else's problem

Okay that makes sense.

I think what is bugging me here is the test case names, which seem to be:

org.centos.prod.ci.pipeline.package.ignored
org.centos.prod.ci.pipeline.package.complete

but those aren't really test cases. They are events.

To take it back to the RPMDiff example, the test case name is dist.rpmdiff.comparison and if the tests pass you get a result in ResultsDB with a PASSED outcome. And if it decides to skip you (will) get a result under the same name but with a note saying that it didn't actually run tests. But since it's the same test case name in either case, it means Greenwave just has to look for that test case with PASSED. No need for this "one or the other" logic.

So in this case, would it make more sense if the test case name were actually this?

org.centos.prod.ci.pipeline.package

which I guess means, "package-level tests in the CentOS CI pipeline".

The CI pipeline does not have the notion of test cases, it runs all the tests it is planned to and report pass/fail.

I think we need to be careful not to approach it too much with the view of taskotron as they are different pipeline and working quite differently.

OK - I think this PR is out-moded by other work. Closing without merging.