So, this is partly due to an error in Bodhi: it is not including a scenario when issuing waivers. When we look for waivers on the non-verbose query path, we include the answer's scenario in the filter:

    if not self.verbose:
        for answer in self.answers:
            if not answer.is_satisfied:
                self.waiver_filters.append(dict(
                    subject_type=answer.subject.type,
                    subject_identifier=answer.subject.identifier,
                    product_version=self.product_version,
                    testcase=answer.test_case_name,
                    scenario=answer.scenario
                ))

so we will not find a waiver with a scenario of 'null'. Things would work OK if Bodhi included the scenario when waiving.

Still, Greenwave's behaviour seems inconsistent here. It clearly considers that a result is waived if a waiver with scenario null otherwise applies to it. But it won't find such a waiver in a non-verbose query.

Either it should not consider a waiver with scenario null to match a test result with a scenario, or it should find null scenario waivers when doing non-verbose queries (by broadening the filter somehow I guess).

I think something like this would fix it:

[adamw@xps13k greenwave (master *%)]$ git diff
diff --git a/greenwave/decision.py b/greenwave/decision.py
index 759aa6a..c1756d5 100644
--- a/greenwave/decision.py
+++ b/greenwave/decision.py
@@ -104,6 +104,13 @@ class Decision:
                         testcase=answer.test_case_name,
                         scenario=answer.scenario
                     ))
+                    self.waiver_filters.append(dict(
+                        subject_type=answer.subject.type,
+                        subject_identifier=answer.subject.identifier,
+                        product_version=self.product_version,
+                        testcase=answer.test_case_name,
+                        scenario=None
+                    ))
         if self.waiver_filters:
             self.waivers = waivers_retriever.retrieve(self.waiver_filters)

but it's a bit hard to be sure, it's hard to test this code in anger.

Both queries give "All required tests passed" now. Maybe it is an issue with caching.

But you are probably correct that Greenwave should always also fetch waivers with scenario=null.

They're actually both now returning "All required tests passed" because we've cleaned things up so that almost all update tests pass for F37 updates now, and I re-ran the previously-failed tests on all updates. So the waivers are no longer important.

Sorry, I should've thought to not re-run the tests on this update so we could keep using it as a sample :|

Forgot to add, I did test my proposal in a dev instance of greenwave, and it did work. It's not very elegant, but I can't think of an elegant way to do it since waiverdb doesn't seem to provide any way to provide a list of conditions for a given filter instead of just one.

You could probably still test this out in a dev instance quite easily by just hacking the code (on both paths) to print a list or count of waivers it found, I guess.

:thumbsup: Thanks for the clear report and providing a solution!

This should fix the waiver inconsistency: https://github.com/release-engineering/greenwave/pull/73

If a matching waiver has scenario=null, it should apply to all test results disregarding their scenario values.