@pingou not sure if you're the right person to ask for guidance here (?)

@lholecek any idea maybe?

Maybe something has changed, I do not see "no tests are required" anymore for the bodhi update.

@msrb indeed, I see the same thing as @lholecek now ....

weird ....

It's hard to tell as the issue is 2 months old now. I think we can close this now and reopen if we encounter the same problem in future.

The issue is still present. Another glibc update in rawhide went straight to stable today:

• https://bodhi.fedoraproject.org/updates/FEDORA-2021-cf7d6a2c19

Would you please reopen this issue? Thanks.

@msrb ptal

There is no matching policy for "fedora-36" (see fedora.yaml):

❯ http -b https://greenwave.fedoraproject.org/api/v1.0/decision <<< '{"product_version":"fedora-36","decision_context":"bodhi_update_push_stable_critpath","subject":[{"item":"glibc-2.34.9000-5.fc36","type":"koji_build"},{"item":"FEDORA-2021-cf7d6a2c19","type":"bodhi_update"}]}'
{
    "applicable_policies": [
        "kojibuild_bodhipush_no_requirements",
        "bodhiupdate_bodhipush_no_requirements"
    ],
    "policies_satisfied": true,
    "satisfied_requirements": [],
    "summary": "no tests are required",
    "unsatisfied_requirements": []
}

There is no matching policy for "fedora-36" (see fedora.yaml)

Oops, sorry, read it incorrectly. There are matching policies, but maybe other policies are expected to match the new product_version.

FTR the gating.yaml for glibc: https://src.fedoraproject.org/rpms/glibc/blob/rawhide/f/gating.yaml

So seems to me quite some time ago there were some changes to the decision contexts ... which seems users (and us) are not aware of :( Looks to me glibc is in _critpath decision context, but the gating.yaml is not reflecting that, also the docs are unware of this:

https://docs.fedoraproject.org/en-US/ci/gating/

:(

CC @kevin @adamwill

@lholecek do we agree the gating.yaml now should look like:

# recipients: glibc-bugzilla
--- !Policy
product_versions:
  - fedora-*
decision_contexts:
  - bodhi_update_push_stable
  - bodhi_update_push_stable_critpath
subject_type: koji_build
rules:
  - !PassingTestCaseRule {test_case_name: baseos-qe.koji-build.scratch-build.validation}
--- !Policy
product_versions:
  - rhel-9
decision_context: osci_compose_gate
rules:
  - !PassingTestCaseRule {test_case_name: baseos-ci.brew-build.tier1.functional}

For all components wanting to enable it?

do we agree the gating.yaml now should look like

Looks good except decision_context should be decision_contexts if it contains list.

❯ http -b https://greenwave.fedoraproject.org/api/v1.0/validate-gating-yaml < gating.yaml | jq -r '.message'
Policy 'untitled': Attribute 'decision_context': expected a scalar node, but found sequence
  in "<unicode string>", line 5, column 3:
      - bodhi_update_push_stable
      ^

BTW, it still won't work with fedora-36 because it requires only *_no_requirements policies and those don't even contain RemoteRule.

@lholecek sorry i am still lost what needs fixing in the global gating.yaml then?

https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/greenwave/templates/fedora.yaml#_69

This is then what?

Trying to fix docs here:
https://pagure.io/fedora-ci/docs/pull-request/71

Pls review

Oh, I somehow missed the fedora-36 is in the other policies with RemoveRul too. 🤦‍♂️

I'm having hard time figuring out what is wrong. Maybe it's a bug in Greenwave. Let me check in my dev environment and get back to you.

Ah, this is confusing: kojibuild_bodhipush_remoterule is not mentioned in applicable_policies in the decision response because the only rule is RemoteRule but the gating.yaml does not contain policy with bodhi_update_push_stable_critpath decision context.

So, yes, adding bodhi_update_push_stable_critpath to the gating.yaml would fix the problem.

Does this mean we need to change the gating.yaml file for the glibc package? If yes, would someone who understands the required changes please send a pull request?

I think I may have run into a similar problem with: https://bodhi.fedoraproject.org/updates/FEDORA-2021-fc45233544

However, not all packages in the update are in the critical path, just annobin.

I just did an update with only clang: https://bodhi.fedoraproject.org/updates/FEDORA-2021-7a516aa7be and gating works.

So does this mean that having 1 critical path package in an update requires all other packages to have bodhi_update_push_stable_critpath ?

I confirmed clang is not in the critical path with this command:

user@host$ curl -X GET 'https://pdc.fedoraproject.org/rest_api/v1/component-branches/?name=rawhide&active=true&critical_path=true&type=rpm&global_component=clang' 
{"count":0,"next":null,"previous":null,"results":[]}

I've attempted to fix this for glibc itself here:
https://src.fedoraproject.org/rpms/glibc/pull-request/39

For updates with more packages this, alas, remains broken.

Another update that I have which lists no required tests, even though several packages having gating.yaml and the llvm package even has bodhi_update_push_stable_critpath listed as one of the decision contexts: https://bodhi.fedoraproject.org/updates/FEDORA-2021-622def0653

Would it be possible to prioritize this issue? It's hard to benefit from all the new CI features if the gating mechanism doesn't work.

I think the new _critpath context obfuscates gating for mere mortals even more than it was before :/

The idea is good, but I think, implementation-wise, it shouldn't leak to individual gating.yaml files in dist-git.

Could the _critpath be a different "gate" and Bodhi would then check if normal packages pass standard gates (_testing/_stable), and if the package is also on a critical path, it would check if it also passes the second _critpath gate? Just checking what's needed for critical path packages this time.

It really seems like multi-build updates just pass gating even if there are red test results that are marked as required in Bodhi (so Bodhi knows).

The last @tstellar 's update is a good example: https://bodhi.fedoraproject.org/updates/FEDORA-2021-304d0f8429

@adamwill ^ WDYT?

yes, sorry for this problem, it's my fault. We have actually already been kicking it around for a bit:
https://github.com/fedora-infra/bodhi/issues/4259
but the design of greenwave and bodhi makes it not exactly trivial to solve, unfortunately. To deal with the problem for now, you have to add the _critpath contexts to the decision_contexts in the
package policies. It would be safe and correct to do this in every single package policy, because any package could happen to be pulled into a critpath update, and Bodhi is querying at the update level here, not the package level.

I haven't had time to work on the "have Bodhi query both contexts" approach (which seems to be the least worst one we have ATM) yet, because of F35 release crunch. If anyone else has the time to do it, it'd be appreciated. Most of the change required is actually quite trivial, but there's one specific awkward point I mentioned in the description of the other issue.

since this is causing real trouble, I'm kicking it up the priority list and trying to write a Bodhi fix today. It'll still need to be reviewed, merged, a new Bodhi release made, and that new release deployed, though.

I filed https://github.com/fedora-infra/bodhi/pull/4266 , reviews appreciated.

@adamwill Thank you for the quick response! :wink:

cool, seems this will be fixed soon \o/ thanks for prioritizing this @adamwill

Note in the last few days the 5.7.1 release has been made and deployed to stg. Hopefully it'll get to production soon. It might even be possible to test the fix, I'll see if I can try it when Koji is back up.