Huh. A fun wrinkle with this - which I only just realized - is that there is a fun issue here with the greenwave contexts for updates.

We define two: bodhi_update_push_testing and bodhi_update_push_stable. The intent is clear: bodhi_update_push_testing should gate push to updates-testing, and bodhi_update_push_stable should gate push to stable.

Now, you might be thinking...there's an obvious chicken-and-egg problem here, right? If I'm right that CI doesn't schedule tests until the update is pushed to updates-testing, how can you possibly gate the push to updates-testing on CI tests?

And the answer is: actually you can't, because Bodhi only checks the gating status when pushing to stable! The bodhi_update_push_testing decision context is basically useless (if I'm correct).

So, well, that could be better. But only if we can solve the chicken/egg problem, of course - we can't gate push-to-testing unless we can get the tests run before the update lands in testing.

Filed https://github.com/fedora-infra/bodhi/issues/5476 for the Bodhi side of the issue with the updates-testing gate.

Related: https://github.com/fedora-infra/bodhi/issues/4187

Fedora CI runs tests for individual components, but if there are multiple builds in the update, then those extra builds are taken into account during testing.

Fedora CI is using testing.koji-build-group.build.complete because there were no bodhi.update.request.testing messages for Rawhide in the past. But that's probably no longer the case (?)

Minor drawback of bodhi.update.request.testing messages is that they don't have the Koji task id in them, so the pipeline would need to query Koji just to get that id. The task id is required as that's the unique identifier that is used by resultsdb (iirc).

No, you're right, there are no bodhi.update.request.testing messages for (most) Rawhide updates as (most) Rawhide updates are queued directly for stable.

This is why openQA listens for both messages: it's to trigger tests as early as possible both for Rawhide and non-Rawhide. That's the reason for the logic I explained above. Effectively, Rawhide updates trigger on bodhi.update.status.testing.koji-build-group.build.complete (non-retrigger) immediately on creation (because Bodhi publishes the message immediately for Rawhide updates, as they don't actually go through updates-testing). Updates for other releases trigger on bodhi.update.request.testing immediately after creation, then when bodhi.update.status.testing.koji-build-group.build.complete (non-retrigger) arrives later, we do not re-schedule the tests.

To make this easier we could enhance Bodhi to just always publish a single message for all updates immediately upon creation, of course. That would make it much easier to handle the scheduling. It shouldn't be too difficult in Bodhi.

Looking at CI results in resultsdb, item - which is the kinda 'standard' resultsdb field - is the build NVR. But you do also include the task ID, yes. I don't know if anything actually expects it to be there, I don't think Bodhi does. https://resultsdb.fedoraproject.org/results/41527565 for e.g.

So I'm writing a PR to add an update.new message to Bodhi. But, I have a few questions.

My first effort made the update.new message very similar to the bodhi.update.status.testing.koji-build-group.build.complete, but I don't love that. I would like to create something between that message and the 'standard' format used by all other messages.

It seems like in response to the requests here, the koji-build-group.build.complete was made completely different from all other messages, but I don't think it really needed to be. So my new idea is to create a BuildV2 schema which includes the additional information needed by CI, and make the update.new message - and possibly all the other messages, because why not? - use that schema.

That schema would add all the stuff from the 'build artifact' schema used in the koji-build-group.build.complete messages - type, id, task_id, component, issuer and scratch - to the BuildV1 schema, which is used for each build item in the list within the update dict in a normal Bodhi message.

The other stuff that seems to be specific to the koji-build-group.build.complete messages is:

• The contact dict. Does CI actually need this for publishing messages, or can it construct this info itself? The contents of this dict is just hardcoded to be the same for every single message:

  "contact": {
  "docs": "https://docs.fedoraproject.org/en-US/ci/",
  "email": "admin@fp.o",
  "name": "Bodhi",
  "team": "Fedora CI"
  }

• The rest of the artifact dict (aside from the list of builds), which consists of:

  • id - the update alias plus a suffix indicating the current 'version' of the update - the two parts of this are already available separately in the update dict, as alias and version_hash
  • type - this is always "koji-build-group"
  • release - this is e.g. "f40", it is also available in the update dict as ["release"]["dist_tag"]
  • repository - this is the absolute URL of the update, e.g. https://bodhi.fedoraproject.org/updates/FEDORA-2023-14a6959569 . It's not a repository, so it is not really providing the info the ci-messages schema wants. It's also available in the update dict as url

So...of all this stuff we put into the koji-build-group.build.complete messages back in the day in response to https://pagure.io/fedora-ci/general/issue/70 , how much of it is really needed? Would it actually be sufficient if we just have messages with the normal update dict, but with the dicts for each build in the update having the additional information?

Thanks!

Looking at this myself, I don't think CI actually uses any of the "contact" data. The only bits in CI that actually consume this code are the triggers, and they only really seem to use the artifact dict. The CI code that publishes messages is in the pipeline repos, e.g. https://github.com/fedora-ci/rpminspect-pipeline/blob/master/Jenkinsfile , and that code doesn't refer to Bodhi messages. It creates the 'contact' dict itself.

AFAICS, the only info the triggers actually use is the release, the builds, and the task_id for each build. Unless I'm missing something, if we extend the build dicts in the update dict in the messages a bit as I described (we wouldn't even need to put everything there!), updating the triggers would be as simple as:

diff --git a/Jenkinsfile b/Jenkinsfile
index a28c44f..f24a172 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -26,7 +26,7 @@ pipeline {
                        queue: 'osci-pipelines-queue-10'
                    ],
                    checks: [
-                       [field: '$.artifact.release', expectedValue: '^f[3-9]{1}[0-9]{1}$']
+                       [field: '$.update.release.dist_tag', expectedValue: '^f[3-9]{1}[0-9]{1}$']
                    ]
                )
            ]
@@ -45,16 +45,16 @@ pipeline {
                     if (msg) {
-                        if (msg['artifact']['builds'].size() > 20) {
-                            echo "There are way too many (${msg['artifact']['builds'].size()} > 20) builds in the update. Skipping..."
+                        if (msg['update']['builds'].size() > 20) {
+                            echo "There are way too many (${msg['update']['builds'].size()} > 20) builds in the update. Skipping..."
                             return
                         }
-                        msg['artifact']['builds'].each { build ->
+                        msg['update']['builds'].each { build ->
                             allTaskIds.add(build['task_id'])
                         }
-                        def testProfile = msg['artifact']['release']
+                        def testProfile = msg['update']['release']['dist_tag']
                         if (allTaskIds) {
                             allTaskIds.each { taskId ->

and we wouldn't have to change anything else.

Hmm, on another topic, it does occur to me that CI is probably also relying on koji-build-group.build.complete messages to re-test edited updates. When an update is edited and its builds are changed, it gets shoved back to 'queued for testing', so CI will re-test it when it gets pushed to testing again and a new koji-build-group.build.complete message is published.

So...maybe rather than creating a bodhi.update.new message we should do what @msrb suggested in https://github.com/fedora-infra/bodhi/issues/4187 and change when koji-build-group.build.complete is published, and document clearly that this message is specifically intended to be used for triggering tests. What if it was published, for all updates (Rawhide and otherwise):

• Immediately on update creation
• Immediately on update edit, if the edit changes the builds in the update
• When the re-trigger tests button in Bodhi is pressed (as currently)

I...think that would be sufficient? And at the same time, we could change it so the builds list-of-dicts within the update dict in the message contains the info CI needs, then get rid of the contact and artifact dicts...

just as an update, I'm working on this now; there's kind of a few different ways you could go about fiddling with the schema stuff here, but I've picked an approach to at least try initially and written it. next up, get the tests to pass. :D

update again: so my new approach is to change all the CI triggers to use as little of the artifact dict as possible:

• https://github.com/fedora-ci/rpminspect-trigger/pull/2
• https://github.com/fedora-ci/rpmdeplint-trigger/pull/2
• https://github.com/fedora-ci/installability-trigger/pull/3
• https://github.com/fedora-ci/dist-git-trigger/pull/4
• https://github.com/fedora-ci/eln-build-group-trigger/pull/1
• https://github.com/fedora-ci/compose-ci-trigger/pull/2

and change the Bodhi UpdateReadyForTesting message to be mostly like all the other update messages, but with a minimal version of the artifact dict still present, containing the bits the triggers need. That is in https://github.com/fedora-infra/bodhi/pull/5538 now. That PR also makes Bodhi publish UpdateReadyForTesting any time an update is edited and its builds are changed.

The next bit, however, is turning out to be rather difficult: publish UpdateReadyForTesting any time an update is created. Simple, right?

Well...not so much. It's running into quite a lot of difficulties, because we wind up kinda fighting sqlalchemy a bit. The logical thing to do is put the message publishing right into the __init__ of the Update model class - but at the point __init__ runs, I think, the new update has not been flushed into the db, which means any default values set at the model level aren't actually set yet, notably the update status, which needs to be set for the message to be valid.

It's also exposed that not every existing update creation call (in the tests, at least - I think, or at least hope, that every 'real' update creation call does this...) sets a user, which again, needs to be set for the message to be valid.

And then there's the 150 or so tests which need their expectations about what messages will be published when to be updated. That's the easy bit I hope to get to after fixing all the tricky bits.

So, this is taking a while!

I'm working on it in the same PR, so you can follow along there with your popcorn if you like...

end-of-day edit: https://github.com/fedora-infra/bodhi/pull/5538 is mostly ready now. Just needs some tidying around the edges, I think.

OK, I've marked https://github.com/fedora-infra/bodhi/pull/5538 as ready now. It's a pretty major change, so let's hope it works out as planned. Once it's deployed to staging I'll check and make sure messages are being published as intended.

I believe this is all implemented and working now. anyone opposed to closing the ticket?