releng

#10546 ejected builds in epel9-stable push
Closed: Fixed by humaton. Opened by kevin.

Closed: Fixed
Reopen Issue

bodhi is ejecting a bunch of builds. Somehow they got mistagged...

[19:59:27] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-2f2eb219f9
[19:59:31] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-73fc5e4d8b
[19:59:38] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-2be954c76e
[19:59:44] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-ee0737b3d8
[19:59:47] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-bdaaa03d2a
[19:59:53] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-4c978716b8
[19:59:59] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-1d748eecb8
[20:00:03] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-1e093e7e13
[20:00:07] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-18aee15f92
[20:00:09] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-3fae2d3742
[20:00:12] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-d96beb7033
[20:00:20] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-30e3391cd8
[20:00:21] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-c84c7d48ff
[20:00:25] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-c9b43f9208
[20:00:29] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-aa7b5a2be5
[20:00:31] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-8398d1cf68
[20:00:35] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-045ee8fed0
[20:00:36] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-9b088e14e7
[20:00:40] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-6c963b3527
[20:00:43] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-8d842cb59a
[20:00:47] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-18f4236e6c
[20:00:49] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-3f88fea6ee
[20:00:54] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-cbde73c6eb
[20:00:56] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-c643a1c74f
[20:00:58] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-d87c73168a
[20:01:03] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-b82ffb1c93
[20:01:06] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-2571e387ca
[20:01:08] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-7abc237e0f
[20:01:10] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-9092711848
[20:01:14] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-af87736f6b
[20:01:15] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-a690f02845
[20:01:18] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-a2744ff1b4
[20:01:21] <fedora-notif> bodhi.update.eject --   https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-b9354d8558

@humaton can you look at tagging them and doing another epel9 stable push to get them sorted out?

Metadata Update from @zlopez:
- Issue tagged with: medium-gain, medium-trouble, ops

Some of these are mine. Bodhi seems to be out of sync with koji. My updates are tagged with "epel9" and "epel9-pending" and have actually been pushed to the EPEL-9 repository (I can see the packages under https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/). So it seems they missed the steps of dropping the "epel9-pending" tag and the notifications/bugzilla updates that go with the push to stable.

I checked 2/3 of the builds in mentioned updates. They all are included in the repo. The weird thing is that packages are from random dates and some build in side-tags and some are not.

Maybe they need adding back into the epel9-testing tag so that they won't fall foul of the check that's causing them to be ejected from the push, so that the push can then complete correctly?

yes, the epel9-stable push is failing every day... I have had to resume it for the last few days. So, we need to fix the tagging on those so bodhi knows they completed the push to stable and doesn't just eject them and fail the push. ;(

I fixed the first 10 or so to confirm that it would fix them... resuming now.

That didn't work, so I think we need to untag them from epel9 and tag them into epel9-testing?

As an experiment I tagged perl-Test-NoTabs-2.02-7.el9 into epel9-testing (in addition to the epel9 and epel9-pending tags it was already in) yesterday. The update for that got pushed:
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-c643a1c74f

I have tagged the rest let's see what happens.

Seems it worked.

Well yes it did but now I see a similar issue for some packages being pushed to testing, e.g.
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-1eededec1a
(perl-Crypt-PBKDF2-0.161520-12.el9). I can see this package is present in the epel9-testing repo.

I guess packages affected will need to be added back to tag epel9-testing-candidate?

Might possibly be related to https://pagure.io/koji/issue/3220 - if builds are getting tagged but bodhi thinks an error happened whilst tagging, it would account for things getting pushed and bodhi not knowing about it.

For the failure to push to testing, I managed to get the updates pushed by resubmitting them for testing in bodhi (they had gone back to the "pending" state) and also manually adding them back into the epel9-testing-candidate tag in koji.

This should be now resolved.

Metadata Update from @humaton:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None

medium-gain medium-trouble ops

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.